FIREFLY & KINECT

Like many people, I've been anxiously awaiting the official release of the Microsoft SDK for the Kinect.  Now, that its officially out, I spent some time over the last two weeks working on a set of Kinect related components that I hope to include in the next release of Firefly (1.007).  The first component I tried to implement was the Skeleton Tracker... and I have to admit that the result are quite promising.  It's surprisingly fast and as long as you stay within the specified range of the sensor, the results are quite good.  Using this component I put together two very quick demo videos.

There has been a big push over the last decade to develop novel 3D technology for multimedia displays (whether its new ways for stereoscopic projection, refractive lens, etc.) One of the most successful implementations and inventive (in my opinion) was Johnny Chung Lee's reverse engineering of the Wii sensor bar.  Another recent example (and equally impressive) is this hack using the Kinect sensor and head tracking. 

The video above is my first attempt to create a real-time 3D display system within Grasshopper using Firefly's newSkeleton Tracker component and some simple camera manipulation. The Skeleton Tracker component outputs a list of points (click here for further explanation).  From there, I simply use the Horster Camera Control component (another 3rd party plugin for Grasshopper) to position the camera at the viewers head and the camera target at a point in space locating the Kinect sensor.  It really is that easy.  Turn on some real-time shadows and you've got a real-time 3D display.  It still needs some tweaking but it's pretty fun to play with. 

This next demo shows how easy it is to turn gestural movements into physical actuation using an Arduino.  The setup is very simple.  My z-value of my right hand (basically the height of my hand) controls the brightness value (or Pulse Width Modulation - PWM) of the LED.  My left hand controls the servo.  When my hand is by my side, the servo goes to position 0 and if I raise my hand above my head the servo moves to position 180.  So simple.  Of course, this could be expanded to control all sorts of things... perhaps that is next. 

FIREFLY NEW FEATURES

I was extremely excited to announce the official release of Firefly version 1.006 earlier this week.  For those who aren't familiar with Firefly, allow me to provide a short introduction.  Firefly is a set of software tools dedicated to bridging the gap between Grasshopper (a free plug-in for Rhino), the Arduino micro-controller, the internet and beyond. It allows real-time data flow between the digital and physical worlds and will read/write data to/from internet feeds, remote sensors and actuators, mobile phone devices, the Kinect, and more.  There are a lot of new components in this release (including the Arduino Code Generator, Upload to I/O Board, UDP and OSC Listeners and Transmitters, XML Search, and State Detection) that I thought it would be a good idea to put together a few videos showing some of the latest features.  So without further ado...

This first video shows the potential of the new Arduino Code Generator and the Upload to I/O Board components.  In my opinion, one of the greatest limitations of the previous versions of Firefly was that your Arduino board always had to be tethered to your computer via the USB cable.  This was because Firefly communicates back and forth to Grasshopper through serial communication.  However, sometimes you just want to use Grasshopper (and its visual programming interface) to prototype your design and then unplug it from your computer to run off external power.  Now, you can! 

The Arduino Code Generator attempts to convert any Grasshopper definition into Arduino compatible code (C++) on the fly.  It works by detecting components that are 'upstream' from the Uno/Mega Write component.  The Code Generator checks the component ID against a library of custom C++ functions which then get added to the code if there is a match. The code can be simultaneously saved as a .pde (Arduino Sketch) file to be opened in the Arduino IDE.

In addition, there is also a new Upload to I/O Board component which allows you to upload any sketch (could be from the Code Generator or any other sketch) directly to your Arduino board from within the Grasshopper environment. A lot of stuff happens behind the scenes with this component.  Essentially it creates a dynamic MakeFile and calls a shell application to convert the .pde file into a .cpp (C++) file and then into .hex code (machine readable code) to be uploaded to the microcontroller. Note: WinAVR is required to be installed on your machine in order to properly upload sketches to your board.  You can download the latest version here. 

There are also a lot of great network tools included in this release, including the UDP and OSC Listener and Transmittercomponents.  OSC (Open Sound Control) messages are essentially specially formatted UDP messages which can be particularly handy when you want to send some sort of information across a network (either wirelessly or LAN).  OSC messages are particularly useful because each message contains some metadata and a value, giving you more information about what type of data the message contains.  These new components open up a whole new world of possibilities by allowing you to send/receive data from smart phones (iphone or android) or by sharing documents among friends or colleagues over a network.

The video above uses the BreathOSC application (free from the iphone app store) developed by Thomas Edwards to simulate wind effects in Grasshopper.  Simply breathe into the microphone and an OSC message is sent to a specified IP address on a UDP port.  I then simply use the OSC Listener to decode the message and uses its value to create a wind vector to drive the Kangaroo (another 3rd party plugin for Grasshopper) wind simulation.  Daniel Piker, the developer of Kangaroo, helped setup this demo... and I have to say... it's quite fun. 

Another useful networking application for smart phones is TouchOSC (available for both iphone and android).  It supports sending and receiving Open Sound Control messages over a Wi-Fi network using the UDP protocol.  You can also create your own interfaces using the TouchOSC Editor and sync them directly to your phone. In this example, I've created a simple layout to control a few LED's, a tri-color LED, and a standard servo using the new OSC Listener in Firefly.  This is just a simple test, but the sky is the limit with this type of control over mobile phone interface design.

If you are interested in learning more about Firefly, check out our website at: http://www.fireflyexperiments.com/

The website has a lot of good tutorials and examples to get you up and running in no time.  As always, if you have a suggestion or want to send us a comment, you can reach us at info@fireflyexperiments.com

Acknowledgements:

It is without a doubt that this release would not have been possible without the tremendous support from Prof. Panagiotis Michalatos at Harvard's GSD.  His guidance over the last 6 months strongly influenced the development of the Firefly_X toolset and I owe him a great debt of gratitude for his assistance.  Firefly is built upon the Grasshopper plug-in for Rhino, both developed by Robert McNeel and Associates.  The Arduino language syntax is based on Wiring by Hernando Barragan. The Arduino environment is based on Processing byBen Fry and Casey Reas, and is now supported by an amazing team of software and hardware developers that continue to refine and expand its capabilities.

USING GRASSHOPPER TO CONTROL SERVOS

The above is a follow up video to the posting I made last week showing how we can use a Wii nunchuck to control parametric data in Grasshopper.  However, in this video, we are looking at reversing the flow of information and using parametric data (numeric sliders in Grasshopper) to control two different servos.  In this case, one slider controls the panning servo, while the other controls the tilting servo.

In this demo, we need to set up two different files.  The first is the Grasshopper definition (written for ver. 0.6.0019) which sends the numeric slider information over the serial port via a custom VB.NET component.  This component formats the two variables (the pan slider and the tilt slider) into a string that is separated by a comma.  This comma will serve as our delimeter when we read the data over the serial port on the Arduino side.  You can download the Grasshopper source file below.

On the Arduino side, we first have to understand that serial communication receives individual bytes of information as a stream.  This can become complicated because most data types are more than 1 byte... so you have to understand the data type and then convert it on the Arduino side to the format you are looking for.  In this example, we are reading a string into the buffer and then working through each individual byte until we reach our delimeter (don't forget about the comma).  Once we reach the comma, we pass the data that has been read into the buffer into a variable that ultimately controls one of the servos.  We do a similar process for the second set of bytes after the comma.  The Arduino source code can also be downloaded below.

Again, I'd like to thank Robert Lee for his explanations and insights on how to set up this demo.

Source Files:
Grasshopper_controlling_PanTilt_Servo.pde (Right-click to 'Save Link As') This is the Arduino source code needed to program the micro-controller.  This file was written for Arduino IDE ver. 0017.

Grasshopper to PanTilt Servo.3dm (Right-click to 'Save Link As') This is the Rhino file that accompanies the Grasshopper definition.

Update: Please use the new FireFly toolbar to facilitate the Grasshopper to Arduino connection.  Click here for more information.

Note: These documents are in the public domain and are furnished "as is". The author, Andrew Payne, makes no warranty, expressed or implied, as to the usefulness of the software and documentation for any purpose. This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License. http://creativecommons.org/licenses/by-sa/3.0/us 

USING A WII TO CONTROL GRASSHOPPER

The video above demonstrates how we can feed a stream of live sensor data (in this case, I chose to use the accelerometer data from a Wii nunchuck) into Grasshopper to control a parametric model.  I'll be the first to admit that this is not a 'new' demo... In fact, this demo was heavily inspired (in concept) by this video that was created in 2008 that shows how a Wii nunchuck can control a mechanical arm in 3D Studio Max . 

However, my own research was primarily concerned with understanding how to get the sensor data into Grasshopper (and how we can also reverse this flow of information to get Grasshopper to control servos, motors, LED's, etc...) in order to create more complex and sophisticated digitial and physical environments.

Before we get too far, I'd like to say thank you to Robert Lee for his insights on how to setup the VB.NET listener.

Now onto the more technical aspects.  Let's start with the hardware setup.  I am sending the accelerometer data from the Wii nunchuck to an Arduino Diecimila micro-controller which allows me to format the data into a block of code that the Grasshopper VB.NET listener will be able to understand.  I could have spliced up my Wii controller to connect the various parts of the circuit to the appropriate pins on the Arduino board, but I decided to buy a wonderful and cheap little adapter that simplifies the entire process.  You can buy your own from these online stores:

- FunGizmos.com. International shipping for $1 more.
- Tinker.it (UK)
- Little Bird Electronics (Australia)
- Sparkfun. Ships domestic & internationally. Be sure to order header pins too!
- Freduino.eu (EU)

Now, that we've properly connected the Wii nunchuck to our micro-controller, we'll need to program our Arduino board so that it can read the incoming sensor data and send it over the serial port.  For this, I have used a custom built library written by Tod E. Kurt that was designed to help the communication process between a Wii controller and the Arduino.  You can download the "nunchuck_funcs.h" library from his website at: http://todbot.com/blog/.  You can also get the source file needed to program your own board by clicking on the links below.

On the Grasshopper side of things, we need to create a custom VB.NET listener component that can read a string of data coming in over the serial port.  The nice thing about this listener is that it will display any string that is being sent from the Arduino... What you do with that string is another matter... but in this example, we are feeding a string of information that is separated by a comma ",".  The comma will be our delimeter which essentially tells us where to split the string so we can extract the integer value on the left side of the comma (which cooresponds to the X-value of the accelerometer) and the integer value on the right side of the comma (which will cooresponds to the Y-value of the accelerometer). 

Once we have successfully read the data from Grasshopper, we can connect a Timer Component to automatically refresh the solution at a given time interval.  In the video above, I was able to refresh the sensor data at 50 millisecond (the smallest time interval that is currently available in Grasshopper) without any lag or delay.  You can download both the Rhino file and the Grasshopper definition needed for this demo by clicking on the links below.

Source Files:
Wii Nunchuck to Grasshopper.pde (Right-click to 'Save Link As') This is the Arduino source file.  This file was written for Arduino IDE ver. 0017.

Wii Nunchuck to Grasshopper.3dm (Right-click to 'Save Link As') This is the Rhino file that accompanies the Grasshopper definition.

Update: Please use the new FireFly toolbar to facilitate the Grasshopper to Arduino connection.  Click here for more information.

Note: These documents are in the public domain and are furnished "as is". The author, Andrew Payne, makes no warranty, expressed or implied, as to the usefulness of the software and documentation for any purpose. This work is licensed under a Creative Commons Attribution-Share Alike 3.0 United States License. http://creativecommons.org/licenses/by-sa/3.0/us 

G-CODE WRITER FOR SURFACE MILLING

In this video, we will walk through how to setup a Grasshopper definition that will write the G-code for surface milling on a 3-axis CNC milling machine.  When surface milling - the contouring engine is crucial.  For speed reasons, I found the C# contouring definition written by Giulio Piacentino works extremely well because it allows the user to specify a step distance (tool path offset).  After we have generated the contours (or tool paths) we can subdivide each curve, using the evaluate length component, to create a series of points which will guide the CNC mill head in order.   Because we now have data structures, we can weave formatted text strings that have been derived from the point data into a G-code that will work with the ShopBot CAM software.

As in the previous example, we have formatted our text strings to work with the opensource CAM software, Shopbot.  You can download a free version of the software here: http://www.shopbottools.com/controlsoftware.htm.

 You can also check out this video in HD after the jump.

Note: This video is for demonstration purposes only. Andrew Payne and LIFT architects will not be held responsible for any damages that might arise from using this method. Please consult your machine's operating manual before implementing this fabrication method. 

G-CODE WRITER FOR 2D SHAPE MILLING DEMO

In this video, I will explain how to setup a Grasshopper definition that will write the G-code for 2D shape milling on a 3-axis milling machine.  G-codes are the codes that position the tool and do the actual work, as opposed to M-codes which manages the machine.  Actually, there are a whole host of different codes that can be written to control a CNC mill, however G-codes are what we are most interested in for this definition as it will allow us to streamline our fabrication process by sending the parametric data from Grasshopper directly to a CNC mill.

In this example I will be writing the tool path information for a 2D Vornoi Pattern generator, originally written by Sang Hoon Yoon.  You can find his vornoi diagram definition on his site www.sac3.blogspot.com.  This video explains how to use your data-structure (trees) to help weave together various text strings which control how the CNC mill head will operate.

I have formated the text strings to work with the open source ShopBot CAM software.  This software is excellent for many reason, not least of which is the fact that it's free and has a large online help community... much like Grasshopper.  It also allows the user to preview the cut even if their computer isn't connected to an actual CNC mill which can come in handy while testing out your cuts.  Syntactically, ShopBot part files are setup slightly differently from traditional G-code which harkens back to the days when cutting files were stored on punched tape and it's format was optimized for brevity.  You can find a lot of helpful resources on how to setup ShopBot part files in their Programming Handbook.  

You can check out this video in HD after the jump.

Note: This video is for demonstration purposes only.  Andrew Payne and LIFT architects will not be held responsible for any damages that might arise from using this method.  Please consult your machine's operating manual before implementing this fabrication method.