64x64 icon dark hosting
Get a Tuts+ subscription for just $45! Deploy New Relic now to claim.

Using the Kinect to Control Flash: An Introduction


I'm sure that you've heard of the Kinect from Microsoft by now, the revolutionary new technology that recognises the human form and lets you control games and apps with it. When the Kinect was released, there was big news surrounding it, not due to what it could do, but what developers were doing with it. In this tutorial, you'll learn how to create a Pong game that can be controlled by moving your hands, and a picture gallery that can be navigated in the style of Minority Report -- all in the browser's Flash Player with no drivers required, if you're on the Mac.

The Kinect got hacked within a few weeks, and developers started using it to control things other than the Xbox 360. This was huge, suddenly every nerd's dream of living out that famous scene from minority report was coming to life. But for us flash dev's we couldn't get in on the action without having to learn C++. That makes it tricky, for someone like myself who just wants to make games with it.

Well, luckily for us, we now can, thanks to something called TUIO and some very nice developers who got kinect recognition streaming out the data via TUIO.

This description of TUIO is taken directly for their site, tuio.org:

TUIO is an open framework that defines a common protocol and API for tangible multitouch surfaces. The TUIO protocol allows the transmission of an abstract description of interactive surfaces, including touch events and tangible object states. This protocol encodes control data from a tracker application (e.g. based on computer vision) and sends it to any client application that is capable of decoding the protocol.

That's quite a description. Basically, what it does is allow the decoding of touch interfaces like touch tables, through blob recognition. What clever dev's have done is transcode the Kinect data into "blobs" so that TUIO can recognise them. From there, Flash can read the data with the help of TUIO's native as3 libraries.

What TUIO can also do is recognise gestures. This is quite a nice feature as it allows you to do things with the kinect as if it was a big imaginary touch table. You can shrink or expand objects using a larger, more elaborate version of the pinch gesture, it can regonise swipes and touch gestures too.

For basic implementations though, a Kinect plus TUIO lets you track hands, which is what I'll be covering first since that is something I've most recently used with great success in my project Spaced Out.

Spaced Out is a very experimental game. It used the Kinect to track players' hands which served in game as the equivalent of a mouse. You can track the position of hands on screen and assign a cursor to them. In my game I used this in conjunction with a brainwave reading headset to blow up enemies on screen by tracking the players hands and also by reading their concentration levels.

You can see a video of all this in action here:
Spaced out gameplay

Spaced Out Gameplay from jon reid on Vimeo.

Different TUIO Trackers

There are a few different TUIO trackers for the Kinect right now. I'll explain briefly what each one does in a minute, but they all have their advantages and uses for different occasions. Being able to have the freedom to choose which one we can use is a great advantage for us.

The great thing about TUIO is that all of these will use the exact same AS3 code that I'm going to show you today. So for different uses, you can just run a different TUIO tracker and your game or app will need no rewriting whatsoever.

The downside to TUIO trackers for the kinect right now is that they are generally mac only. This is due to driver issues when the Kinect first got hacked. I'm sure though, that as time goes on that more developers will release TUIO trackers for windows, especially since the official Microsoft SDK for Kinect has been released on Windows. In fact, there is a TUIO client for Windows right now! But it's still in beta and you cant control the depth of the camera so I had to stand very far back to get it to work. It's not something thats as easy to use as the Mac versions.

So the first one, known as TUIOKinect, tracks blobs, which it determines are hands. I personally find this one great as it needs no setup or calibration from the user. A player can just walk up to your game and use it instantly. The downside is doesn't know exactly what a hand is, it's just guessing. so if you are standing too close to the Kinect, it could think that any part of you is your hand. Only a minor issue when working with the zero calibration.


The second tracker, OpenNI2TUIO, tracks the entire skeleton to determine which part of you is a hand. Brilliant, very accurate tracking of hands. This one however does require the "superman" pose, so it's good for singular apps that you are using for lengths of time, or ones where you require more accurate recognition of hands.


The Windows version, if you are interested in playing around with that, is called Open Exhibits TUIO Kinect. I got this running pretty quickly; you have to follow the readme.txt that comes with it and install some drivers, but it does just about work. It doesn't give you the same control over what it tracks like the other two do, but it tracks what it thinks are hands just like the Mac version of TUIO kinect. But you Windows guys can play along too.

Windows TUIOKinect

Step 1: Downloading Software

For this tutorial, I'll be using TUIOKinect, just because I like being able to sit at my desk and still use the kinect. Of course, you can use one of the other trackers, but if you follow along as I cover the basics, then hopefully using another tracker will become nice and easy.

Download one of the trackers from the links above, for mac, the trackers dont need any installing and run straight away. For Windows you'll need to install some drivers. They cover that in their readme.txt so I won't go over that here.

Step 2: Download the Bridge

Downloading TUIO kinect isn't the only step, you need a way to send that data specifically to flash. There is a bridge that does just that and is very simple to run. udp-flashlc-bridge by Georg Kaindl is the software I'll be using today. You can get that here:


NOTE, if you are running the Windows version, you don't need this as it comes bundled with similar software.

Step 3: Unzip Flash Bridge

Once you have downloaded that, extract to somewhere like your desktop or your documents folder, somewhere thats easy to get to. I say this because we have to run this bridge via a terminal command.

Step 4: Rename the Folder

Rename the folder to "flashBridge", without the quotes, just because its default name is very long winded.

Step 5: Run the Bridge

Lets do that now, we can start it now and it'll be running for the rest of the tutorial. Once it's up and running, you don't need to worry about it again, it's a very stable bit of software.

Open up terminal and enter the following, I use desktop here, but replace that with wherever you put your folder:

Your terminal window should now show two extra lines and confirm that its running fine. If it isn't, it'll tell you, so make sure you go over the last couple of steps and confirm that your are following them exactly.

It should look something like this:

Step 6: Download the Necessary Libraries

Download the newest TUIO library from this link: TUIO Library
When you unzip the folder, you'll have two folders inside, one being demos, one being org. We just want the org folder. You don't need to move it anywhere just yet, I'll get to that in a minute, you just need to have it there ready.

You also need to download the greensock tweening library, TweenMax. You can find that here: TweenMax.

Create a new folder where we'll be saving all our code into. Call it KinectGame. You'll need to copy over the org folder from the TUIO library into this folder and also the com folder from the TweenMax library.

Step 7: Starting to Code

Download my source code and open up the folder KinectGameStart. You'll find a FLA called Main that contains all the assets you'll be needing. Copy that into your build folder and open it.

Type in the class box Main and click the pencil icon.

Save the file as Main.as.

Now, at the top of your file where we import other classes, add the following:

This just imports the basic TUIO classes that we'll be working with for now.

Step 8: Adding Variables

Next we add in some variables and constants:

These just handle the basic parameters to help set up our TUIO connection.

Step 9: The Constructor Method

Next you'll want to add the following into your constructor method:

You'll also want to alter the following line:

to the following:

This just allows us to create graphics and add in the listener function so that we can access the TUIO data directly in this class.

It's all been pretty straightforward at the moment, but nothing exciting has happened yet either. Let's change that shall we? Let's start accessing the Kinect data.

Step 10: Making a Cursor Class

When we move our hands around in the air, like we just don't care, we need to have something show up in Flash that shows us where our hands are. It's not much fun playing a game where you can't see the cursor.

For now we shall just make a simple class that adds a MovieClip to the stage every time it finds a hand.

Go ahead and create a new ActionScript file. Call it Cursor and save it in the same directory you saved Main.as.

Copy and paste the following code into this class:

All this code does is add a movieclip and allow you to delete it from the display list. Pretty straightforward stuff.

You'll also notice the moveTo function. This, when we call the update in Main.as class, will move the cursor around the screen and ease it to the new coordinates. It just makes it feel a bit nicer.

Step 11: Adding Cursors to the Screen

So now that we have something to add to the screen, lets code in the ability to add the ball to the screen and then remove it, depending on whether your hands are in or out of range of the Kinect.

Now, this isn't overly complicated, you just need to have several functions in your Main class. These grab the data from TUIO and then you tell the cursor object to update its x and y position every frame.

Updating the position is already done for you in the Cursor.as file, you just need to tell it where to go.

Go back into your Main.as file.

Copy and paste the following after the constructor method in your Main.as file.

Now, if you try and run it as is, you'll get a handful of errors saying that it's missing some extra functions etc. That is because the TUIO library allows the use of all the different tracker output methods from cursor to blobs. What we need to do is edit a file within TUIO so that we only need these three functions.

In the folder org, you'll find the folder tuio. In that folder, open up the file lTuioListener.as

What you need to do is comment out the following lines:

And now save the file.

It should look something like the following:

You can run the file now, but you'll see that you can't control anything. That's okay; run TuioKinect, you'll need to set the far threshold to something so that when your hands are outstretched, TuioKinect looks like this:

As a vague guide, I sit about half a meter away from my Kinect, and I need to set my far threshold to 90.

When this is set, you should be able to see balls appear in the flash file on screen where your hands are! Fantastic stuff, you're controlling a Flash file with your hands! Imagine the possibilities for what you can do with this.

Step 12: Pong Kinect

Now that you have control of Flash with your Kinect, what on earth can you do with it? Well, you can use the Kinect to control all sorts of things in Flash. Often you can just use it to replace an object that could be controlled by the mouse or keyboard.

What I'm going to show you now is how to make a pong game where both paddles are Kinect controlled. Two player Kinect pong! A great party game.

To control the paddles, you'll need to add in the following code into the updateTuioCursor function in Main.as; this goes in the try { } method, after the hands.moveTo line:

And add the following amongst the other variable definitions.

Also, add the following to the constructor method:

Go ahead and try that out, what you've got now is a control mechanism that controls the left paddle if your hand is on the left half of the screen, and controls the right paddle if your hand is on the right half of the screen. It's a very easy to use and intuitive way of controlling Pong with two players with zero setup, and thats what I'm all about.

I havn't done anything too special here, what I've done is setup a check to see if the position of the ball is on the left portion of the stage or the right. It then controls the appropriate paddle up and down in relation to your hand movements.

Some of you may be wondering what the following line is all about:

That, is some basic easing code which I have learned from reading the brilliant Foundation Actionscript 3.0 Animation book by Keith Peters. It just figures out where you want the paddle to be in relation to itself then adds an amount, the "easing" number, to smooth the transition. Keith can explain it a lot better than I can though, I just figure out where to put it.

You might be wondering why you would put any easing code in at all when it will move up and down perfectly well without it. Well, thats all just to make it nicer to play, it adds a little weight to the paddle, just to give it some feeling to it. Just a little player psychology there, making the things you control in the game feel real.

Step 13: Bouncing Balls

Now, I'm not going to go through and explain what this code is doing thoroughly, since I'm not doing a tutorial about making Pong, this tutorial is all about controlling a game with a kinect.

This code makes a ball bounce off the paddles and the top and bottom of the screen, the basic components needed to make pong work. With this and the Kinect controls, you can see for yourself how easy it is to make a game work with the Kinect.

Add the following to your variables:

In the constructor, add the following:

And then after the constructor, add the following:

What all that does is, it moves the ball around the stage and bounces it off the paddles and the top and bottom of the screen. The code also sets the starting point of the ball randomly and adds it to the stage. To save repetitive code, I add in an "added to stage" listener to the ball so that when we remove the ball, then add it to the stage again, it assigns a random velocity and direction to the ball's movement.

This works well for our small game, and it manages the ball for us.

You can go ahead and try this all out with the kinect. The ball will bounce around between the two paddles that you are controlling with your hands. Pretty nifty ay?

If you are having any trouble with this and are getting errors, have a look at my source code in the folder KinectGameFinished. There you can see all the source code for this little game and check it against what you have got already.

Step 14: Gestures

Some of you may remember the film Minority Report, specifically the scene where Tom Cruise stands in front of a huge transparent display and with a few waves of his hands, he's moving images and data around the screen.


Look at that interface, it's gorgeous. It's very exciting that this can actually be made by us in our own homes now.

Well, with the help of TUIO, the Kinect and Flash, you can build such apps. I'm going to show you how to build a simple photo viewer app where you can move the photos around the screen, make them bigger and smaller with a "pinch" gesture, and rotate the images using another gesture.

It works in a similar way as gestures do on a touch screen phone or touch table, but bigger and involving two arms. In fact, the code I'll be showing you takes advantage of the built-in gesture handler in Flash.

In the source files I've provided for you, open up the folder KinectGesturesStart. This holds all the files that you need to get started with.

If you open up the class Main.as, you'll be able to see that I've provided you with a base class that looks similar to the file we created for the kinect game earlier. This time, we won't actually be assigning an object to our hands, we'll be using the debug feature instead to see them.

In an earlier step, we only used the functions add, remove and updateTuioCursor. Now you can see that there are a lot of empty functions for object and blob, which I told you to comment out of TUIO earlier. You may be wondering why they are suddenly back. It's simply because gestures wont work without them. I'm not sure of the reason behind that, but when I've been playing about with this, it wouldn't work without all the functions running.

You don't need to worry about having to go back and uncomment anything, I've already taken care of that for you in the TUIO library in this folder.

So let's get you set up with some gesture control.

Step 15: Setting Up Gestures

Firstly, you need to add in some extra variables. Add the following to your variables:

Now add the following to the constructor method:

Now let me guide you through what that chunk of code does.

This just sets up the gesture manager and initialises the stage with the TUIO connector client.

This is quite a vital line as it tells the gesture manager to let you use the drag gesture on objects. Other gestures work without this line, but you wont be able to drag objects around the screen. So if you wanted stationary objects that you can rotate or resize, remove this line.

This would be useful for a virtual DJ app, where you rotate a virtual vinyl to alter the playback of music. You can do a lot with a few simple gestures.

This sets up TUIO's debug feature which you'll get to see in action soon.

Finally, this for loop adds our images to the stage. Here I take advantage of how for loops run to pass a number through to the class we are about to create which tells a movie clip which frame it needs to show. This way we use one for loop to show all our images. Saves the hassle of writing all that extra adding and positioning code.

Step 16: In a Class All of Our Own

So we just referenced a class called Images, which doesn't exist yet. We'd better do something to fix that.

Go ahead and create a new class file and call it Images.

Let's start at the top of the file and work our way down.

Step 17: Importing Goods

Add the following imports to the start of the file:

Step 18: Variable Success

You need to extend the class using the sprite class, so the main class definition line looks as follows:

Now add the following variables:

Picture is a movieclip that is already set up and made for you in the file Main.fla. It contains all the images that are shown when you load up the app. All the images are ones I've taken myself, so feel free to use them how you wish.

Step 19: Constructors

The main constructor line looks as follows, currently:

Now, you may remember that we passed this function a bunch of parameters for setting up the image's position, size and frame. To handle the importing of these parameters, that line needs to change to this:

Now you can add in the code that sets up the whole look of each image.

What this code does is draw a white box and drops the picture MovieClip inside it. This creates a nice photo-like effect as it gives each image a border.

You can also see that it changes the frame of the movieclip to be whatever position it currently is in in the for loop.

Finally, we'll add the event listeners to set up the different gestures which we'll be handling in a second.

Step 20: Scale That Building. Of Gestures.

Add in the following function after the constructor:

This function just handles scaling when Flash detects you are doing the scale gesture. This is Flash's built-in method for doing scale gestures, so you don't need to do any math to figure anything out, great!

Step 21: Rotate Me Right Round

Add this function underneath the last one:

Again, this uses the built-in function for handling rotation gestures. Simple.

Step 22: Such a Drag

Add the following function underneath the last:

This function handles the dragging behaviour so you are able to put your hand over an object and drag it around the screen.

Step 23: Up and Down

Add in the following functions:

Right, these functions handle what happens when you move your hand over and object on screen. Basically, it brings the current object you are hovering over to the top of the display list so that it becomes the top most object. This just makes it easier to control objects that are underneath others.

Step 24: Work Those Arms

Give it a test! Hopefully, if you followed all the steps carefully, then you should now be able to move pictures around the screen, resizing and rotating them just by moving your hands around in thin air! Welcome to the future.

The gestures work as follows:

Zoom is two hands on one image and then you expand them away from each other.

Rotate is two hands on an image, one going up and the other going down.

Imagine all the cool apps you could build with this kind of tech. You could add in gesture recognition for your Kinect-enabled Flash games and merge the two things you've learned today together. Easily accessible menus by doing a zoom gesture to the stage, objects you can move around using the drag gesture, all kinds of stuff.

But, you have to be aware that it's very difficult to keep your arms extended in the air for long amounts of time. So if you are making a game, its best to keep any arm extension movements to short amounts of time.

It's no secret that I'm not a big fan of the Minority Report style interface because it's completely flawed as an interface. But, using it in small bursts with rest periods works well -- it's what I did in my game, Spaced Out: each round was one minute long with at least a minute if not more rest time in between. Any longer and people complained of arm fatigue.

Of course, what's even better is a touch table type app, where you use the Kinect to detect fingertip touches on tables and then build apps that works using that. Touch tables everywhere! You could make a touch table tower defence game using nothing more than a Kinect, Flash and a projector. Tabletop gaming with animations, which can react to your touches.

I have given you the knowledge, now use it responsibly and for good, not evil. Have fun playing with the future.