Video icon 64
Learn to Code. Start your free trial today.
Advertisement

Control a Flex Video Application Using Mouse Gestures

by
Student iconAre you a student? Get a yearly Tuts+ subscription for $45 →

Almost all PC user interfaces are designed around the notion of moving a cursor and clicking on objects. Buttons, menus, combo boxes, lists and every other standard UI control work in this way. The mouse is used to position the cursor, and the position of the cursor is used to perform an action.

It’s a design paradigm that has worked well, but lately new technologies like touch screens and motion sensors (think the Wii) have challenged the notion of using a mouse, or even having a cursor on the screen at all..

Long before the focus on touch screens or the Wii, programs like the Opera web browser introduced the notion of mouse gestures. The concept was simple: instead of using the position of the cursor to determine what action to perform, the motion of the cursor itself would indicate an action. So by moving the cursor in a circular motion you would trigger a web browser to refresh. By moving it left you would go back in the page history, and by moving it right you would move forward.

For Flash developers there's a free mouse gesture library that enables this type of interaction with very little effort. To demonstrate how it's used, we'll create a simple video player that uses mouse gestures, rather than buttons, to modify the video playback.

Step 1: Mouse Gesture Library

Download Didier Brun's mouse gesture library here (http://www.brighthub.com/hubfolio/matthew-casperson/media/p/50540.aspx), and extract it to a convenient location.

Step 2: New Application

Create a new Flex web application and add the mouse gesture library to the list of build paths. You can access the build path property by selecting Project > Properties > Flex Build Path within Flex Builder.

Step 3: FLV

You'll need an FLV video file to test the application with. The demo version of the IMToo FLV converter (http://www.imtoo.com/flv-converter.html) will convert short videos from almost any format to FLV. If you don’t have a collection of FLV movies lying around, this tool is ideal for converting almost any free video you can download from the web. I have converted one of the demo videos that ships with Vista for this demo.

Step 4: Application Attributes

You will need to modify some of the attributes of the Application element, contained in the MXML file. Here we've specified the width, height, background color (backgroundGradientColors) and transparency (backgroundGradientAlphas). We've also set the appComplete function to be called in response to the applicationComplete event. This gives us an entry point in the application where the mouse gestures will be set up.

<mx:Application 
 xmlns:mx="http://www.adobe.com/2006/mxml" 
 layout="absolute" 
 width="600" 
 height="400"
 applicationComplete="appComplete()" 
 backgroundGradientAlphas="[1.0,  1.0]" 
 backgroundGradientColors="[#000000,  #000000]">


</mx:Application>

Step 5: mx:VideoDisplay

Add the following mx:VideoDisplay element as a child of the mx:Application element.

<mx:VideoDisplay 
 id="videoPlayer"
 autoPlay="false"
 bottom="30" 
 top="10" 
 left="10" 
 right="10" 
 source="Butterfly.flv"
 metadataReceived="metadataReceived(event)" 
 playheadUpdate="playHeadUpdate(event)"/>

The id attribute assigns a name to the VideoDisplay that we can reference from ActionScript.

The top, bottom, left and right properties define the position of the VideoDisplay.

The autoPlay attribute is set to false, which means the video will not start playing straight away.

The source attribute points to the location of the video file. If you have your own FLV video file, then you will need to modify this attribute to point to it.

The metadataReceived attribute points to a function that will be called once the details of the video have been loaded by the VideoDisplay. We use this to find out how long the video is, so we can modify the maximum value of the HSlider.

The playheadUpdate attribute points to a function that will be called as the video is played. This allows us to track the current position of the video file, and update the HSlider accordingly.

Step 6: mx:HSlider

Add the following mx:HSlider element as a child of the mx:Application Element

<mx:HSlider 
 id="videoPosition" 
 enabled="false" 
 bottom="10"
 right="10" 
 left="10"
 change="videoPositionChanged(event)"/>

The id attribute assigns a name to the VideoDisplay that we can reference from ActionScript.

The top, bottom, left and right properties define the position of the VideoDisplay.

The change attribute defines a function to be called when the user changes the sliders position.

Step 7: Interface

You should now have a GUI that looks like the one below.

Step 8: mx:Script

Add an mx:Script element as a child of mx:Application. This element will hold the ActionScript code for our application.

<mx:Script>
<![CDATA[
 //  code goes here

]]>
</mx:Script>

Step 9: Import Packages

We need to import a number of packages. This is done inside the mx:Script element. Three classes from the mx.events package, MetadataEvent, SliderEvent and VideoEvent , are used as the parameters in event listener functions. The com.foxaweb.ui.gesture package includes the classes from the mouse gesture library.

import mx.events.MetadataEvent;
import mx.events.SliderEvent;
import mx.events.VideoEvent;
import com.foxaweb.ui.gesture.*;

Step 10: Define Constants

A number of constants are then defined. The VIDEO_STEP constant defines how much time the video's current position will be moved by when we step forward or backward. The other strings all define the names of actions that will be associated with mouse gestures. In general it is prudent to map strings to constants when they are used as an identification, as it allows the compiler to pick up misspellings like if (action == SETP_FORWARD), instead of running into issues at runtime with misspelled strings like if (action == "setp_forward").

private static const VIDEO_STEP:Number = 1;
private static const PLAY:String = "play";
private static const STOP:String = "stop";
private static const PAUSE:String = "pause";
private static const STEP_FORWARD:String = "step_forward";
private static const STEP_BACKWARD:String = "step_backward";

Step 11: Define Variables

The last variable we need to define is a reference to a MouseGesture object. It's this object that implements the logic for the mouse gestures.

private var mg:MouseGesture = null;

Step 12: New Function

Add a new function called appComplete. This is the function we assigned to the applicationComplete attribute in the mx:Application element. It is here that we will initialise the mouse gestures.

private function appComplete():void
{
 mg  = new MouseGesture(this.stage); 
 mg.addGesture(PLAY,"0");
 mg.addGesture(STOP,"4");
 mg.addGesture(PAUSE,"2");
 mg.addGesture(STEP_FORWARD,"67012");
 mg.addGesture(STEP_BACKWARD,"65432"); 
 
 mg.addEventListener(GestureEvent.GESTURE_MATCH,matchHandler); 
 
}

Step 13: Mouse Gestures

First we create a new MouseGesture object. The MouseGesture constructor needs to be passed a reference to the stage in order to respond to mouse events.

mg  = new MouseGesture(this.stage);

Next we define a number of mouse gestures. A mouse gesture is defined as a sequence of mouse movements, with numbers representing movement directions as per the image below. All mouse gestures start by clicking the left mouse button and end with the button being released.

For a simple gesture where the mouse is moved in a straight line we use a string with a single number in it. Here the "play" mouse gesture is defined as mouse movement in a single direction, right, which is represented by the numeral 0.

mg.addGesture(PLAY,"0");

Likewise the "stop" and "pause" mouse gestures are defined as mouse movements down and left.

mg.addGesture(STOP,"4");
mg.addGesture(PAUSE,"2");

The "step_forward" mouse gesture is more complex. It is defined as a half circle mouse movement, starting at the left and then moving over and to the right. The red arrow shows the start of the movement.

This movement is defined by the string "67012". You can see how this string is derived by matching the movement of the mouse with the numbers that are assigned to those movements. We start moving up (6), diagonally up and right (7), left (0), diagonally down and right (1) and then down (2).

mg.addGesture(STEP_FORWARD,"67012");

The "step_backward" is defined in the same way, only this time it is a half circle mouse movement, starting at the right and then moving over and to the left.

mg.addGesture(STEP_BACKWARD,"65432");

We then assign the matchHandler function to be called when a mouse gesture has been detected.

mg.addEventListener(GestureEvent.GESTURE_MATCH,matchHandler);

Step 14: The MatchHandler Function

The matchHandler function is called when a mouse gesture has been detected. The event parameter contains a property called datas, which will match one of the constants we assigned to the mouse events in the appComplete function. Depending on the mouse gesture that has been detected we perform certain actions on the VideoDisplay. The play, stop and pause actions are all quite straight forward. With the step_forward and step_backward actions we either increase or decrease the playheadTime property of the VideoDisplay, which has the effect of skipping forward or backward.

private function matchHandler(event:GestureEvent):void
    {
        switch (event.datas)
            {
                case PLAY:
                    this.videoPlayer.play();
                    break;
                case STOP:
                    this.videoPlayer.stop();
                    break;
                case PAUSE:
                    this.videoPlayer.pause();
                    break;
                case STEP_FORWARD:
                    var newFowardTime:Number = this.videoPlayer.playheadTime  + VIDEO_STEP;
                    while (newFowardTime > this.videoPlayer.totalTime)  newFowardTime = this.videoPlayer.totalTime;
                    this.videoPlayer.playheadTime  = newFowardTime;
                    break;
                case STEP_BACKWARD:
                    var newBackwardTime:Number = this.videoPlayer.playheadTime  - VIDEO_STEP;
                    if (newBackwardTime < 0) newBackwardTime = 0;
                    this.videoPlayer.playheadTime  = newBackwardTime;
                break;
            } 
        }
    
    }

Step 15: HSlider

This demo is all about modifying the video playback using mouse gestures, but for convenience a HSlider can also be used. The metadataReceived function is called once the VideoDisplay has loaded the meta data, which includes the total length of the video. In this function we set the maximum value of the slider to the total length of the video. We then enable the slider – until we know how long the video is the slider can’t be used to set the position.

private function metadataReceived(event:MetadataEvent):void
{
 this.videoPosition.maximum  = this.videoPlayer.totalTime;
 this.videoPosition.enabled  = true;
}

Step 16: playHeadUpdate Function

The position of the slider needs to be kept in sync with the current play back position of the video. The playHeadUpdate function is called at regular intervals by the VideoDisplay, and it is here that we set the value of the HSlider to the playheadTime of the VideoDisplay.

private function playHeadUpdate(event:VideoEvent):void
{ 
 this.videoPosition.value  = event.playheadTime;
}

Step 17: videoPositionChanged Function

The slider can also be used to change the current play back position of the video. Here we do the reverse of the playHeadUpdate function, and set the playheadTime of the VideoDisplay to the value of the HSlider.

private function videoPositionChanged(event:SliderEvent):void 
{
 this.videoPlayer.playheadTime  = this.videoPosition.value;
}

Conclusion

When you load up the application you should see the video file. Because we set the autoPlay attribute of the VideoDisplay object to false, the video will be stopped, showing the first frame.

Click the left mouse button, move the mouse left and release the button, and the video should play. Click, move the mouse down, and release, and the video should pause. Click, move the mouse in a top half circle from left to right and release, and you should see the video skip a second ahead.

Another benefit of mouse gestures is that they remove the need for other UI controls, which can be a huge advantage when screen space is at a minimum (like banner ads). You could even use them for those "feed the monkey" or "do the most chin-ups" banner ad games.

Mouse gestures are very easy to implement in Flash, and they provide an intuitive way to interact with a PC. With only a few lines of code you can redefine how users interact with your application, and they free up the screen space that was reserved for more traditional UI components.

Thanks for reading, I hope you learned something!

Advertisement