How to Add Mouse Gesture Control to Your Flash Projects: Multi-Stroke Gestures

In the first part of this series, I introduced a class to handle single mouse stroke detection: MGesture. This tutorial takes it a step further, by detecting a sequence of strokes.

We'll be using the class GManager for this, in tandem with MGesture. After briefing through GManager functionality, an application demonstrating its use will be developed.

Final Result Preview

Let's take a look at the final result we will be working towards. To use the Flash presentation below, use your mouse to perform the gesture as indicated by the arrow in the top left corner. Gesture by pressing the left mouse button, moving the mouse while holding the button, and then releasing it.

Step 1: Recap Direction Numbering

First step, a revision on integers used to represent different directions from Part 1. Hopefully, you can form a mental picture of the diagram above as we shall refer to it heavily throughout this tutorial.

Step 2: Strokes Detected

As user makes a gesture by any pointing devices (mouse, tablet pen, etc), the vectors successfully detected over time actually form a series of integers. Refer to diagram below.

Notice the unique integers in the sequence are 0, 4, 1. However, I shall say unique integers are 0, 1. Lets face it: it's difficult to make an accurate stroke on my Bamboo tablet, and even more so with a mouse. Errors are inevitable. Our algorithm should then be based upon the high repetition of integers in a continuous string. They are 0, 1 here - 4 is probably due to inaccurate stroke. Therefore, this gesture can be uniquely identified as a sequence of 0, then 1.

I've included another few gestures with their accompanying unique sequences below.

Step 3: Introducing GManager

The purpose of this class is simple:

1. To register unique stroke sequences to compare against.
2. To capture sequence detected of current gesture made by user.
3. To compare sequence in (2) against all stroke sequences in (1).
4. To return search result.

Developer has to define a set of stroke sequences to detect initially. Once done, event listeners to capture user gesture at runtime can be programmed. If the current gesture's sequence successfully matches any one sequence in the predefined set, different operations can then be carried out.

Step 4: Variables

GManager has the following variables:

Step 5: Methods

Methods in GManager are as below:

Properties of GManagerare as below:

Step 6: Register Predefined Stroke Sequences

As mentioned, the first step is to register unique stroke sequences that the entered gesture can be compared against. The constructor initates variables needed (highlighted) and methods that follow registers into and removes from predefined stroke sequences.

/**
* Constructor to initiate gestSeq, stokes, gestName
*/
public function GManager(){
	gestSeq = new Vector.<Vector.<int>>;
	gestName = new Vector.<String>;
	strokes = new Vector.<int>;
}

/**
* Method to register a set/ sequence of gesture
* @param sequence Numerical representation of gesture sequence
* @param identifier Name representation of gesture
* @param allowed Stokes allowed: Both (0), main four (1), diagonals (2)
*/
public function register(sequence:Vector.<int>, name:String, onlyStrokes:int = 0):void {
	gestSeq.push(sequence);
	gestName.push(name);
	strokes.push(onlyStrokes);
}

/**
* Method to remove a specific set of gesture
* @param selectSet Selected sequence to remove
*/
public function remove(selectSet:int):void {
	gestSeq.splice(selectSet, 1);
	gestName.splice(selectSet, 1);
	strokes.splice(selectSet, 1);
}

/**
* Method to remove all predefined gestures.
*/
public function removeAll():void {
	gestSeq = new Vector.<Vector.<int>>;
	gestName = new Vector.<String>;
	strokes = new Vector.<int>;
}

Step 7: Properties

Below are the properties specifically for set holding stroke seqeunces, gestSeq:

/**
* Property to get total sets of predefined gestures
*/
public function get length ():int {
	return gestSeq.length;
}

/**
* Property to get current setting on sequential gesture detection
*/
public function get useOrder():Boolean {
	return _order;
}

/**
* Property to determine whether sets of predefined gestures should be detected sequentially
*/
public function set useOrder(value:Boolean):void {
	_order = value;
}

Step 8: Record Current Stroke Sequence

After registering stroke sequences to detect against, we can prepare gestCurrent and record consequtive valid singular strokes into it. Methods below will provision for it. Tracer is a method to trace the current gesture's sequence.

/**
* Method to start recording current set of gesture made
*/
public function start():void {
	gestCurrent = new Vector.<int>;
}

/**
* Method to populate gesture into current set of gesture
* @param gestureCode Gesture detected from MGesture
*/
public function populate (gestureCode:int):void {
	//Accept only valid singular gesture (0~8) into current sequence
	if(gestureCode > -1) gestCurrent.push(gestureCode);
}

/**
* Method used for debugging. Traces gestCurrent
*/
public function tracer():Vector.<int> {
	return gestCurrent;
	trace(gestCurrent);
}

Step 9: Looking into gestCurrent

By now, we can already write a program to check out current gesture's sequence as mentioned in Step 2. You may download the source and look at the class CheckOut2. Turn on your Output panel in FlashDevelop to view the sequence of integers committed by your current gesture.

I've also included a demo of it below and placed a TextField to display the sequence instead of the Output panel in FlashDevelop.

Step 10: Implementation of CheckOut2

I have included source code of checkOut2 as below. Its not tough to understand the logic provided you have understood how event handlers are assigned. (Read this tutorial to find out more about event handlers.) I've also highlighted important points where GManager 's methods are being used.

public class CheckOut2 extends Sprite
{
private var ges:MGesture; //instance of singular gesture detection
private var gesMan:GManager; //instance of sequence gesture detectio
private var t:TextField;

public function CheckOut2()
{
	//initiating MGesture to allow singular gesture detection
	ges = new MGesture(stage);

	//initiating GManager to allow sequential gesture detection
	gesMan = new GManager();

	t = new TextField()
	t.x = 50;
	t.y = 50;
	t.autoSize = TextFieldAutoSize.LEFT;
	addChild(t);
	stage.addEventListener(MouseEvent.MOUSE_DOWN, start);
	stage.addEventListener(MouseEvent.MOUSE_UP, end);
}

private function start(e:MouseEvent):void
{
	//Start detecting singular and sequential gesture
	ges.start();
	gesMan.start();
	stage.addEventListener(MouseEvent.MOUSE_MOVE, check);

	//Start drawing gesture
	graphics.clear();
	graphics.lineStyle(3);
	graphics.moveTo(mouseX, mouseY);
}

private function check(e:MouseEvent):void
{
	gesMan.populate(ges.evalDirections()); //Feed singular gesture into a sequence
	graphics.lineTo(mouseX, mouseY) //Drawing gesture
}

private function end(e:MouseEvent):void
{
	stage.removeEventListener(MouseEvent.MOUSE_MOVE, check);
	t.text = gesMan.tracer().toString()
}
}

Step 11: Clean Up gestCurrent of Unnecessary Strokes

Cleaning up strokes in current gesture is important, as there are:

1. Strokes that make difficult detection of gestures.
2. Strokes that are duplicated
3. Strokes that are invalid

I have divided the clean-up job into two phases, specifically dropStrokes and dropDuplicates. dropStrokes eliminates strokes that made it difficult to detect of gestures. (This feature will be explained in detail in Part 3 when we detect alphabetical letters.) dropDuplicates eliminates duplicates and invalid strokes.

dropStrokes will only keep strokes in gestCurrent according to the third paramenter, onlyStrokes, in register. It is defined when we register the gesture sequences initially. For further understanding, again, watch out for Part 3. But for now, just keep it at the back of your head. In fact, this tutorial does not make use of this feature just yet.

/**
* Method to rid irrelevant strokes based upon condition
* @param strokesCondition
* @return Vector of 
*/
private function dropStrokes(inputArray:Vector.<int>, strokesCondition:int):Vector.<int> {
	var xDiagonals:Vector.<int> = inputArray.slice(0,inputArray.length);

	//Keep mains only
	if (strokesCondition == 1) {
		for (var i:int = 0; i < xDiagonals.length; i++) {
			if (xDiagonals[i] > 3) {
				xDiagonals.splice(i, 1);
				i--
			}
		}
	}

	else if (strokesCondition == 2) {
		for (var j:int = 0; j < xDiagonals.length; j++) {
			if (xDiagonals[j] <4) {
				xDiagonals.splice(j, 1);
				j--
			}
		}
	}

	return xDiagonals
}

Step 12: Clean Up gestCurrent

The second phase of clean up is to rid the duplicate and invalid strokes as mentioned in Step 2. The implementation is written below.

I've placed a Flash presentation above to ease your understanding of the algorithm. Make a right gesture using mouse to go to next frame, make a left to go back to previous frame. Up gesture brings you to the final frame, down gesture brings you to the first frame.

/**
* Method to rid duplicates and possible mistaken gestures
* @param minDuplicates Repetition to be considered high repetition
* @return Vector array w/o duplicates and possible mistaken gestures
*/
private function dropDuplicates (inputArray:Vector.<int>, minDuplicates:int = 1):Vector.<int> {

	//Append end of line indicator to integer array
	var xDuplicates:Vector.<int> = inputArray.slice(0,inputArray.length);
	xDuplicates.push( -1);

	//Initiate variables
	var count:int = 1;
	var keepIndex:int = 0;

	//Ripping duplicates and invalid moves
	for (var i:int = 0; i < xDuplicates.length - 1; i++) { 
		if (xDuplicates[i] == xDuplicates[i + 1]) count ++
		else{
			if (count > minDuplicates) { 
				xDuplicates.splice(keepIndex, count - 1)
				keepIndex++
				i = keepIndex
				count = 1
			}
			else xDuplicates.splice(keepIndex, minDuplicates)
			i--
		}
	}

	xDuplicates.splice(xDuplicates.length - 1, 1);
	return xDuplicates;
}

Step 13: Check for a Match

After trimming down gesCurrent, our next task is to find a proper match in the set of predefined gesture sequences. We can verify match between current gesture and a member of predefined gestures using checkMatch. Input the index of gestSeq you would like to check match with gestCurrent. A return value of -1 indicates no match; the index of gestSeq is matched otherwise.

/**
* Method to check if gestCurrent matches with gestSeq
* @param index Member of gestSeq
* @return Match (1) or fail (-1)
*/
private function checkMatch(index:int):int {
	//Pessimistic search, assumes not found initially
	var matched:int = -1;

	//clean input first
	var cleaned:Vector.<int> = dropStrokes(gestCurrent, strokes[index]); trace(cleaned, " xstrokes");
	cleaned = dropDuplicates(cleaned);trace(cleaned," xduplicates")

	//Check only those of same length
	if (cleaned.length == gestSeq[index].length) {
		var counter:int = cleaned.length;

		//Scroll through each integer of selected predefined gesture
		for (var member:int = 0; member < cleaned.length; member++) 
		{
			if (cleaned[member] == gestSeq[index][member]) counter--;
			else break;
		}

		//if all integers matched, current index of gestSeq matches gestCurrent
		if (counter == 0 ) matched = index;
	}

	return matched;
}

Step 14: Output the Result

Finally, we can output the result. The result will be based upon whether _order is turned on or not. We check match between gestCurrent with current gestSeq only if _order is turned on, otherwise we will need to search for match with any member in gestSeq.

Regardless of the search outcome, we need to output the result. Search result will be -1 if there's no successful match; if there's a match, an array of stroke index coupled with its name will be returned. The result is -2 if gestSeq is empty.

/**
* Method to evaluate gesture according to preferred approach (in sequential order or not)
* @return No predefined gestures (-2), No match of gesture (-1), array with index of match and its description
*/
public function end():Array {
	//Pessimistic search, -2 indicates that there are no predefined gesture
	var result:Array = [ -2, "No predefined gestures"];

	//If there are predefined gestures
	if (gestSeq.length > 0) {

		//Find the match in order
		if (_order) {
			if (checkMatch(orderCurrent) > -1) {
				result = [orderCurrent, gestName[orderCurrent]];
				orderCurrent++;
			}
		}

		//Find a match in no specific order
		else if (!_order) {
				//Scroll through all gestures
			for (var i:int = 0; i < gestSeq.length; i++) 
			{
				if (checkMatch(i) > -1) {
					result = [i, gestName[i]];
				}
			}
		}

		//Handle when cant find
		else {
			result = [ -1, "Gesture is invalid."]
		}
	}
	return result
}

Step 15: Prepare Graphical Assets in Flash

With Step 14 we completed GManager. We need to create graphical assets next. I have created the graphical assets in Flash Pro and will export them in .swc format. Press Ctrl + Shift + F12 to pop the Publish Setting window out in Flash and check "Export SWC" under Flash tab. I've made my assets available for download but feel free to develop your own.

Step 16: Importing Assets Into FlashDevelop

Bring your SWC into FlashDevelop's lib folder as shown above. Create a class named "Main2" in FlashDevelop and instantiate the MovieClip that holds the graphics. I've highlighted the relevant ActionScript below.

private var ges:MGesture;
private var gesMan:GManager;
private var a:Arrows;
private var seq:Boolean = false;

public function Main2() 
{
	//Importing graphics in form of movieclip
	a = new Arrows();
	a.stop();
	a.x = 50;
	a.y = 50;
	addChild(a);

Step 17: Register Corresponding Gestures

We shall register our gesture sequences in the order that graphics are arranged in our published MovieClip. We do so to make easy scrolling of frames once the result has been attained. Note as well that I have highlighted line 33 of the implementation; you may uncomment this to in my source file to see matching done in order.

public function Main2() 
{
	//Importing graphics in form of movieclip
	a = new Arrows();
	a.stop();
	a.x = 50;
	a.y = 50;
	addChild(a);
	
	//initiating MGesture to allow singular gesture detection
	ges = new MGesture(stage);
	
	//initiating GManager to allow sequential gesture detection
	gesMan = new GManager();
	//this.seq = true;
	
	//Detecting gestures in sequence
	if (seq)	{
		a.gotoAndStop(2);
		gesMan.useSeq = true;
	}
	
	//Register gesture sequences in line with frames
	//Register gesture sequences in line with frames
	gesMan.register(new < int > [2, 1], "2 Stroke Down, CCW");
	gesMan.register(new < int > [1, 0], "2 Stroke Right, CCW");
	gesMan.register(new < int > [0, 3], "2 Stroke Up, CCW");
	gesMan.register(new < int > [3, 2], "2 Stroke Left, CCW");
	gesMan.register(new < int > [3, 2, 1], "3 Stroke Down, CCW");
	gesMan.register(new < int > [2, 1, 0], "3 Stroke Right, CCW");
	gesMan.register(new < int > [1, 0, 3], "3 Stroke Up, CCW");
	gesMan.register(new < int > [0, 3, 2], "3 Stroke Left, CCW");
	
	gesMan.register(new < int > [3, 0], "2 Stroke Right, CW");
	gesMan.register(new < int > [0, 1], "2 Stroke Down, CW");
	gesMan.register(new < int > [1, 2], "2 Stroke Left, CW");
	gesMan.register(new < int > [2, 3], "2 Stroke Up, CW");
	gesMan.register(new < int > [2, 3, 0], "3 Stroke Right, CW");
	gesMan.register(new < int > [3, 0, 1], "3 Stroke Down, CW");
	gesMan.register(new < int > [0, 1, 2], "3 Stroke Left, CW");
	gesMan.register(new < int > [1, 2, 3], "3 Stroke Up, CW");
	
	gesMan.register(new < int > [0, 3, 0, 3, 0],"Step Up");
	gesMan.register(new < int > [0, 1, 0, 1, 0], "Step Down");
	gesMan.register(new < int > [4], "South East");
	gesMan.register(new < int > [5], "South West");
	gesMan.register(new < int > [6], "North West");
	gesMan.register(new < int > [7], "North East");
	gesMan.register(new < int > [4, 5, 4, 5], "Zigzag");
	
	stage.addEventListener(MouseEvent.MOUSE_DOWN, start);
	stage.addEventListener(MouseEvent.MOUSE_UP, end);
}

Step 18: Handling Results

Recording of gesture sequence follows the same approach as CheckOut2. I assume readers have gone through Step 10. So finally, we shall output the result. Upon successful match (> -1), we shall scroll to the appropriate frame. Again, this is subject to whether you have enabled the use of sequence by turning on seq or not.

private function end(e:MouseEvent):void 
{
	stage.removeEventListener(MouseEvent.MOUSE_MOVE, check);
	
	//evaluate gesture and output result
	var output:Array = gesMan.end()
	if (output[0] > -1)	{
		if (seq)a.nextFrame();
		else	a.gotoAndStop(output[0] + 2);	//offset needed as first frame is empty.
	}
}

Step 19: Publish Your Project

Press Ctrl + Enter to publish your project. Gesture with your mouse and what was detected. I've included both projects (one requests you to match a given gesture; the other displays the gesture you drew, if it matches one it knows). Have fun.

Step 20: Application Development

Inaccuracies in user gestures are common, and any application that incorporates gestures should consider this. In the next part of this tutorial, I shall attempt to use MGesture and GManager to develop an alphabet recognition application. I shall point out several details to tweak accordingly to improve gesture detection.

Conclusion

You should now find it easy to develop applications that detect gesture sequences of any combination. Hope this tutorial has helped you in some ways. Let me know of your comments, queries and bugs encountered. Terima kasih.