1. Code
  2. ActionScript

How to Add Mouse Gesture Control to Your Flash Projects: Multi-Stroke Gestures


In the first part of this series, I introduced a class to handle single mouse stroke detection: MGesture. This tutorial takes it a step further, by detecting a sequence of strokes.

We'll be using the class GManager for this, in tandem with MGesture. After briefing through GManager functionality, an application demonstrating its use will be developed.

Final Result Preview

Let's take a look at the final result we will be working towards. To use the Flash presentation below, use your mouse to perform the gesture as indicated by the arrow in the top left corner. Gesture by pressing the left mouse button, moving the mouse while holding the button, and then releasing it.

Step 1: Recap Direction Numbering

Numbers on different directions

First step, a revision on integers used to represent different directions from Part 1. Hopefully, you can form a mental picture of the diagram above as we shall refer to it heavily throughout this tutorial.

Step 2: Strokes Detected

As user makes a gesture by any pointing devices (mouse, tablet pen, etc), the vectors successfully detected over time actually form a series of integers. Refer to diagram below.

Examples of gesture sequence coding

Notice the unique integers in the sequence are 0, 4, 1. However, I shall say unique integers are 0, 1. Lets face it: it's difficult to make an accurate stroke on my Bamboo tablet, and even more so with a mouse. Errors are inevitable. Our algorithm should then be based upon the high repetition of integers in a continuous string. They are 0, 1 here - 4 is probably due to inaccurate stroke. Therefore, this gesture can be uniquely identified as a sequence of 0, then 1.

I've included another few gestures with their accompanying unique sequences below.

Examples of gesture sequence coding

Step 3: Introducing GManager

The purpose of this class is simple:

  1. To register unique stroke sequences to compare against.
  2. To capture sequence detected of current gesture made by user.
  3. To compare sequence in (2) against all stroke sequences in (1).
  4. To return search result.

Developer has to define a set of stroke sequences to detect initially. Once done, event listeners to capture user gesture at runtime can be programmed. If the current gesture's sequence successfully matches any one sequence in the predefined set, different operations can then be carried out.

Step 4: Variables

GManager has the following variables:

Variable Datatype Purpose
gestSeq Vector.<Vector.<int>> 2D array to record different stroke sequences.
gestName Vector.<String> 1D array to record names of stroke sequences.
strokes Vector.<int> 1D array to record restriction on gestCurrent's strokes when compared with stroke sequence.
gestCurrent Vector.<int> 1D array to record current stroke sequence.
_order Boolean Determines whether gesture sets in gestSeq should be detected in order.
orderCurrent int Current sequence to detect whether _order is turned on; starts at 0.

Step 5: Methods

Methods in GManager are as below:

Method Input Output Description
GManager void void Class initiation, gestSeq and gestName, strokes initialised
register Vector.<int>, String, int void Register stroke sequence, name and stroke restriction (optional).
remove int void Remove selected stroke sequence
removeAll void void Remove all registered stroke sequences
start void void Prepare variable to record current stroke sequence, gestCurrent
populate int void Populate gestCurrent with detected strokes (singular)
tracer void Vector.<int> Traces and outputs gestCurrent for debugging purposes.
dropStrokes Vector.<int>, int Vector.<int> Eliminate unnecesary strokes in gestCurrent. Keep mains and diagonals (0), keep only mains (1), keep only diagonals (2). (Part 3 will further explain its use.)
dropDuplicates Vector.<int>, int Vector.<int> Identify valid unique integer in current sequence and drop
duplicates. Input minimum duplicates (apart from self) to be considered valid stroke
checkMatch int int Checks if member of gestSeq matches with gestCurrent
end void Array Returns result of search. Result is Array with index of matched
sequence in gestSeq and its name in gestName

Properties of GManagerare as below:

Property Accessor Types Purpose
useOrder Getter/ Setter Gets and sets whether gestSeq is to be detected in order
length Getter only Gets the length of gestSeq

Step 6: Register Predefined Stroke Sequences

As mentioned, the first step is to register unique stroke sequences that the entered gesture can be compared against. The constructor initates variables needed (highlighted) and methods that follow registers into and removes from predefined stroke sequences.

Step 7: Properties

Below are the properties specifically for set holding stroke seqeunces, gestSeq:

Step 8: Record Current Stroke Sequence

After registering stroke sequences to detect against, we can prepare gestCurrent and record consequtive valid singular strokes into it. Methods below will provision for it. Tracer is a method to trace the current gesture's sequence.

Step 9: Looking into gestCurrent

Examples of gesture sequence coding

By now, we can already write a program to check out current gesture's sequence as mentioned in Step 2. You may download the source and look at the class CheckOut2. Turn on your Output panel in FlashDevelop to view the sequence of integers committed by your current gesture.

I've also included a demo of it below and placed a TextField to display the sequence instead of the Output panel in FlashDevelop.

Step 10: Implementation of CheckOut2

I have included source code of checkOut2 as below. Its not tough to understand the logic provided you have understood how event handlers are assigned. (Read this tutorial to find out more about event handlers.) I've also highlighted important points where GManager 's methods are being used.

Step 11: Clean Up gestCurrent of Unnecessary Strokes

Cleaning up strokes in current gesture is important, as there are:

  1. Strokes that make difficult detection of gestures.
  2. Strokes that are duplicated
  3. Strokes that are invalid

I have divided the clean-up job into two phases, specifically dropStrokes and dropDuplicates. dropStrokes eliminates strokes that made it difficult to detect of gestures. (This feature will be explained in detail in Part 3 when we detect alphabetical letters.) dropDuplicates eliminates duplicates and invalid strokes.

dropStrokes will only keep strokes in gestCurrent according to the third paramenter, onlyStrokes, in register. It is defined when we register the gesture sequences initially. For further understanding, again, watch out for Part 3. But for now, just keep it at the back of your head. In fact, this tutorial does not make use of this feature just yet.

Step 12: Clean Up gestCurrent

Swipe left-to-right to go to the next frame.

The second phase of clean up is to rid the duplicate and invalid strokes as mentioned in Step 2. The implementation is written below.

I've placed a Flash presentation above to ease your understanding of the algorithm. Make a right gesture using mouse to go to next frame, make a left to go back to previous frame. Up gesture brings you to the final frame, down gesture brings you to the first frame.

Step 13: Check for a Match

After trimming down gesCurrent, our next task is to find a proper match in the set of predefined gesture sequences. We can verify match between current gesture and a member of predefined gestures using checkMatch. Input the index of gestSeq you would like to check match with gestCurrent. A return value of -1 indicates no match; the index of gestSeq is matched otherwise.

Step 14: Output the Result

Finally, we can output the result. The result will be based upon whether _order is turned on or not. We check match between gestCurrent with current gestSeq only if _order is turned on, otherwise we will need to search for match with any member in gestSeq.

Regardless of the search outcome, we need to output the result. Search result will be -1 if there's no successful match; if there's a match, an array of stroke index coupled with its name will be returned. The result is -2 if gestSeq is empty.

Step 15: Prepare Graphical Assets in Flash

With Step 14 we completed GManager. We need to create graphical assets next. I have created the graphical assets in Flash Pro and will export them in .swc format. Press Ctrl + Shift + F12 to pop the Publish Setting window out in Flash and check "Export SWC" under Flash tab. I've made my assets available for download but feel free to develop your own.

Create graphics in Flash Ide
Linkage panel opened
Export for Actionscript
Publish SWC

Step 16: Importing Assets Into FlashDevelop

Publish SWC

Bring your SWC into FlashDevelop's lib folder as shown above. Create a class named "Main2" in FlashDevelop and instantiate the MovieClip that holds the graphics. I've highlighted the relevant ActionScript below.

Step 17: Register Corresponding Gestures

We shall register our gesture sequences in the order that graphics are arranged in our published MovieClip. We do so to make easy scrolling of frames once the result has been attained. Note as well that I have highlighted line 33 of the implementation; you may uncomment this to in my source file to see matching done in order.

Step 18: Handling Results

Recording of gesture sequence follows the same approach as CheckOut2. I assume readers have gone through Step 10. So finally, we shall output the result. Upon successful match (> -1), we shall scroll to the appropriate frame. Again, this is subject to whether you have enabled the use of sequence by turning on seq or not.

Step 19: Publish Your Project

Press Ctrl + Enter to publish your project. Gesture with your mouse and what was detected. I've included both projects (one requests you to match a given gesture; the other displays the gesture you drew, if it matches one it knows). Have fun.

Step 20: Application Development

Inaccuracies in user gestures are common, and any application that incorporates gestures should consider this. In the next part of this tutorial, I shall attempt to use MGesture and GManager to develop an alphabet recognition application. I shall point out several details to tweak accordingly to improve gesture detection.


You should now find it easy to develop applications that detect gesture sequences of any combination. Hope this tutorial has helped you in some ways. Let me know of your comments, queries and bugs encountered. Terima kasih.

Looking for something to help kick start your next project?
Envato Market has a range of items for sale to help get you started.