Unlimited Plugins, WordPress themes, videos & courses! Unlimited asset downloads! From \$16.50/m

How to Add Mouse Gesture Control to Your Flash Projects: Multi-Stroke Gestures

Difficulty:IntermediateLength:LongLanguages:

In the first part of this series, I introduced a class to handle single mouse stroke detection: `MGesture`. This tutorial takes it a step further, by detecting a sequence of strokes.

We'll be using the class `GManager` for this, in tandem with `MGesture`. After briefing through `GManager` functionality, an application demonstrating its use will be developed.

Final Result Preview

Let's take a look at the final result we will be working towards. To use the Flash presentation below, use your mouse to perform the gesture as indicated by the arrow in the top left corner. Gesture by pressing the left mouse button, moving the mouse while holding the button, and then releasing it.

Step 1: Recap Direction Numbering

First step, a revision on integers used to represent different directions from Part 1. Hopefully, you can form a mental picture of the diagram above as we shall refer to it heavily throughout this tutorial.

Step 2: Strokes Detected

As user makes a gesture by any pointing devices (mouse, tablet pen, etc), the vectors successfully detected over time actually form a series of integers. Refer to diagram below.

Notice the unique integers in the sequence are 0, 4, 1. However, I shall say unique integers are 0, 1. Lets face it: it's difficult to make an accurate stroke on my Bamboo tablet, and even more so with a mouse. Errors are inevitable. Our algorithm should then be based upon the high repetition of integers in a continuous string. They are 0, 1 here - 4 is probably due to inaccurate stroke. Therefore, this gesture can be uniquely identified as a sequence of 0, then 1.

I've included another few gestures with their accompanying unique sequences below.

Step 3: Introducing `GManager`

The purpose of this class is simple:

1. To register unique stroke sequences to compare against.
2. To capture sequence detected of current gesture made by user.
3. To compare sequence in (2) against all stroke sequences in (1).
4. To return search result.

Developer has to define a set of stroke sequences to detect initially. Once done, event listeners to capture user gesture at runtime can be programmed. If the current gesture's sequence successfully matches any one sequence in the predefined set, different operations can then be carried out.

Step 4: Variables

`GManager` has the following variables:

 Variable Datatype Purpose `gestSeq` `Vector.>` 2D array to record different stroke sequences. `gestName` `Vector.` 1D array to record names of stroke sequences. `strokes` `Vector.` 1D array to record restriction on `gestCurrent`'s strokes when compared with stroke sequence. `gestCurrent` `Vector.` 1D array to record current stroke sequence. `_order` `Boolean` Determines whether gesture sets in `gestSeq` should be detected in order. `orderCurrent` `int` Current sequence to detect whether `_order` is turned on; starts at 0.

Step 5: Methods

Methods in `GManager` are as below:

 Method Input Output Description `GManager` void void Class initiation, `gestSeq` and `gestName, strokes` initialised `register` `Vector., String, int` void Register stroke sequence, name and stroke restriction (optional). `remove` `int` void Remove selected stroke sequence `removeAll` void void Remove all registered stroke sequences `start` void void Prepare variable to record current stroke sequence, `gestCurrent` `populate` `int` void Populate `gestCurrent` with detected strokes (singular) `tracer` void `Vector.` Traces and outputs `gestCurrent` for debugging purposes. `dropStrokes` `Vector., int` `Vector.` Eliminate unnecesary strokes in `gestCurrent`. Keep mains and diagonals (0), keep only mains (1), keep only diagonals (2). (Part 3 will further explain its use.) `dropDuplicates` `Vector., int` `Vector.` Identify valid unique integer in current sequence and drop duplicates. Input minimum duplicates (apart from self) to be considered valid stroke `checkMatch` `int` `int` Checks if member of `gestSeq` matches with `gestCurrent` `end` void `Array` Returns result of search. Result is `Array` with index of matched sequence in `gestSeq` and its name in `gestName`

Properties of `GManager`are as below:

 Property Accessor Types Purpose `useOrder` Getter/ Setter Gets and sets whether `gestSeq` is to be detected in order `length` Getter only Gets the length of `gestSeq`

Step 6: Register Predefined Stroke Sequences

As mentioned, the first step is to register unique stroke sequences that the entered gesture can be compared against. The constructor initates variables needed (highlighted) and methods that follow registers into and removes from predefined stroke sequences.

Step 7: Properties

Below are the properties specifically for set holding stroke seqeunces, `gestSeq`:

Step 8: Record Current Stroke Sequence

After registering stroke sequences to detect against, we can prepare `gestCurrent` and record consequtive valid singular strokes into it. Methods below will provision for it. `Tracer` is a method to trace the current gesture's sequence.

Step 9: Looking into `gestCurrent`

By now, we can already write a program to check out current gesture's sequence as mentioned in Step 2. You may download the source and look at the class `CheckOut2`. Turn on your Output panel in FlashDevelop to view the sequence of integers committed by your current gesture.

I've also included a demo of it below and placed a `TextField` to display the sequence instead of the Output panel in FlashDevelop.

Step 10: Implementation of `CheckOut2`

I have included source code of `checkOut2` as below. Its not tough to understand the logic provided you have understood how event handlers are assigned. (Read this tutorial to find out more about event handlers.) I've also highlighted important points where `GManager` 's methods are being used.

Step 11: Clean Up `gestCurrent` of Unnecessary Strokes

Cleaning up strokes in current gesture is important, as there are:

1. Strokes that make difficult detection of gestures.
2. Strokes that are duplicated
3. Strokes that are invalid

I have divided the clean-up job into two phases, specifically `dropStrokes `and `dropDuplicates`. `dropStrokes `eliminates strokes that made it difficult to detect of gestures. (This feature will be explained in detail in Part 3 when we detect alphabetical letters.) `dropDuplicates `eliminates duplicates and invalid strokes.

`dropStrokes `will only keep strokes in `gestCurrent `according to the third paramenter, `onlyStrokes`, in `register`. It is defined when we register the gesture sequences initially. For further understanding, again, watch out for Part 3. But for now, just keep it at the back of your head. In fact, this tutorial does not make use of this feature just yet.

Step 12: Clean Up `gestCurrent`

Swipe left-to-right to go to the next frame.

The second phase of clean up is to rid the duplicate and invalid strokes as mentioned in Step 2. The implementation is written below.

I've placed a Flash presentation above to ease your understanding of the algorithm. Make a right gesture using mouse to go to next frame, make a left to go back to previous frame. Up gesture brings you to the final frame, down gesture brings you to the first frame.

Step 13: Check for a Match

After trimming down `gesCurrent`, our next task is to find a proper match in the set of predefined gesture sequences. We can verify match between current gesture and a member of predefined gestures using `checkMatch`. Input the index of `gestSeq` you would like to check match with `gestCurrent`. A return value of `-1` indicates no match; the index of `gestSeq `is matched otherwise.

Step 14: Output the Result

Finally, we can output the result. The result will be based upon whether `_order` is turned on or not. We check match between `gestCurrent` with current `gestSeq` only if `_order` is turned on, otherwise we will need to search for match with any member in `gestSeq`.

Regardless of the search outcome, we need to output the result. Search result will be `-1` if there's no successful match; if there's a match, an array of stroke index coupled with its name will be returned. The result is `-2` if `gestSeq` is empty.

Step 15: Prepare Graphical Assets in Flash

With Step 14 we completed `GManager`. We need to create graphical assets next. I have created the graphical assets in Flash Pro and will export them in .swc format. Press Ctrl + Shift + F12 to pop the Publish Setting window out in Flash and check "Export SWC" under Flash tab. I've made my assets available for download but feel free to develop your own.

Step 16: Importing Assets Into FlashDevelop

Bring your SWC into FlashDevelop's `lib` folder as shown above. Create a class named "Main2" in FlashDevelop and instantiate the MovieClip that holds the graphics. I've highlighted the relevant ActionScript below.

Step 17: Register Corresponding Gestures

We shall register our gesture sequences in the order that graphics are arranged in our published MovieClip. We do so to make easy scrolling of frames once the result has been attained. Note as well that I have highlighted line 33 of the implementation; you may uncomment this to in my source file to see matching done in order.

Step 18: Handling Results

Recording of gesture sequence follows the same approach as `CheckOut2`. I assume readers have gone through Step 10. So finally, we shall output the result. Upon successful match (> -1), we shall scroll to the appropriate frame. Again, this is subject to whether you have enabled the use of sequence by turning on `seq` or not.

Step 19: Publish Your Project

Press Ctrl + Enter to publish your project. Gesture with your mouse and what was detected. I've included both projects (one requests you to match a given gesture; the other displays the gesture you drew, if it matches one it knows). Have fun.

Step 20: Application Development

Inaccuracies in user gestures are common, and any application that incorporates gestures should consider this. In the next part of this tutorial, I shall attempt to use `MGesture` and `GManager` to develop an alphabet recognition application. I shall point out several details to tweak accordingly to improve gesture detection.

Conclusion

You should now find it easy to develop applications that detect gesture sequences of any combination. Hope this tutorial has helped you in some ways. Let me know of your comments, queries and bugs encountered. Terima kasih.