Notes from a Leap Motion Presentation, part 3: Socket to me

This is part three of a write-up based on a recent presentation I gave showing off the Leap Motion Controller.

You can read part one here

You can read part two here

We interrupt this program …

I had planned on continuing by showing the code for my simple browser-based game, focusing on how it talks to the WebSocket server provided by the JRuby Leap app described in part two.

I’ll do that, because it’s really really simple, but I will also go over some other Leap WebSocket stuff because I’ve since seen some Leap-in-the-browser demos that are quite impressive. Plus there exists a Leap JavaScript library that makes dealing with the raw Leap JSON data easier.

Actually, I know of at least two such libraries. The first one I came across is by James Blaha , who runs the very interesting Leap Gamer site. It is a Leap Motion/jQuery integration plug-in

The second JavaScript library is from Leap Motion. It’s their Leap JavaScript framework, LeapJS

I have not used jQuery plug-in library. I have played with the demos that are part of the LeapJS repo. Very cool. My original concerns over speed seem unfounded.

The plan is to look at the original game, and then a version that goes straight to the Leap for data.

The Game

The game has two characters, Skullhead and Critter.

You control the location of Skullhead by using a single finger to place where you want it to move. The critter will come and go at a random location; when visible it moves back and forth along a line. Your job is to move Skullhead next to the critter and make yourself big and scary, enough to scare the critter away. When scared the critter turns green and disappears.

You make Skullhead grow by using two fingers; the distance between finger tips determines the amount of growth.

All of the hand commands are picked up by the JRuby program, which in turn broadcasts WebSocket messages.

Some code

The game is built using EaselJS, a very handy library for working with the HTML5 canvas element. My particular interest was in how easy it make working with sprites. I learned about EaselJS when I was looking around for a suitable HTML5/JavaScript game framework. There are quite a few. I took a liking to EasleS, I think, because I found this tutorial on writing a platformer game. EasleJS is part of CreateJS, which also happens to have the backing of Adobe. There’s also this Atari site.

In any event, based on that tutorial example and some hacking around on my own, I devised my own simple game. All the artwork and music is my own. You can grab code from the Leap Hacking GitHub repo

Some basics

This is not an EaselJS tutorial. Please see the MSDN game tutorial mentioned above for a start. However, I want to point out what at least is happening in my game.

The main page, index.html, loads the required JavaScript files and provides a canvas element for the game play. There’s a button as well that will actually start the game. All the game logic is then defined in js\main.js

This file first sets up a WebSocket connect when the page is loaded.

It’s pretty simple; it registers functions to be called whenever a message is received, when the socket is closed, and when the socket is opened. There is zero error handling here, nothing to tell the user that there is no socket connection. It’s demo code.

The onmessage function, and the basic code to support it, looks like this:

processWsEvent gets the event object, from which it can pull the message content. That gets parsed from JSON to a regular JavaScript object. The game assumes that message have a known structure: a command part that says what to do (“walkto” or “grow”) and an args part that holds an array of function arguments.

The function grabs that command part and uses it to dynamically invoke a corresponding method on a Handlers object, which defines the methods that do the actual calls into the game objects.

Having the game respond to new or different messages means adding or changing the functions defined for Handlers.

There are two things that you make happen: have Skullhead move to a new location on the game canvas, and grow bigger so as to scare away the critter.

The WebSocket messages are coming from a JRuby program that is reading the Leap data. It sends a “walkto” message when it detects a single finger, passing along the location of the finger-tip. (It remaps that first, though, since the it needs to be adjusted to fit within the game canvas dimensions.)

If it detects two fingers then it sends a “grow” message and the distance between the two finger tips. It scales that distance as the raw value is in millimeters and would be too large as-is.

That, in essence, is how the Leap (indirectly) controls the game. The rest of the code is just EaselJS game code, but I’ll show some of it because there are a few places where the use of WebSocket messages created some interesting conditions.

The EaselJS game tutorial showed a game that is played using the keyboard. That means that the game was unlikely to be handling a continuous series of movement request. Even if you held down one of the WASD or arrow keys to move it is nothing like the stream of messages coming over the WebSocket. However, if you are trying to handle a steady, fast stream of events and use them to trigger animation behavior you need t take some tings into account.

Movement and animation

Among other things, EaselJS makes it really easy to create and use sprites, or bitmap animations. You need to create (or get) an image file that has some number of “frames.” For example, this is the source image for Skullhead:

It contains 21 frames, each frame being 48 pixels square. While it is one single image, it defines two separate animations; an animation is simply a series of frames, and you get to decide what range of frames to use for what animation. (I created my animation images using Tile Studio and very nice, and free, sprite and tile editor for Windows. It has some handy features for making images that are variations of one another, and will export you sequence to a single image.) It is a PNG file, with a transparent background (here shown as pink).

The MSDN game tutorial explains (somewhat) about creating animations for use in a game. Here’s the code for creating the Skullhead animation from that image; it is in a function that gets called when the base sprite image is loaded:

Side note: I started from code glommed from the MSDN tutorial and mucked about some as I explored. I ended up with a working game, and that was fine, but decided that the were perhaps too may loose functions and vars floating around. I took a stab at name-spacing things so that top-level code could, for example, call Skullhead.move() and not some direct, lower-level function. Overall I’m happier but it may not be as clean as it could be. (In version two of the app I re-did this, to better results.)

This function takes the source image file and uses it to create a SpiteSheet which in turn is used to create a BitmapAnimation.

The SpriteSheet creation is where you pass along the image as well some details on how each animation is defined. You tell it how big a frame is, what frame location should be used as the reference point, and what frame ranges are to be assigned to what animations.

I defined two animations: walk, which is frames 0 to 8, and grow, for frames 9 through 20. That last number in each animation definition adjusts the frame rate.

For the BitmapAnimation I assigned a few properties (name, movement velocity) and set the scale. Each frame is 48 pixels square but I want to render it larger so I scale it up by 2.0.

The animation is then added to the stage (which is basically the canvas element) and told to execute the walk animation.

The last two lines are one place I needed to account for the use of WebSocket messages in place of keyboard input.

Ordinarily such a game would use a “clock tick” of some kind to determine game state and update the stage as needed. In fact this game does have that; there is a tick() function that controls a few things, including the appearance and movement of the critter sprite. The movement of Skullhead is also tied to this clock tick, but whereas one might normally move a sprite by indicating (via the keyboard) forward, or back, or jump, in this game the movement is based on trying to reach some desired X and Y point.

The result is a mix of tick-based movement and targeted location. The use of a single clock makes things simpler; movement is based on increments based on a few rules, and it all happens at the same time. When a “walkto” message arrives from the WebSocket, the desired X and Y for Skullhead gets updated. On each tick, the code attempts to move Skull to that point. There’s some simple math there to make traveling larger distances a little faster, slowing down as the desired location approaches.

I discovered a quirk, though, with this. (Or a bug, depending on your point of view). Each call to Skullhead.walkTo triggers the execution of the walk animation. Since the WebSocket server is sending these message fast and furious this animation is constantly restarted. The result is that the animation rarely gets the change to play all the way through, and you end up seeing a sort of jittery image instead.

I was OK with that for the walking part, but when it came to to grow Skullhead to scary proportions it didn’t play so well. That animation is more lively, and seeing just the first two frames or so flicker over and over lost some of the fun.

To account for this I add a time check:

Skullhead.grow() is passed a scaling factor (it’s passed as an array for now, though only the first value is used, for whatever mysterious reason I had at the time). This value is used to render the animation larger than the default size, and comes from the finger-tip distance detected by the Leap.

As with walking, invoking the grow gesture results in a slew of similar messages. The scaling factor will change, and that’s useful, so it gets applied by this function. However, to avoid constantly restarting the animation the function tracks the time since the last invocation. It’s set to one second, which is enough time for the animation to loop at least once.

A few other things to wrap up this part

So, tick() is called, and a few things happen. SKullhead gets moved towards whatever is the desired location, based on the most recent WebSocket messages. The critter is shown or hidden based on some set clock-tick counts. That is, it will only be on screen for a fixed number of ticks, then disappear for some other fixed number of ticks, back and forth. When it is first show it is randomly placed, and proceeds to walk back and forth. Your job, guiding Skullhead, is to get right up to the critter and grow big and scary. If you do this before the critter disappears the critter should turn green, make a noise, then run off. At that point the critter show/hide counts are reset, and the critter before long will reappear someplace else.

There’s no score kept. You’ll know in your heart, though, just how good you are.

Version 2: Leap Straight-up

This next version uses the leap.js library from the Leap Motion devs. The GitHub repo includes a number of sample apps that are useful in understanding how to use the library, but there is no formal documentation for it.

The README describes using the code with node.js but you can also use it straight from the browser (and that’s how the various examples use it).

leap.js provides a wrapper over the raw JSON that comes from the Leap WebSocket. The API is very much like the standard SDK API but it’s not identical. It makes it easy to get hand and finger data, for example, but certain details, such as the distance between finger tips, you have to code yourself.

I did that, with the help of some code that was in one of the examples that did some Vector math.

What’s different

First off, I re-did the namespacing of the sprites because, well, the first way was clumsy and basically wrong. It works, so great, but a lot of it was hacky. Consider it an example of running with the wrong idea.

More important, this version does not deal directly with the WebSocket. That is handled by the leap.js library. That mean having to change the Handler code. But, other than some code re-org, the game stayed pretty much the same.

Getting Leap events

In order to get Leap data you need to create a Leap.Controller instance. I put this into a function that also sets up “region.” A region (or, specifically, a Leap.UI.Region), defines the 3D area that the Leap sensor will pay attention to. As we’ll see a bit later the region also provides a method to map its coordinates to screen coordinates, so that, for example, the left edge of the region bounding box corresponds to the left edge of the game canvas (and likewise for the right edge, and top and bottom).

In playing with this I found that some values for the region bounding box worked better than others. A smaller box meant the user would not have to move their fingers very far, but it made it harder to be precise. Larger numbers seemed to fail at the edges; I’m not sure why this was, it may a calibration matter.

I added some form fields to the page holding the game canvas so that I could more easily try out different values, hence the code in this method that is using jQuery to pull values and parse them into integers.

This line is interesting:

controller.addStep(new Leap.UI.Cursor())

From looking at the leap.js code is appears that addStep inserts functions into a processing pipeline. As each frame of data is acquired it is passed to each of these pipeline functions. I grabbed some of code straight from th UI.html example in the leap.js repo. The Cursor thing provides a way to get a general location for whatever “pointables” have been detected, though for my purposes I could also just as well have used the tip location from a single finger.

Since setting all this up requires access to canvas I also moved around some of the original setup code. I’m still not quite happy with the whole initialization process. In particular there is nothing in place to reliably ensure that all the game assets and resources are in place, properly prepped, and ready to roll. I’ve been running all of this from a local server so I’ve not bothered to consider network latency as the page loads. There are ways to do this (CreateJS offers a some asset management stuff) but it’s not here.

So the game code goes along pretty much as it was, except at the end of the startGame() method it kicks off the Leap controller event loop:

The game updating mechanics are still the same as in the first version. createjs.Ticker is invoking tick() every 60 seconds. Meanwhile, in tandem, this controller loop is handing frame events (much as what happens in the JRuby code).

In the first version the code was taking the WebSocket event and parsing the raw JSON messages, converting them into Handlers invocations. Here, there’s no JSON to be dealt with. I changed some of the code in the Handlers functions to deal with the data more directly.

As with the JRuby game code, if one finger is detected then the finger-tip location is used to direct the placement of Skullhead. I went with frame.cursorPosition partly because that’s just what I copy-and-pasted from the sample code, and partly because it was curious about how it worked. (In fact, in this case, it just makes for extra computational overhead as the underlying code goes and re-checks how many pointables are there and then selects one.)

By the way, the v1.distanceFrom(v2) bit is available because I scavenged some code from the “Earth Rotation” example, which appears to have grabbed this particular code from the Sylvester library. Very handy, and I’m sort of surprised the default leap.js did not have better Vector operations available.

The Handlers functions are the basically same, so the actual placement of Skullhead is handled (indirectly) by tick() and the step-wise relocation of the sprite based on desiredX and desiredY.

Those values are assigned from the x and y of frame.cursorPosition as re-mapped by region.mapToXY. This ensures that the location is always within the game canvas bounds.

While converting the code and playing around with it I gave some thought to having Skullhead placed at whatever point we get (after remapping) from frame.cursorPosition. I’m sure I will do that in the future, either in yet another version of “Skullhead v. Critter” or a new game. However, the current method introduces an interesting lag to the movement, making it slightly harder. I like that.

Wrapping up

There’s more to leap.js; there’s more to the Leap SDK, too, as recent versions have introduced some gesture detection. I’ll be writing about all that in the near future, as well as developing more (and better) browser games. But this, I think, wraps up the topics I touched in my recent show-and-tells.

comments powered by Disqus