My primary concern is still software for collaboration – two or more people working within the same space to achieve a common objective. In this trial, I consider two people finger-painting on the same screen canvas. Each person is able to use a different tool and colour combination and both can draw at the same time.
For finger painting, a sequence of events for a user to draw a line on the screen canvas might be:
- touch drawing tool to be used
- touch colour to be used
- touch finger to paper to start line
- drag finger to draw line.
On a single user touch system, this action is mimicked  by putting the canvas into a drawing mode: the user selects a paint tool and colour and that tool and colour is active on the canvas for all subsequent drawing touches. If this modal canvas is used on a multi-user touch system, all users are forced to draw with the same tool and colour.
To achieve the desired goal of multiple people drawing on the same canvas at the same time with different tool and colour combinations, the tool and colour information must be associated with individual fingers.
The sequence of tool and colour selection followed by drawing cannot be easily achieved on a per-finger basis with a standard multi-touch screen because a finger is only tracked while it is in contact with the screen. It is not possible for one touch sequence to be associated with previous touch sequences, particularly when there are several users involved.
In this trial I used a Leap Motion to track finger movements in air so that the sequence of drawing events could be linked together. In this way, a tracked finger can accumulate the various drawing attributes like tool, colour and size and then be positioned over the paper for line drawing.
The initial intent was to position the Leap Motion in front of a multi-touch screen such that the screen could be used for selection and drawing and the leap Motion for air tracking. Calibrated software would provide the link between the two types of tracking.
In the event, I found that the Leap Motion did not work well next to the screen and so I concentrated on the air tracking and used a standard screen.
I created a Leap Motion plug-in for my existing touch framework to use the raw finger position feed from the Leap Motion SDK. Â The leap motion sensor is placed in front of the computer screen and the plug-in software creates a vertical touchdown plane in the air above it.
As the finger approaches the plane, a dashed circle appears on the screen. This circle is centered on the finger x,y  position and shows where the user’s finger is in relation to screen objects. The diameter of the circle reduces as the finger gets closer to the plane, so guiding the user into the object they wish to touch.
A touchdown is signaled as the finger crosses the plane and is maintained while the finger is at or behind the plane.
I created some special controls to take advantage of in-air tracking:
- a tool picker giving access to pen, circle rectangle and line
- a colour palette with a selection of red, green, blue and alpha together with a tool to allow mixing and storage of colours
- a paint canvas
To pick a tool, touch the associated icon in the tool picker. Â The tool will remain associated with the finger while the finger is tracked. The tool icon appears on the dashed circle that indicates that the finger is being tracked in the air above the touchdown plane.
To pick a colour, touch the colour in the palette. The amount of colour can be dialled up by stirring the finger in the colour (to simulate picking up more colour with a paint brush). The colour remains associated with the finger while the finger is tracked. Stirring the finger in other colours or alpha allows a selection of colours and transparencies to be created.
This stirring motion can also be used with the pen and line tools to change the thickness of the line drawn.
To draw, move the tracked finger to the canvas, touchdown and drag to produce the required effect.
Moving the finger away from the touchdown plane such that it is no longer tracked in air will cause the selected tool and its attributes to be forgotten.