Chris Jay

Monday, February 19, 2007

Touch-screen displays on the web

Recently I saw a fabulous demo of touch-screen displays in action.

In the demo, the user is shown manipulating shapes on the screen with both hands - squeezing images, grabbing multiple shapes simultaneously and pushing them together, and simultaneous drag-and drop.

This reminded me of Steve Job's iPhone demo, and my comments at the time that HTML can probably handle multi-touch UIs, but javascript might struggle.

Experience tell us that developers need several different methods to handle user interaction. There should be a simple method with default behaviours, and a more detailed method giving fine control; and there should be a declarative approach for XML developers, and a procedural approach for those that prefer scripting.

It's clear that several things will have to change before websites cater appropriately for touch screen displays.

Firstly, we need more support in CSS for simple effects like drag and drop (our simple, declarative method).
Secondly, we need more declarative support for animation (fine-grained, declarative method).
Thirdly, if there is no mouse, there is no right mouse button - so we'll need to rethink the approach for context sensitive menus. Microsoft have innovated here with the new 'ribbon' interface in Office 2007 - I'll save this piece for another post.

CSS user interaction

CSS styles fit the bill perfectly for a simple, declarative approach. If we add a series of user interface CSS styles, the user gets a consistent experience, and the developer doesn't have to worry about endless code:

draggable = "no | yes" - elements with this style can be moved across the page via user interaction.
resizable ="none | x | y | preserveAspectRatio | all" - elements with this style can be re-sized via user interaction, along either or both axes.
zoomable = "no | yes" - elements with this style are containers (e.g. <html> or <div> tags) and zooming commands are available on the contents of the container.
pannable = "no | x | y | all" - elements with this style are containers (e.g. <html> or <div> tags) and panning commands are available on the contents of the container (e.g. panning around Google Maps). This could be scrollbars, or some other user interface method, depending on the browser.

For each of these styles, the exact user interaction method doesn't matter to the web developer - it could be a mouse, a touch screen, voice commands, or something else, as set by the browser or the operating system. In some cases (e.g. touch screen) there could be multiple user interactions at the same time; that's all handled by the browser. All the web developer need care about is setting the appropriate styles.

Declarative animation

Anyone who's tried to program drag and drop knows that the DOM is painfully awkward at tracking certain user interactions - but imagine dragging two objects on a touch screen simultaneously! Which event object would you use?

The real pain here is for events like mousemove. These are "continuous events", a contradiction in terms which reveals the flaw in the underlying approach. For continuously evolving features, languages should use Functional Animation instead (see my previous post).

Imagine if the browser maintained user interaction state (mouse position, touch screen location, etc) in a read-only XML file directly accessible to developers. For example:

<pointers>
<pointer status="active" screenX="100" screenY="100" elementref="div0" relativeX="5" relativeY="5"/>
</pointers>

For the mouse, there would only be one <pointer/>, with "active" status when the mouse was down, and "inactive" when up. For touch screens, there would any number of <pointer/> elements (including zero), each representing a finger or stylus touching the screen. The elementref attribute stores a reference to the element that the pointer is currently over, the relativeX and relativeY commands store the location relative to this element, and the screenX and screenY elements store the location relative to the screen.

Once you have this file, you can do functional animation based on it. For example, using the XForms <bind> tag:

<bind infoset="id('img1')" calculatewhen="//pointer[@elementref = 'img1' && @status='active']">
  ./@css:left = "//pointer[@elementref = 'img1']/@screenX;
  ./@css:top = "//pointer[@elementref = 'img1']/@screenY;
</bind>

which once activated, binds img1 to the evolving location of the mouse / stylus.

As you can see, this approach avoids the need to use javascript at all - event handlers and declarative functional animation are enough.

Touch screens are the future

Computer mice have been around for so long that it's tempting to see them as a permanent fixture in computing. But actually they're pretty uninuitive - remember seeing someone using a mouse for the first time?

As touch screens spread, web developers will be faced with an interesting set of challenges, which are best overcome using a few simple CSS tags, and a declarative approach.

Thursday, February 15, 2007

My Proposal: Functional Animation

In this blog entry I'm going to explain some of the detail behind my proposal for functional animation, which was introduced in a previous entry.

The goals are to allow any XML document node or CSS stylesheet property to:

Evolve as an explicit function of time and other (possibly also evolving) nodes or CSS values f(t, a, b, c, …)
Apply the function conditionally, or depending on events such as key presses
For continuously varying values, set or reference their speed and acceleration

Goal 1 says that the language should be functional. Rather than calculating an incremental change every millisecond, as per javascript, the property evolves according to a function. For example, it might be "width = 2 * height" which would maintain the width at twice the height, no matter how the height evolves.

This requires us to incorporate a referencing language, so that each element can refer to the others. There are lots of them around – there’s one in CSS and another in SMIL – but there is a more powerful standard, XPath, to take advantage of. The other benefit of XPath is that it introduces standard mathematical and string functions. For example, why not specify "width = avg(//img[@class='ball']/@width)", which sets the width to the average of every image width with 'ball' class.

It also implies that the language works like a spreadsheet - any time a value changes, the effects can ripple through all the dependent values. Like Microsoft Excel, the system needs a dependency engine in order to quickly figure this out and work all the way down the dependency chain (looking out for circular references).

Finally, XPath allows us to work with t, the time. Unfortunately there’s no pre-defined time variable that starts from zero when the animation begins – but we can create our own, by using the XPath system clock function and measuring from when the animation began.

Goal 2 demands that our function language has some conditional statements in it, possibly dependent on node values. For example, perhaps an SVG image is programmed to be repelled by another image, but only if they get too close. Or perhaps it speeds up when the mouse clicks (which requires integration with the events model in HTML/XML).

There are two existing XML animation technologies that go some way to meeting these two goals. The first is Synchronized Multimedia Integration Language (SMIL), which despite its name incorporates a general-purpose model for animating any XML document. It is very straightforward, and works well for basic animations.

Unfortunately, for more complex animations SMIL has severe limitations, which stem from the fact that it doesn’t meet Goal 1 fully. The most basic limitation is that SMIL animations follow a pre-defined path. SMIL doesn’t handle situations where the desired path is dependent on unpredictable evolving conditions – for example, the mouse position, or the location of other moving objects or even a random number generator. SMIL also doesn’t incorporate XPath, which means it’s difficult to reference the values of other nodes.

The second existing XML animation technology is XForms. It may seem surprising that a web forms technology incorporates sophisticated animation functionality, but it does - because it includes a functional binding language based on XPath, which meets Goal 1. All it needs is a few simple extensions.

For example, consider the following XForms line:

<bind calculate="2" infoset="//img/@css:width">

which takes every image tag in the document, and sets the width to equal twice the height (using the XPath statement). Now, any time the height of any image is altered, its width will automatically reset to be double.

This might not be sophisticated animation, but it’s not possible with SMIL, and let’s take things a step further:

<script >document.getElementById(‘divTime’).setAttribute(‘timeStarted’, now())</script >
<bind infoset="id(‘divTime’)">
./@timeElapsed = current-dateTime() - //divTime/@timeStarted;
</bind>
<bind infoset="//img">
./@css:top = 200;
./@css:left = 100 + 100 * sin(id(‘divTime’)/@timeElapsed);
</bind>

First I have used javascript to set the timeStarted attribute to the system time when the page loads. Next I have extended XForms so the contents of the <bind> element works just like a series of calculate attributes.

The first <bind> element sets up a counter – the timeElapsed attribute – that holds the number of seconds since the page loaded.

The second <bind> element animates every image on the page from side to side according to a sine function.

You can see the immediate parallels between the <bind> element and CSS. It’s just the same, except it uses XPath as a referencing tool, and you can assign functions to each variable, not just static values. You could even put the <bind> tags in a separate stylesheet, just like CSS. Or you could get rid of your exising .css files, and replace them with the syntax above.

If the W3C followed this approach, it would pull the XForms <bind> element into a separate XML Functional Animation spec, which would form a foundation for CSS and supersede most of SMIL.

I’d like to give some more examples to show just how powerful this approach is. First, I’ll introduce three more new pieces

The calculatewhile attribute, which is an XPath boolean statement that controls whether the <bind> element should be run or ‘paused’
The new XPath function d_dt(), which sets or retrieves the rate of change (speed) of any node
The new XPath function d_dt2(), which sets or retrieves the acceleration of any node – the equivalent of d_dt(d_dt())

These pieces enable Goal 3 to be achieved:

<bind infoset="//img">
d_dt2(./@css:left) = - ./@css:left;
</bind>

which turns the images into simple harmonic oscillators (i.e. springs), vibrating backwards and forwards like a child on a swing.

Or consider:

<bind infoset="id(‘img1’)">
d_dt2(./@css:left) = id(‘img2’)./@css:left – id(‘img1’)./@css:left;
</bind>
<bind infoset="id(‘img2’)">
d_dt2(./@css:left) = id(‘img1’)./@css:left – id(‘img2’)./@css:left;
</bind>

which models two balls, joined by a spring.

Once you’ve thought about it, you realize that this approach to animation allows pretty much anything in classical physics to be modeled – wind resistance, friction, magnetism, gravity, etc. Which means it’s pretty useful in programming computer games! And there isn’t a tougher test for an animation language than this.

In summary, I think the XForms <bind> element contains, with a few simple extensions, everything the XML developer needs to produce world-class animations.

Monday, February 12, 2007

Animating the web: functional styles

From the first time I saw it, I’ve thought that the equals sign in programming languages is wrong. In maths, the statement x=5 says that no matter what happens, x will always equal five. In C++, or Java, or Visual Basic, it only sets x instantaneously to 5; x can still evolve over time.

This is important, because mathematical symbols (like the equals sign) are a result of centuries of learning about the best way to represent fundamental concepts. They have proved their worth time and time again in explaining the natural world. In fact, many revolutions in science have only taken place by the introduction of new symbols – Newton’s differentiation and integration symbols to explain dynamics, Einstein’s use of Reinemann equations in General Relativity, and Heisenberg’s use of matrices in Quantum Mechanics are cases in point - but the equals sign has remained constant.

So why do programming languages not follow standard mathematics?

Actually, there's one that does - spreadsheet formulas. It's probably why Excel formulas are the only language to spread out of the IT department. In Excel, if you set cell A2 to equal A1 + 5, then this will always be true – when A1 changes, A2 will update automatically to maintain the equality.

Excel handles the equals sign properly because it’s a functional programming language. Except for spreadsheets, functional languages are niche - the most common apart from Excel is probably Lisp, which was invented back in the 1960s.

Since functional programming expressions stay true even as the program evolves, they come into their own in animations.

Think back to spreadsheet functions, and imagine if you could write an HTML expression like this:

          <img id="img1" src="img1.jpg" width="10" />
          <img id="img2" src="img2.jpg" width="=2*img1.width" />

Note the extra equals sign, as per Excel, controlling img2 width. If img1 were resized, img2 would automatically resize too, in order to maintain its double width.

If you’ve ever tried to enable drag and drop in Javascript, you’ll know how awkward it is. With the functional approach, it’s just one line of code:

 <img id="img1" style="left:=mouse.left; top:=mouse.top" begin="img1.mousedown" end="img1.mouseup" />

This uses a simple condition to control when the animated style should work.

Once you see how this could work, you realise how cumbersome the procedural events model is in Javascript. The setInterval() function is unreliable (since you can't rely on processing speed) and inelegant - far better to think continuously, rather than triggering new events every millisecond.

I've called this approach "functional styles", because it's basically a functional programming extension to CSS styles. In this approach, any CSS style can be animated by assigning it a function (via the equals sign), rather than a direct value. And these functions have access to two continuously varying variables - the mouse position, and the time variable t.

Functional styles would open up animation on the web. Think Powerpoint animations, think Flash timelines, think interactive games, think interactive graphs and charts. In fact, think web spreadsheets! All currently require mountains of javascript and a very fast processing engine. Using functional styles, they would simply require CSS.

In a later post, I will go into the details of how this could technically work - there are very few elements other than basic CSS, XPath, and SMIL. It's possible to prove that the entire of classical physics can be incorporated into functional styles - gravity, friction and wind resistance, angular momentum, electric fields, and magnetism are all a matter of getting your CSS functions right.

For now, think how the simple equals sign, done properly, enables rich animations on the web.