Showing posts with label binding. Show all posts
Showing posts with label binding. Show all posts

Thursday, August 02, 2007

Harry Potter technology: animated paintings

There's a great scene in the film Harry Potter and the Order of the Pheonix when Hogwart's caretaker, Argus Filch, is taking down an old tudor oil painting. As he twists the painting to remove it, the men in ruffs get angry, shaken from side to side, and eventually fall off the bottom.

It's visually stunning, but also emotionally engaging - it gives the viewer a real connection with the men in the painting. Imagine owning a photo frame that did this!

In fact, it must already be possible to create this effect for real, perhaps using the iPhone (since it has tilt sensors).

Surely picture animation is the next huge area for art - a way to break out of the static image and into lifelike, arresting motion. Why shouldn't the next Lucian Freud create animated paintings?

It's also one of the first digital art forms that isn't a direct copy of an analog one - unlike photography or film cartoons, you simply can't do it using paper. And it seems an even better idea than that other Harry Potter gem, the whereabouts clock.

All we need now is an open standard that describes such animation - I don't think SMIL really cuts the mustard...

Sunday, July 01, 2007

Functional Animation, revisited

Since my blog post on Functional Animation, there's been some movement towards the idea of setting DOM nodes to equal functions of other (potentially varying) nodes.

In fact, this idea already exists in XSL - see the code below, where the second paragraph is set to contain the same text as the first paragraph.


<?xml version="1.0" encoding="ISO-8859-1"?>
<html xmlns="http://www.w3.org/1999/xhtml/" 
  xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
<body>
<p id="p1">p1 content</p>
<p id="p2" title="{//p[@id='p1']}">
          <xsl:value-of select="//p[@id='p1']"/>
</p>
<a href="javascript:document.getElementById('p1').textContent 
  = 'test'">Test</a>
</body>
</html>

When you click on the "Test" link, the value of p1 is changed - and p2 should automatically update too (both its content and its title attribute).

Unfortunately, p2 doesn't update - it seems that browser makers don't automatically re-execute XSL stylesheets. Interestingly, CSS stylesheets do re-execute when you add or change nodes, and there's no mention of what should happen in the XSL spec. I don't see why browser makers couldn't extend their code.

An idea whose time has come

This idea of Functional Animation is so useful and intuitive - probably replacing most awkward DOM Events code - that it's a real shame there aren't any widescale implementations.

Having said that, I recently saw the idea cropping up in several places:

Personally, I like the XSL approach with inline XPath, because it's much more natural at selectors. On the other hand, some might prefer javascript due to its familiarity and extra features (e.g. Math and String functions).

Either way, it's clearly an idea whose time has come!

Friday, May 04, 2007

Sensors

This week's Economist contains some predictions about machine-to-machine wireless communications. Most of the devices mentioned seemed to be sensors - whether used by the military, civil engineers, security guards, doctors, or retailers.

This got me thinking about how sensors can be handled on the web. We're all used to dealing with mice and keyboards - what about location or pressure sensors, or thermometers, accelerometers and gyroscopes, which are already being integrated into phones?

Providing sensory data to the web
My idea is that browsers should pull together all this information and make it available to web pages, in a standard way. Sensory information is becoming more and more important, especially in the mobile web, where knowledge of location, direction, and acceleration are vital to display great web pages.

For example, imagine if the following XML fragment was accessible via a javascript sensors() function:

<sensors xmlns="www.sensors.org/namespace">
<keyboard shift="" ctrl="" alt="" ins="" value="a"/>
<mouse x="20px" y="30px" left="down" right="none" middle="none"/>
<touch pressure="30" x="150px" y="50px"/>
<temperature value="23C"/>
<video src="file://c/program%20files/webcam/webcam.mov"/>
<accel x="2" y="0" z="0"/>
<location latitude="37.386013" longitude="-122.082932"/>
</sensors>

Here, the browser is presenting all the information it can find about its environment from connected sensors - the A button is down on a keyboard, the mouse is being clicked, the screen is being touched, the temperature is being read, there is a connected webcam, the device is being accelerated, and it knows its position. All defined in a (fictitious) standard XML data format.

Using sensory data
Different devices have different sensors - the Wii has an accelerometer, the Nokia N95 has GPS, my phone has a camera - so the sensory data will be different in each case. And there may be privacy implications - you might configure your browser to grant location data only to the emergency services and your favourite map website.

So the web developer's first step will be to parse the data to find out which sensors are available. They could do this using XPath - for example sensors('//accel/@x') only returns a value if there is an accelerometer.

Imagine using the following javascript:

window.setTimeout($(div1).innerHTML = sensors('//location/@latitude'),10);

which in a single line, updates the div1 tag to contain up to date latitude information every 10 milliseconds.

The possibilities are endless

  • satellite navigation in the browser
  • scroll web pages using acceleration
  • pen doodling on the web, using a touch screen and SVG / VML
  • website games using local web cams.

Personalising your pages
Sensory data is the ultimate way to personalize web pages. They can react in realtime to the local environment that web page visitors are experiencing.

There is currently no standard framework for accessing this data - but the simple ideas above would bring the web to the next level.

Wednesday, March 28, 2007

XForms has a problem

XForms is the W3C’s XML language for programming forms on the web. It handles new user interface controls, better data validation, and easier form submission.

It does this by strictly following the "MVC" approach to data management – a ‘model’ is stored in the XHTML head, containing data structures bound to user interface form controls. When you enter values into the form controls, the model automatically updates too, and it’s the model that gets submitted.

In order to get the model to work, XForms defines an incredibly powerful binding engine, like a spreadsheet where cell values can depend on each other. It’s here that I have the problem. Why is the binding technology written inside the XForms spec? It’s useful for so many purposes (e.g. spreadsheets) that have nothing to do with forms – or even XHTML - and it stands on its own as a fundamental use of XML.

For example, I might have an SVG document where I want to bind the width of a circle to the length of a line. I can’t use XForms binding, because it’s not an XHTML document and even if it was, I don’t want to follow the MVC approach.

So, the tag should be pulled out of the XForms spec and put in a new XML bindings spec, that doesn’t force you to follow MVC (or even XHTML).

That way, all XML developers will gain a new technology that provides so much value, especially in animations. And XForms will become much simpler, and more targeted towards its role in enabling web forms.

Friday, March 02, 2007

XML Events 2.0

The latest draft of XML Events 2.0 contains a new, simple, and very powerful idea - allowing XPath to handle events and even animate elements.

But there are a few ways it could be extended even further.

XPath observers

Firstly, the draft spec hints that the "observer" attribute could in future be used to give any URI, not just from the current document. But it would be far better to allow an XPath statement, rather than a URI. Then you could specify a whole set of nodes. For example, consider:

<.script event="mouseclick" observer="//img">.
  document.getElementById("paragraph2").style.fontSize = "5";
<./script>.
which uses the XML Events namespace "ev" to alter the paragraph font when any image is clicked.

XPath event() function

Secondly, there's a new XPath event() function, which coupled with the new "while" attribute allows for animation. Consider this implementation of drag and drop:

<.script observer="//img" while="event(button) =1" />.
left="event.clientX";
top="event.clientY" 
<.script/>.

Not bad eh, especially compared to mountains of javascript. The new XPath event() function allows you to figure out mouse position and even the time when the event occured - which is very useful when it comes to animation and controlling timing.

<.style observer="//img" while="true()">.
./@css:left = //div1/@css:left + 10;
<./style>.

Note the "style" element - this is just a CSS stylesheet using XPath. It positions each image 10 pixels to the right of the div1 element. If the div1 element is moved, then the images will move to maintain their relative position.

There are two benefits to this approach. Firstly, rather than using CSS selectors, you can use the full power of XPath to indicate the elements being styled. And secondly, you can use all sorts of XPath functions to assign values - like avg() or sum() or even sin().

XPath + XML Events

So I would urge the XML Events authors to extend the use of XPath in their spec. Firstly, XPath should be used in the observer attribute, to allow any collection of elements to be handled. Secondly, please keep the new event() funtion - it's great! And thirdly, XPath should be used to allow the animation of XML values.

Thursday, February 15, 2007

My Proposal: Functional Animation

In this blog entry I'm going to explain some of the detail behind my proposal for functional animation, which was introduced in a previous entry.

The goals are to allow any XML document node or CSS stylesheet property to:

  1. Evolve as an explicit function of time and other (possibly also evolving) nodes or CSS values f(t, a, b, c, …)
  2. Apply the function conditionally, or depending on events such as key presses
  3. For continuously varying values, set or reference their speed and acceleration

Goal 1 says that the language should be functional. Rather than calculating an incremental change every millisecond, as per javascript, the property evolves according to a function. For example, it might be "width = 2 * height" which would maintain the width at twice the height, no matter how the height evolves.

This requires us to incorporate a referencing language, so that each element can refer to the others. There are lots of them around – there’s one in CSS and another in SMIL – but there is a more powerful standard, XPath, to take advantage of. The other benefit of XPath is that it introduces standard mathematical and string functions. For example, why not specify "width = avg(//img[@class='ball']/@width)", which sets the width to the average of every image width with 'ball' class.

It also implies that the language works like a spreadsheet - any time a value changes, the effects can ripple through all the dependent values. Like Microsoft Excel, the system needs a dependency engine in order to quickly figure this out and work all the way down the dependency chain (looking out for circular references).

Finally, XPath allows us to work with t, the time. Unfortunately there’s no pre-defined time variable that starts from zero when the animation begins – but we can create our own, by using the XPath system clock function and measuring from when the animation began.

Goal 2 demands that our function language has some conditional statements in it, possibly dependent on node values. For example, perhaps an SVG image is programmed to be repelled by another image, but only if they get too close. Or perhaps it speeds up when the mouse clicks (which requires integration with the events model in HTML/XML).

There are two existing XML animation technologies that go some way to meeting these two goals. The first is Synchronized Multimedia Integration Language (SMIL), which despite its name incorporates a general-purpose model for animating any XML document. It is very straightforward, and works well for basic animations.

Unfortunately, for more complex animations SMIL has severe limitations, which stem from the fact that it doesn’t meet Goal 1 fully. The most basic limitation is that SMIL animations follow a pre-defined path. SMIL doesn’t handle situations where the desired path is dependent on unpredictable evolving conditions – for example, the mouse position, or the location of other moving objects or even a random number generator. SMIL also doesn’t incorporate XPath, which means it’s difficult to reference the values of other nodes.

The second existing XML animation technology is XForms. It may seem surprising that a web forms technology incorporates sophisticated animation functionality, but it does - because it includes a functional binding language based on XPath, which meets Goal 1. All it needs is a few simple extensions.

For example, consider the following XForms line:

<bind calculate="2" infoset="//img/@css:width">
which takes every image tag in the document, and sets the width to equal twice the height (using the XPath statement). Now, any time the height of any image is altered, its width will automatically reset to be double.

This might not be sophisticated animation, but it’s not possible with SMIL, and let’s take things a step further:

<script >document.getElementById(‘divTime’).setAttribute(‘timeStarted’, now())</script >
<bind infoset="id(‘divTime’)">
./@timeElapsed = current-dateTime() - //divTime/@timeStarted;
</bind>
<bind infoset="//img">
./@css:top = 200;
./@css:left = 100 + 100 * sin(id(‘divTime’)/@timeElapsed);
</bind>

First I have used javascript to set the timeStarted attribute to the system time when the page loads. Next I have extended XForms so the contents of the <bind> element works just like a series of calculate attributes.

The first <bind> element sets up a counter – the timeElapsed attribute – that holds the number of seconds since the page loaded.

The second <bind> element animates every image on the page from side to side according to a sine function.

You can see the immediate parallels between the <bind> element and CSS. It’s just the same, except it uses XPath as a referencing tool, and you can assign functions to each variable, not just static values. You could even put the <bind> tags in a separate stylesheet, just like CSS. Or you could get rid of your exising .css files, and replace them with the syntax above.

If the W3C followed this approach, it would pull the XForms <bind> element into a separate XML Functional Animation spec, which would form a foundation for CSS and supersede most of SMIL.

I’d like to give some more examples to show just how powerful this approach is. First, I’ll introduce three more new pieces

  • The calculatewhile attribute, which is an XPath boolean statement that controls whether the <bind> element should be run or ‘paused’
  • The new XPath function d_dt(), which sets or retrieves the rate of change (speed) of any node
  • The new XPath function d_dt2(), which sets or retrieves the acceleration of any node – the equivalent of d_dt(d_dt())
These pieces enable Goal 3 to be achieved:
<bind infoset="//img">
d_dt2(./@css:left) = - ./@css:left;
</bind>
which turns the images into simple harmonic oscillators (i.e. springs), vibrating backwards and forwards like a child on a swing.

Or consider:

<bind infoset="id(‘img1’)">
d_dt2(./@css:left) = id(‘img2’)./@css:left – id(‘img1’)./@css:left;
</bind>
<bind infoset="id(‘img2’)">
d_dt2(./@css:left) = id(‘img1’)./@css:left – id(‘img2’)./@css:left;
</bind>
which models two balls, joined by a spring.

Once you’ve thought about it, you realize that this approach to animation allows pretty much anything in classical physics to be modeled – wind resistance, friction, magnetism, gravity, etc. Which means it’s pretty useful in programming computer games! And there isn’t a tougher test for an animation language than this.

In summary, I think the XForms <bind> element contains, with a few simple extensions, everything the XML developer needs to produce world-class animations.

Monday, February 12, 2007

Animating the web: functional styles

From the first time I saw it, I’ve thought that the equals sign in programming languages is wrong. In maths, the statement x=5 says that no matter what happens, x will always equal five. In C++, or Java, or Visual Basic, it only sets x instantaneously to 5; x can still evolve over time.

This is important, because mathematical symbols (like the equals sign) are a result of centuries of learning about the best way to represent fundamental concepts. They have proved their worth time and time again in explaining the natural world. In fact, many revolutions in science have only taken place by the introduction of new symbols – Newton’s differentiation and integration symbols to explain dynamics, Einstein’s use of Reinemann equations in General Relativity, and Heisenberg’s use of matrices in Quantum Mechanics are cases in point - but the equals sign has remained constant.

So why do programming languages not follow standard mathematics?

Actually, there's one that does - spreadsheet formulas. It's probably why Excel formulas are the only language to spread out of the IT department. In Excel, if you set cell A2 to equal A1 + 5, then this will always be true – when A1 changes, A2 will update automatically to maintain the equality.

Excel handles the equals sign properly because it’s a functional programming language. Except for spreadsheets, functional languages are niche - the most common apart from Excel is probably Lisp, which was invented back in the 1960s.

Since functional programming expressions stay true even as the program evolves, they come into their own in animations.

Think back to spreadsheet functions, and imagine if you could write an HTML expression like this:

          <img id="img1" src="img1.jpg" width="10" />
          <img id="img2" src="img2.jpg" width="=2*img1.width" />
         
Note the extra equals sign, as per Excel, controlling img2 width. If img1 were resized, img2 would automatically resize too, in order to maintain its double width.

If you’ve ever tried to enable drag and drop in Javascript, you’ll know how awkward it is. With the functional approach, it’s just one line of code:

 <img id="img1" style="left:=mouse.left; top:=mouse.top" begin="img1.mousedown" end="img1.mouseup" />
This uses a simple condition to control when the animated style should work.

Once you see how this could work, you realise how cumbersome the procedural events model is in Javascript. The setInterval() function is unreliable (since you can't rely on processing speed) and inelegant - far better to think continuously, rather than triggering new events every millisecond.

I've called this approach "functional styles", because it's basically a functional programming extension to CSS styles. In this approach, any CSS style can be animated by assigning it a function (via the equals sign), rather than a direct value. And these functions have access to two continuously varying variables - the mouse position, and the time variable t.

Functional styles would open up animation on the web. Think Powerpoint animations, think Flash timelines, think interactive games, think interactive graphs and charts. In fact, think web spreadsheets! All currently require mountains of javascript and a very fast processing engine. Using functional styles, they would simply require CSS.

In a later post, I will go into the details of how this could technically work - there are very few elements other than basic CSS, XPath, and SMIL. It's possible to prove that the entire of classical physics can be incorporated into functional styles - gravity, friction and wind resistance, angular momentum, electric fields, and magnetism are all a matter of getting your CSS functions right.

For now, think how the simple equals sign, done properly, enables rich animations on the web.