On Apple's CSS animation proposal

Apple recently published new proposals for CSS transitions and animations. Having spent some time reviewing approaches to animation on the web, I conclude that their animation proposal has serious shortcomings, and identify a better approach.

Animation - controlling the evolution of styles like position, colour, size, fonts, and layout - has always been a crucial gap for the open web. That's why it's been a key selling point for plug-ins such as Flash, albeit in a proprietary way that doesn't integrate well with the rest of the page. Animation is a good thing and should be brought to the web asap.

For more than ten years now, the W3C's answer to animation has been SMIL. But SMIL has a fundamental problem - it can only animate XML. On the web, you don't want to animate content - you want to animate style. Style is not stored as XML, or even as markup - it's stored as CSS. Finally, Apple has overcome the inertia with CSS animation. Now there is a chance to shape the way it works - hopefully this review will play a role!

There are two fundamentally different ways to evolve style - transitions and animations. In transitions, you don't know in advance what the before and after styles are, you just want to control how quickly the transition takes place for each style property. For example, you could say that background-color always takes two seconds to change, rather than being instantaneous as normal. In animations, you control both the before and after styles, plus the path between them.

Transitions

Apple's model for transitions is clear and straightforward - you simply apply a transition rule to the relevant CSS properties - for example, perhaps there is a delay of two seconds whenever the div's opacity changes:

div {
   transition-property: opacity;
   transition-duration: 2s;
   transition-timing-function: linear;
}

Transitions enable a huge number of simple effects, from context menus that slide out on mouse over to page sections that fade out when closed.

I like this model because it's simple and orthogonal to all the other styles (you can't set the actual property values, only their timing), yet gives them even more power. Also, the new transition styles follow the proper cascading rules as they are applied through the DOM.

Apple's Animations

Apple takes a very similar approach with animations. Using the same opacity example:

div {
   animation-name: div-opacity;
   animation-duration: 2s;
   animation-iteration-count: 1;
}

@keyframes 'div-opacity' {
  from {
    opacity: 0%;
    animation-timing-function: linear;
  }
  to {
    opacity: 100%;
  }
}

Unlike transitions, animations set the exact values over time of the opacity style, using keyframes. This is where the problems arise.

The first issue is orthogonality. Keyframes provide a new way to set the div's opacity, away from its normal position (under the div selector). This adds unnecessary confusion to parsing and understanding the CSS document - there are now two ways to set a style. It also requires several new CSSOM interfaces to control keyframes via script.

As a result, keyframes have a much bigger issue - they don't cascade. Cascading is one of the most important characteristics of stylesheets - it's the C in CSS. Cascading sets a series of priorities for when to apply style rules, based on the DOM and where they are applied. Because keyframes are a totally separate part of CSS, cascading can't work its magic.

For example, what happens if opacity was set in both the div selector and the keyframe? You could set an arbitrary rule to give one location priority, but it would be just that - arbitrary. And how does opacity apply to any elements inside the div? Apple have proposed that keyframes don't cascade. But this removes much of the power of CSS.

A better approach to animation

There's a better approach to animation that respects both the orthogonality and cascading principles. I also think it's simpler - it certainly requires fewer lines of code. See an example below that does exactly the same as the animation example above:

div {
   opacity: calc(t / 2s * 100%);
}

There are two key elements to the solution:

• The CSS3 calc function, which enables simple mathematics like multiplication and division.
• The new standard variable t, which measures elapsed time in seconds, starting at t=0 when the style is first applied to the element

In the example above, opacity would start at t=0 with a value of (0s / 2s) * 100%, which is 0%. After exactly two seconds, opacity would have the value (2s / 2s) * 100%, which is 100%.

Notice that since t is measured in seconds, we need to divide by a time unit (in this case 2s) in order to get the units right. I've also multiplied by 100% to return a percentage unit accepted by opacity.

The benefits of this inline approach are that it maintains both orthogonality and cascading rules - in fact, animated styles cascade in exactly the same way as static ones. It's also easier (and much shorter) to read, and requires no additional CSSOM interfaces.

To provide the complete picture you need additional animation functions, plus a few discretionary parameters. Following Apple's approach, I recommend the following:

• animate-ease(time, iterationcount=1, direction=normal)
• animate-linear(time, iterationcount=1, direction=normal)
• animate-ease-in(time, iterationcount=1, direction=normal)
• animate-ease-out(time, iterationcount=1, direction=normal)
• animate-ease-in-out(time, iterationcount=1, direction=normal)

In addition, the following functions provide more control

• animate-step(time) which is the step function, returning zero when time<0s id="dh-l">
• animate-keyframes(time0 value0, time1 value1, time2 value2, ...), which returns a curve smoothly connecting the points via a bezier function.This negates the need for a separate cubic bezier function.

For example, the following styles apply the same effect as above, but eased-in and stepped after 1s rather than linear:

div.ease-in {
   opacity: calc(animate-ease(t / 2s) * 100%);
}
div.step {
   opacity: calc(animate-step(t / 1s - 1s) * 100%);
}

The following style iterates linearly every second four times in a row, alternating directions each time:

div.iterate {
   opacity: calc(animate-linear(t / 1s, 4, alternate) * 100%);
}

The following style illustrates more complicated animations, by moving an image downwards with uniform acceleration:

div.gravity {
   top: calc(t*t / (1s*1s) * 1px);
}

Synchronisation

Under this model, separate animations are implicitly synchronised. For example, consider the following animation:

div.projectile {
   top: calc(t*t / (1s*1s) * 1px);
   left: calc(t / 1s * 1px)
}

The instant a div element is given the projectile class, both animations will be set to t=0, and hence will be syncronised together.

On the other hand, the following animations will not be automatically synchronised:

<style>
div.moveright {
   left: calc(t / 1s * 1px);
}
div.movedown {
   top: calc(t / 1s * 1px);
}
</style>
<script>
var div1 = document.getElementsByTagName("div")[0];
div1.className = div1.className + " moveright";
div1.className = div1.className + " movedown";
</script>

The classes have been set at slightly different times, one after the other, and therefore the animations will begin at slightly different times as well.

Conclusion

Animation has the potential to turbo-charge style on the web. It's important that it's done in a way that enables the full power of CSS, including the principles of orthogonality and cascading. Apple's transitions model meets these principles, but their animations model does not, so I have proposed a replacement.

Tabbed Browsing

Tabbed browsing has been one of the key recent improvements to the web. It's made it far easier to work with multiple pages - many people keep dozens of tabs open for days, waiting for the opportunity to read or complete them. It was one of the main selling points behind Internet Explorer 7.

And yet, tabbed browsing is terrible. You can't resize, reshape and move tabs, like you can normal windows - they're all stuck at the size of the browser window. You can't search across every tab. And they blatantly overlap with the taskbar, for those using Windows.

Is there a better approach than tabs? Sure - think how you organise pieces of paper on a desk. They're in piles, at various angles, and at any point you can bring them to the front and work on them. But hmm - pieces of paper tend to get lost or crumpled under others.

I still don't think anyone has properly implemented a simple, powerful, and intuitive interface for working with multiple documents visually. That seems ridiculous - what on earth have we been doing for so long!

We can guess at what a solution might look like - a multi-touch screen allowing document resize and zoom, a quick search function across open documents, some way to remember default document dimensions. Some combination of Mozilla SVG photos and Jeff Han's multi-touch.

In the meantime it's worth pointing out that, for all their advantages, tabbed browsers are only a quick and dirty fix to the problem of working across multiple documents.

The TV and the Computer

The fight for the digital living room continues. Apple TV, the XBox 360, Microsoft's Home Server, and the set top box all compete to provide multimedia services to the family.

This is horribly wrong. I just can't see the value in having a whirring black box control center in the living room - it's a single point of failure, it's a closed solution (since everything else must plug into it), it's a bottleneck against content on the web, and it forces Dad to play system administrator!

As William Gibson said, "the future is already here, it's just not uniformly distributed". Look at the iMac. Take away the keyboard and mouse, and what does it look like? A TV.

Now imagine it only has one application - the browser - and that it boots up in 2 seconds, like an iPod. Your Flickr photos, Amazon Music and BBC iPlayer programmes are now available, on demand, from the web. You can purchase another TV, put it in the kitchen, and access the same websites - there's no need for a central controller or set top box.

For the remote control, all you need is a wireless mouse! Instead of pressing channel numbers, you navigate between your browser favourites. You can type a new URL or search query using a simple onscreen popup keyboard (unless you really want to connect a full wireless keyboard)

All this is surely feasible today. How much extra would it really cost to include a stripped down Linux OS with Firefox on a $1500 widescreen TV?

The future is putting browsers in TVs. I really don't think that even Apple and Microsoft will be able to stop it.

CSS Color Gradients

1. Introduction

This document proposes a new feature in CSS for creating color gradients. These gradients define an additional <color> value type for the CSS3 Color specification; as such, they can be used in place of a traditional color anywhere in CSS, e.g. as a background color gradient, a text color gradient, or a border color gradient.

The proposals are based on the SVG specification. However, CSS color gradients could style any markup element, including HTML.

2. color-gradient-linear

Linear color gradients are created using the color-gradient-linear color value, which takes several parameters:
color-gradient-linear(angle, offset1 color1, offset2 color2, ... , spreadmethod)

angle indicates the angle of the gradient from horizontal, moving clockwise so that an angle of 0 deg creates a gradient from left to right, and 90 deg creates a gradient from top to bottom.

At least one offset & color pair must then follow. offset indicates where to start the gradient, and can take any length unit. Draw a line through the containing element, following the gradient angle; offset takes the values 0% and 100% where the line hits the element's inside border.

For example, if angle=0deg, then an offset of 0% indicates the gradient starts at the left hand border. If angle=90deg, then an offset of 100% indicates the bottom border.

color can be any valid CSS3 color value, including rgba and hsla values that enable opacity.

The spreadmethod parameter, which is not required, can take three values - "pad | reflect | repeat". This indicates what happens if the gradient starts or ends inside the bounds of the target element. Possible values are: pad, which says to use the terminal colors of the gradient to fill the remainder of the target region, reflect, which says to reflect the gradient pattern start-to-end, end-to-start, start-to-end, etc. continuously until the target element is filled, and repeat, which says to repeat the gradient pattern start-to-end, start-to-end, start-to-end, etc. continuously until the target region is filled. The default value is "pad".

Examples:

    /*linear gradient from green to blue, moving left to right*/
em { color: color-gradient-linear(0deg, 5% green, 95% blue) } 
    /*linear gradient from a transparent red to solid purple, top to bottom*/
em { color: color-gradient-linear(90deg, 5px rgba(100,100,100, 0.5), 200px purple) }
    /*linear gradient from green to blue to yellow, right to left*/
em { color: color-gradient-linear(180deg, 3px green, 50% blue, 100% yellow) }
    /*linear gradient from green to blue to green to blue ..., left to right*/
em { color: color-gradient-linear(0deg, 0px green, 10px blue, reflect) }

3. color-gradient-radial

Radial color gradients are created using the color-gradient-radial color value, which takes several parameters:
color-gradient-radial(center-x center-y, radius1 color1, radius2 color2, ... , spreadmethod)

center-x and center-y indicate the center of the radial gradient, offset from the top left corner of the element.

At least one radius & color pair must then follow. radius indicates where to start the gradient; it must be a positive length value, starting from the center of the circle. A value of 100% for radius indicates that the gradient is equal in length with the element's width.

The spreadmethod parameter works exactly as for color-gradient-linear.

Examples:

    /*radial gradient from green to blue, center outwards, starting at the element's center*/
em { background-color: color-gradient-radial(50% 50%, 0px green, 50px blue) } 
    /*radial gradient from a transparent red to solid purple, center outwards, starting top left*/
em { background-color: color-gradient-radial(0px 0px, 5px rgba(100,100,100, 0.5), 200px purple) }
    /*radial gradient from green to blue five times, center outwards, starting at the element's center*/
em { background-color: color-gradient-radial(50% 50%, 0px green, 20% blue, reflect) 

Browser User Interfaces

You can tell we've really begun to understand the web in the last five years, because while browsers have got more powerful, their user interfaces are now simpler and clearer.

This isn't always the case; think back to the office suite wars in the 1990s, when Microsoft and others added endless cluttered options, menus and functions to every release of their word processors and spreadsheets.

See the screenshots below of IE4 versus IE7 - the later release is much more streamlined.

The next version of Firefox, v3.0, is simplifying even further by merging History and Bookmarks into a much more powerful, unified interface: Places.

How far can this go?

Even further! I've listed some ideas below which would simplify and extend whole areas of browser design:

Calendar

Show the current time in the browser bar. When clicked, it opens your pre-defined calendar site (e.g. google calendar), and you can also drag text onto the clock from a webpage (e.g. "meet in Canary Wharf tonight at 7pm") to add events to your calendar.

Location

For devices with GPS. Show the current location in the browser bar (e.g. "Canary Wharf"). When clicked, it opens your pre-defined map site (e.g. google maps), and you can also drag text onto it from a webpage (e.g. "Canary Wharf") to view that place in a map.

Style

Subscribe to an external CSS stylesheet to control default and override settings like font name, font size, link underline, and audio volume. The stylesheet is cached locally and can be edited in a user-friendly manner via the subscribed website. Mozilla could set up their own stylesheet site as a start.

Page Analysis

Display page file size, security information, "view source", "view HTTP headers", error messages, and spelling or grammer checks. This could be done by posting the page to a subscribed analysis website, if security allows.

Tabs and Windows

Drag hyperlinks onto the tab bar to open them in a new tab, or drag them outside the browser window to open them in new window.

The goal is to reduce the browser interface down to a very few, powerful functions.

There's a common theme with the ideas above: functionality that used to be part of the browser - e.g. default fonts, page information - is now provided by a website. You enter the website details in the browser, and the browser hands over the appropriate information.

Websites are likely to be much better than the browser at certain tasks, because of the speed of application development and deployment on the web, the power of HTML and mashups, and the funding of Silicon Valley. It also allows browser makers to concentrate on page rendering, their core competency.

Security Considerations

Of course, the problem with handing features over to websites is security.

The most obvious example is browsing history - this would be much more powerful if it was integrated with your search engine. The problem is, uploading personal information to a web server is arguably even more insecure than storing it on your local computer.

Until privacy is improved on the internet, browsers will have to retain certain functions, like browsing history.

Conclusion

Experience has taught me that a few, simple, deep principles are always far better than many shallow ones. When I see something very complicated, I know it's just as likely to be a weakness in the design as in my understanding.

So it's great that browser makers are able to simplify their products, while extending their features.

Harry Potter technology: animated paintings

There's a great scene in the film Harry Potter and the Order of the Pheonix when Hogwart's caretaker, Argus Filch, is taking down an old tudor oil painting. As he twists the painting to remove it, the men in ruffs get angry, shaken from side to side, and eventually fall off the bottom.

It's visually stunning, but also emotionally engaging - it gives the viewer a real connection with the men in the painting. Imagine owning a photo frame that did this!

In fact, it must already be possible to create this effect for real, perhaps using the iPhone (since it has tilt sensors).

Surely picture animation is the next huge area for art - a way to break out of the static image and into lifelike, arresting motion. Why shouldn't the next Lucian Freud create animated paintings?

It's also one of the first digital art forms that isn't a direct copy of an analog one - unlike photography or film cartoons, you simply can't do it using paper. And it seems an even better idea than that other Harry Potter gem, the whereabouts clock.

All we need now is an open standard that describes such animation - I don't think SMIL really cuts the mustard...

New types of computer: display and desk

For years, there have been three basic form factors for computers - the desktop, the laptop, and the PDA / smartphone. Now, finally, Moore's law and new display technology are changing things.

I'm not referring here to the iMac, Tablet PC or iPhone - these follow traditional factors, albeit in a new way. I'm also not talking about the underlying functionality, which is converging for all factors towards internet access, phone, and camera.

Instead, I'm saying computers will start to flourish in new environments. There are two in particular I've got in mind - the display, and the desk.

Display

We'll soon see many display computers for the home. These are computers used for displaying information or art - usually wall-mounted, or sat on a table or mantelpiece.

The first examples are digital photo frames. Already, some are adding Wifi, to display photos from a home PC or from a photo sharing website like Flickr.

Within a few years, they'll have touch screen web browsers too, enabling them to display any website.

Imagine a display in your kitchen showing a clock, your up to date calendar, the latest weather information, and the news headlines.

Or imagine a display in the living room connected to the British National Gallery website, rotating through their art collection.

Or even imagine a display on the bedside table, showing a clock during the night, and tuning in to a TV to wake you up with the morning news.

Although they'll be standard web browsers, people won't use them much to browse - mostly, they'll be connected to a single website, refreshing regularly to show the latest content.

Desk

When you're in the office, you want more than a computer desktop - you want a computer desk. Watch the Microsoft Surface demonstration to get the idea, and imagine if your entire office desk was a touch screen computer. This replicates the physical paper and files scattered around your desk for the virtual world.

Personally, I can imagine a computer desk being easier to use at an angle rather than horizontal, so you could reach documents further away.

I'm sure the software and processing power for this is here today. We might have to wait a while for the display technology, though - after all, it requires massive, durable, high resolution touch screens. But the first prototypes are coming out now.

Fitting our lifestyles better

People have been talking about the digital home for a long time. Finally, the vision is becoming clearer - touch screen web browsers, connected to personalized services in the cloud (the home server was a red herring).

Now we have this vision, and most of the technology required to achieve it. It's a question of fitting it to our lifestyles - whether it's in the office, the kitchen, or the living room.

SVG as image format

There are two common methods for adding SVG to a page - inline, via <svg>, and externally, via <object>.

The <object> element is bad. It's not semantic - it may as well be called <other> or <miscellaneous>. Although useful in the short term for displaying SVG, I would hope that this use will diminish.

The inline <svg> element is also bad, for the same reason - it's not semantic. It's the equivalent of having a <jpg> element, rather than using <img> - it's named after the format, rather than the purpose.

From the semantic perspective, there are three potential uses of SVG.

• Foreground images: use <img src="x.svg"> to point to an SVG file
• Background images: use CSS "background-image" to point to an SVG file
• Inline with connected DOM: use <iframe> to point to an SVG file.

These are much better because they re-use existing semantic elements.

Unlike foregrounds or inline images, backgrounds should not enable any user interaction - events (e.g. mousedown), hyperlinks, pseudo-classes (e.g. :hover), etc. Some people say javascript should be turned off - this might be a rough and ready first implementation, but some javascript might be appropriate (e.g. random placement of shapes, or animation), so long as the "no user interaction" rule is followed.

The other advantage of <img> and CSS background-image over <svg> is that you don't need to use XHTML. Standard HTML gets round a whole series of issues with mime types, browser control, and backwards compatibility.

Advantages of SVG as image format

SVG images fill a lot of gaps with HTML styling:

• rounded rectangles, circles, and any polygon
• fancy borders (arcs, swirls, etc)
• opacity, color gradients and filters
• shape hyperlinks and :hover, rather than pixel maps
• interaction via the DOM (for foreground images)
• scaling of background images multiple backgrounds (in one SVG)
• background text (e.g. graffitti, murals, etc)
• intricate website 'themes' to each page

The possibilities for graphical designers are huge.

Browser support

I'm very pleased to see that the next version of Opera will support SVG images via <img> and background-image. Unfortunately, it's not on the schedule for either Firefox 3 or Safari 3, although it's an aspiration for both teams.

There are four possible methods of using SVG in a webpage - <svg>, <object>, <img>, and CSS backgrounds.

The SVG implementation status for Firefox and Safari is marked at around 55%. Personally, while they only support two of these four methods, I'd hold them at half this - 22%.

Replacing F1 through F12 with browser keys

Traditionally, keyboards come with function keys - F1 through F12.

These keys are all wrong. It's not clear what they do, in any given application or OS. They take up valuable space and in my experience merely add confusion - they're way too abstract.

I would advocate the following four keyboard buttons instead:

• Home
• Back
• Forward
• Refresh

These actions have proved their worth on the web. They're simple, easily understood, and very powerful. They exert pressure on the software developer to program the right way.

Putting them clearly on the keyboard would make web browsing a much better experience, by reducing unnecessary use of the mouse.

It would also increase consistency with smaller devices, such as the iPhone or Blackberry - even if you don't have a QWERTY keyboard, you still need to navigate around.

And other applications - such as email on Blackberry - could easily use them too, bringing a standard user interface framework to computers.

HTML audio and video

HTML 5 will introduce new <audio> and <video> elements, for including these objects on a web page in a simple, standard way. Just as Netscape originally became successful due in part to the new <img> element, you can expect browser makers to quickly implement and take advantage of sound and video.

Currently, sound and video is only available using the general purpose <object> element and various non-standard techniques for each plug-in (QuickTime, Silverlight, Flash, etc).

The new elements have several benefits:

• accessible - to search crawlers, the visually impaired, etc
• standard API - consistent DOM, javascript, and HTML elements & attributes
• standard user interface - for play, pause, etc
• integrated with the web page - part of the DOM, controllable by javascript for play, pause, volume, etcclear scope - just for video and audio, not for e.g. vector graphics

Possible uses of them include:

• Web page control of play, pause, fast forward, etc
• Web training videos with multiple choice tests on completion
• Video SVG filters (e.g. guassian blurs)
• Synchronized subtitles and sign language

Revealingly, both Flash and Silverlight don't neatly fit into this picture. They don't only enable video - they also handle vector graphics and html-like text. I suppose you could use the <video> element for Flash videos, and the <object> element for Flash graphics and text. But ideally you'd instead use html for text, and an element like <svg> or <vml> for vector graphics, in order to maintain the benefits above.

Once you've got normal html, <video>, <audio>, and <svg>, is every type of multimedia covered? No - there's still a need for interactive gaming user interfaces and 3D, for a start. But you're much further, and there's always the <object> element for missing pieces.

Voice Browsing

There's a famous story about the Microsoft developers perfecting voice recognition for Windows - it worked great until one of them visited a friend's video website of someone shouting "Start - Run - Format C - OK".

It's a great tale, but it's also a fundamental security issue that could derail most attempts at voice control of the client. The microphone should only be available to the application with focus (and not available for OS commands), and for privacy reasons, applications should always make it clear when they're listening and what they're likely to do with the data.

These rules are pretty restrictive - but they fit very neatly onto the web!

The vocal web

The web was designed with accessibility in mind - so that people with visual impairment could still access the web, by using voice browsers that read pages out load. There are even rarely used CSS styles for controlling vocal pitch, volume and tone.

But the web isn't just for people to read data. What's missing is a standard way to write data using speech - especially filling out the standard HTML <input> and <textarea> elements.

If this were possible, developers could create the following:

• Mobile phone search engines - just talk to Google!
• Dictated web email, blogs, or private notes
• Full use of most applications - e.g. Amazon or eBay - for the visually impaired

Vocal HTML

Following the security rules above, any website could be speech-enabled in three steps:

• Users adjust their browser settings to allow speech input (this could be a default on mobile phones)
• Developers prompt speech by styling input boxes with CSS 2.1 "cue-before" and "cue-after" styles
• Browsers vocally prompt form submission when they reach a submit element.

That's it!

There are two methods to do speech recognition:

• Client-side: A browser plug-in converts speech to text, places the text in the relevant HTML element, then submits the form on request.
• Server-side: For POSTed forms, browsers simply attach an mp3 or audio file for translation by the server

Despite the history, still lots of opportunity

Voice recognition has been talked about for ages, but it's still a niche - many people probably still think it's a distant dream.

But that will change, especially with the rise of the internet and mobile phones. And when it does, it won't only be the visually impaired that gains; it will be anyone accessing the internet without a good keyboard.

Paper versus computers

The paperless office has been an IT dream for decades. But despite huge leaps in technology, it's still permanently a few years out, and most people prefer to use both, depending on the scenario.

In order to understand why, and figure out whether, why or when this will ever change, I've listed the pros and cons for computers and paper.

Feature	Best medium	Automated
Distribution costs	Computer	1990s
Marginal costs	Computer	1990s
Document Copy	Computer	1990s
Validation (spellcheck, form values)	Computer	1990s
Workflow	Computer	1990s
Document Editing (delete, move sections around, etc)	Computer	1990s
Store & search	Computer	2000s
Accessibility	Computer	2000s
Collaboration	Paper	2000s
Doodles	Paper	2000s
Handwriting	Paper	2000s
Reading quality	Paper	2010s
Foldable	Paper	2010s
Disposable	Paper	2010s

You can convert formats from computer to paper by printing, and vice versa by scanning. This helps you gain the benefits of that medium, but the conversion process is not perfect.

Recent changes

• Accessible - search engines can now crawl documents and automatically extract important data, because of open formats such as HTML. Browsers can display data according to the user profile (e.g. large fonts).
• Store & Search - search engines have made a massive difference to tehe ability to find documents, and online services such as Photobucket and Google Documents enable online storage of information

Likely to change in the next five years

• Doodles - via pen / touch interface and standard vector graphics (Flash, Silverlight or SVG formats)
• Handwriting - via pen / touch interface, with OS support
• Collaboration - office suites and content management applications will be integrated with new collaboration features such as Wikis, Blogs, and Voice over IP.

So, soon you will scribble much fewer notes and diagrams on bits of paper as it becomes much more natural to use computers for these scenarios. However, you'll still need to print documents out to read them in high quality, since I can't see display technology getting as good as basic paper.

Paper is still better than monitors in many ways. When was the last time you quickly doodled a diagram on your computer? Or scrunched up your monitor to fit in your pocket? But videos, hyperlinks, storage, and search are all much better on a computer. With the advent of wikis, blogs, instant messaging, and other technologies, it's getting easier to work together on computers too.

Paper will only be eliminated when computers have the edge for every feature and every person. This is not likely to happen soon, and in any case most people are very happy working in a world that combines the two.

Scrolling versus Paging

When it comes to the oldest war of the technical formats, you can forget VHS versus Betamax. There's one still raging after more than three thousand years - scrolled versus paged document displays.

Ancient and Medieval documents

In Ancient Egypt, display technology was based on papyrus - the long thin reeds lending themselves to being rolled up into scrolls, rather than sheets. But papyrus decomposes quickly, especially in colder climates, and the Romans invented parchment in the first century BC, made from animal skin, which was more easily folded into paged format.

Parchment was of a more consistent quality, and kept better, but a key advantage was its accessibility - it's much easier to quickly turn to the middle of a book than to the middle of a papyrus scroll.

But pages really came into their own when the printing press was invented, back in the 1400s. Pages could be much better printed on than scrolls, so the quality and efficiency of pages lept ahead of scrolls for more than five hundred years, especially with standard page sizes.

Pros and Cons: the 1950s

I'd summarise three reasons why, by the 1950s, pages were winning the war with scrolls

• Lower costs, higher quality - due to the printing press
• Better accessibility - easier to 'flick through' a book than a scroll
• Standard formats - e.g. letter, A4, and broadsheet sizes introduce economies of scale

Office Computers

The first office computers didn't really challenge the culture of pages. Word processors and presentation software are both inherently paged, because they're designed to be printed onto paper.

But there was a problem - users had differently sized monitors - so in order to fit the page properly on the screen, scroll bars were added.

And computers began to be used in new ways. No amount of paper can replicate automatic formulas in spreadsheets, and emails are not constrained to a certain page width and height. Spreadsheets and emails are scrolled, not paged (that's why they are a pain to print out).

Browsing the web

Most obviously, the web has changed our culture towards scrolling. Browser makers rely heavily on scroll bars and re-arranging content to fit screen size, especially as the mobile web increases display diversity.

There are some interesting exceptions. Google search results are paged (with the top 10 results on the first page, the next 10 on the second page, etc), but this is done to prevent massive amounts of unnecessary data reaching the user, rather than to fit to a certain page width and height.

And many web designers stick to the old mentality, deliberately forcing layout (particularly width) to a certain size. Lazy designers create flash animations or tables that are too big for many screens.

On the web, the word "page" is often used to mean "the document at a given URL". But it's not usually a page, it's a scroll.

Pros and Cons: the 21st century

Nowadays, the cost and quality of paging software is equal to scrolling technology - it's a simple matter of fixing some of the code. Several other factors are more important, and it's clear that scrolls have the advantage:

• Accessibility - it's easier to 'flick through' a scroll than a page, and it's also easier to adjust to user needs (e.g. increased font size for the visibilty impared)
• Screen diversity - screens come in such a variety of shapes and sizes that forcing a fixed page width and height will inconvenience many users

There will always be a few exceptions that prove the rule - for example, pages to browse Google search results, or niche applications designed for a small set of users with the same screen sizes.

But it's clear that two thousands years after their last peak, scrolls are once more the leading display technology.

The homepage problem

There are lots of homepages out there - e.g. Windows / Unix desktops, browser homepages, Google personalized homepages, phone & PDA homepages - but I've never really seen one that satisfied me.

This is partly due to the offline / online schism. My windows desktop doesn't show my online documents, and my Google homepage doesn't link to My Documents and the Control Panel. The taskbar competes with browser tabs for flipping between applications. The start menu has been totally left behind by web applications.

But it goes deeper than that. My phone homepage has 12 clear pre-defined options - phone, calendar, contacts, etc - but desktops can't seem to do this (since Windows 3.1, anyway). Desktop icons are much too vague - every web link has the same icon, of my browser. And we can't give a consistent user experience - every different computer, PDA, and phone has a different homepage, even for the same user.

The only real solution is to further integrate the browser with the operating system. This may bring back bad memories of Microsoft in the 90s, but the internet has changed the rules again.

I've listed five recommendations to address the homepage problem:

Reclaim the browser homepage Your browser homepage should be the same as the desktop homepage - you shouldn't be allowed to change it to anywhere else, whether Google or Myspace.

Standard homepage options On the homepage should be a standard set of options:

• Favourites (editable)
• History (with options to delete it)
• News feeds (editable)
• Browser options (editable) - e.g. view settings, security settings. Some of these will take you to other web pages for more detail.

Replace icons with thumbprints and widgets Today's icons have had their day. It no longer makes sense to store five word documents or websites on the homepage, with no visual way to distinguish between them.

Instead, use thumbprints to show the contents of a documents. Keep the description or filenames underneath the thumbprint.

Another common use of icons is to open an application (rather than a specific document). Internet applications, such as Gmail, Skype and Flickr, should be allowed to display widgets, rather than just icons. These will appear as mini homepages in their own right, highlighting important information and allowing the user to click to open the application to explore further. For example, Microsoft may create a widget on the homepage to cover the basic Office Suite, showing recent documents, highlighting important features, and graphically advertising the applications. Best practice would be to allow the user to select from large, medium and small widgets for each application, to control real estate.

Links to files and control settings (in HTML)

Also on the homepage, there should be hyperlinks to take you to windows explorer, help pages, a comprehensive list of local applications, the control panel, and various widgets.

All of these should be in HTML format, so they open in the browser. This includes windows explorer, as per Google Desktop, so that it can show web content alongside local content. The objective is to improve navigation (those forward and back keys), make the user experience more consistent, and allow web content to be integrated with local content.

Use RSS feeds to synch homepage

All items on the homepage - whether thumprints, widgets, or user options - should be stored as a set of RSS (or Atom) feed, and synched with an internet provider selected by the user. When offline, the locally cached version is displayed. When online, the fully up to date version is displayed.

The homepage problem and the internet
There's no doubt that no one has really solved the homepage problem yet. The answer lies in bringing the power of the web to the desktop.

Browser Acceleration and Orientation

Perhaps the most revolutionary thing about both the iPhone and the Wii is their ability to detect acceleration and orientation. The iPhone automatically converts from landscape to portrait mode depending on which way you hold it, and the Wii was designed to allow tennis strokes, golf shots or boxing matches simply by moving the control.

Ideally, the same functionality should be available on the web. You can imagine a browser that

• Rotates between landscape and portrait mode, depending on device orientation
• Scrolls up, down, left and right based on device acceleration

But what if the web developer wanted access to the same information? You can imagine websites that

• Display maps, orientated to the direction the device is pointing at
• Provide games based on "pointing", e.g. golf games
• Provide games based on "moving", e.g. tennis games

The web developer will want to be able to access acceleration and orientation information, and use it to alter HTML, SVG or Flash.

Unfortunately, there's no standard for this on the internet. There is, however, a fairly obvious place where it could go - the javascript event object. In the same way that this object stores the current mouse location (for devices with mice), you can imagine it also storing x, y, and z axis acceleration and orientation (for devices with accelerometers and gyroscopes).

One interesting question concerns privacy. Does it matter that someone could track the orientation or acceleration of your phone, if you were logged on to their website?

Personally, I can't see this happening soon - the demand just isn't there yet. But once phone browsing takes off, after two or three years, it will be very interesting to see how this field develops.

In the meantime, the W3C should look at extending their standards to allow for acceleration and orientation. And phone browser providers, such as Opera, should consider upgrading their browsers to take advantage of the latest in user interface design.

Too many browser menus

Despite the huge recent improvement in browser interfaces – tabbed browsing, inbuilt search, RSS favourites, etc – I still think there’s a long way to go. In particular there are way too many inconsistent menu options.

Using Internet Explorer or Firefox, running on Windows, there is

• A title bar menu allowing minimize / maximize / close
• A standard menu, e.g. file / edit / view / tools
• A set of browser commands, e.g. back / forward / refresh / homepage
• An address bar
• A tabs menu

What’s more, they’re all inconsistent – some open new web pages, others open dialog boxes, others execute some action on the existing web page.

The result is that half the screen gets taken up by confusing options and buttons before the content itself appears.

So what hope does a web application like Google Spreadsheets or SAP (themselves containing another two or three menus) have?

I think it’s time for a rethink. So I’ve listed some principles to re-organize the browser.

1. Use a ribbon bar The old file / edit / view / favourites / tools / help menu should be replaced by a new ribbon bar. Ribbon bars are simple, clear, and effective, and would merge the currently overlapping file menu and buttons underneath.

2. Hardware for common options The most selected browsers button are undoubtedly “back”, “forward”, “refresh” and “home”. In fact, they’re so common, and natural, that they deserve their own hardware.

The buttons F1 through F12 on the keyboard are rarely used and even more rarely understood. Why not replace them with “Back”, “Forward”, “Refresh” and “Home” buttons? This would free up screen real estate, and avoid unnecessary and error-prone mouse use.

The benefits are even bigger on touch-screen interfaces like the iPhone – you can imagine the standard buttons fitting underneath the screen.

It’s likely to be hardware vendors pushing this change. But Microsoft managed to get the new “Windows Start” button implemented, and a similar trick now could allow them to claim innovation and alignment with the internet.

3. No dialog boxes Dialog boxes crop up throughout browsers, especially in the tools and options menus. They’re not accessible – you can’t change font settings or view source – and they’re visually confusing, since they look different to web pages.

All these dialog boxes should be replaced by web pages that open up within the browser. There should be a local web page to allow you to edit connection settings or security options. And why doesn’t browser help open in the browser?

4. Reclaim the home page This is the most controversial principle. Your homepage should be set by the browser – you shouldn’t be allowed to change it to anywhere else, whether Google or Myspace.

On the homepage should be:

• Your favourites (editable)
• Your RSS feeds (editable)
• Your history (with options to delete it)
• Browser options (editable) – e.g. connection settings, security settings, view settings. Some of these will take you to other pages for more detail - see principle 3 above

5. Lose the title bar What use does the windows title bar have, for browsers? It tells you what the title of the page is – but so does the tab bar. It allows you to minimize, maximize, or close a window – but so does the tab bar.

Once you’ve got tabbed browsing, there’s absolutely no use for the title bar. In fact, it gets in the way – not only does it take up screen real estate, but it makes mouse control trickier (it’s far easier to select a button at the top of the page than one 12 pixels down).

So let’s get rid of the title bar!

Simpler, clearer, more concise So there you have it – five principles to get rid of the clutter of modern browsers. There’s plenty of innovation still to come in browser design!

HTML menus

Menus - like the file menu at the top of every application - have always been tricky to code in HTML. They involved reams of javascript and endless workarounds for the deficiencies in each browser.

The HTML 5 working group are talking about a new HTML tag to enable menus. This would be great, but for now, why not just use the ordered list tag <ol>? After all, that's what menus are - ordered lists.

So it's a great relief to see menus done properly - with no javascript in sight, just the <ol> tag and some CSS.

But before we congratulate ourselves on proving the power of HTML yet again, it's worth asking what menus are for in the first place.

I count three uses:

1. Site navigation hyperlinks (e.g. the left hand pane of http://www.microsoft.com/sql/default.mspx)
2. Standard application menus (e.g. the MS Word file menu)
3. Context-sensitive application menus (e.g. right mouse button options)

On the web, most people just think of the first use, because web pages still aren't seen as applications in their own right. For those using web-based spreadsheets, however, uses 2 and 3 are more important - rather than navigating to different pages, the menu options manipulate the existing page.

Most desktop-based applications are just as poor at menus as web-based ones. Even commonly used ones - such as Internet Explorer itself - do a bad job here. There is a very complicated File-Edit-View-Favorites-Tools-Help, there are the standard buttons (back / forward / refresh), there is the address bar plus optional extra bars, and only then do you get the page itself, which will often have its own menus too.

So Microsoft is due some praise for recognizing this, and innovating with the new Office 2007 ribbon, which combines uses 2 and 3. Maybe they did it to stay ahead of websites like Google Docs; if so, the effect was diminished by successfully mimicking it in the MS Office website!

How is a ribbon menu best achieved on the web? Well, in theory, just using additional <ol> tags and CSS. In practice, this is a nightmare without a properly CSS-compliant browser; even Internet Explorer 7 falls somewhat short. But never underestimate the resourcefulness of web developers - there's plenty of innovation to come using existing tools. I wouldn't be surprised if ribbons started appearing on websites very soon.

Context-sensitive menus are by far the rarest on the web. In Google spreadsheets, an HTML menu opens up when the right mouse button is clicked on a cell. To be frank, I'm not sure this is good practice - think of smartphones, PDAs, Tablet PCs, Apple Macs, and voice-activated browsers - none have a "right mouse button". And a huge proportion of users never think of using the right mouse button (see Jon Udell's discussion on saving web pages) - instead, context-sensitive menus should probably appear as part of a ribbon.

In fact, it's pretty easy to code context sensitive menus using simple javascript. For example, you could dynamically swap out the contents of a menu <ol> tag based on browser focus and DOM events.

So I predict a migration to menus based on <ol> and CSS.

Continued user interface innovation will make menus as friendly and accessible as possible. Menus inspired by the Office 2007 ribbon will become more popular for true web applications, like Hotmail or Google Spreadsheets.

And context-sensitive menus will appear through the web, as developers realise the power of simple HTML, CSS and javascript.

Touch-screen displays on the web

Recently I saw a fabulous demo of touch-screen displays in action.

In the demo, the user is shown manipulating shapes on the screen with both hands - squeezing images, grabbing multiple shapes simultaneously and pushing them together, and simultaneous drag-and drop.

This reminded me of Steve Job's iPhone demo, and my comments at the time that HTML can probably handle multi-touch UIs, but javascript might struggle.

Experience tell us that developers need several different methods to handle user interaction. There should be a simple method with default behaviours, and a more detailed method giving fine control; and there should be a declarative approach for XML developers, and a procedural approach for those that prefer scripting.

It's clear that several things will have to change before websites cater appropriately for touch screen displays.

Firstly, we need more support in CSS for simple effects like drag and drop (our simple, declarative method).
Secondly, we need more declarative support for animation (fine-grained, declarative method).
Thirdly, if there is no mouse, there is no right mouse button - so we'll need to rethink the approach for context sensitive menus. Microsoft have innovated here with the new 'ribbon' interface in Office 2007 - I'll save this piece for another post.

CSS user interaction

CSS styles fit the bill perfectly for a simple, declarative approach. If we add a series of user interface CSS styles, the user gets a consistent experience, and the developer doesn't have to worry about endless code:

• draggable = "no | yes" - elements with this style can be moved across the page via user interaction.
• resizable ="none | x | y | preserveAspectRatio | all" - elements with this style can be re-sized via user interaction, along either or both axes.
• zoomable = "no | yes" - elements with this style are containers (e.g. <html> or <div> tags) and zooming commands are available on the contents of the container.
• pannable = "no | x | y | all" - elements with this style are containers (e.g. <html> or <div> tags) and panning commands are available on the contents of the container (e.g. panning around Google Maps). This could be scrollbars, or some other user interface method, depending on the browser.

For each of these styles, the exact user interaction method doesn't matter to the web developer - it could be a mouse, a touch screen, voice commands, or something else, as set by the browser or the operating system. In some cases (e.g. touch screen) there could be multiple user interactions at the same time; that's all handled by the browser. All the web developer need care about is setting the appropriate styles.

Declarative animation

Anyone who's tried to program drag and drop knows that the DOM is painfully awkward at tracking certain user interactions - but imagine dragging two objects on a touch screen simultaneously! Which event object would you use?

The real pain here is for events like mousemove. These are "continuous events", a contradiction in terms which reveals the flaw in the underlying approach. For continuously evolving features, languages should use Functional Animation instead (see my previous post).

Imagine if the browser maintained user interaction state (mouse position, touch screen location, etc) in a read-only XML file directly accessible to developers. For example:

<pointers>
<pointer status="active" screenX="100" screenY="100" elementref="div0" relativeX="5" relativeY="5"/>
</pointers>

For the mouse, there would only be one <pointer/>, with "active" status when the mouse was down, and "inactive" when up. For touch screens, there would any number of <pointer/> elements (including zero), each representing a finger or stylus touching the screen. The elementref attribute stores a reference to the element that the pointer is currently over, the relativeX and relativeY commands store the location relative to this element, and the screenX and screenY elements store the location relative to the screen.

Once you have this file, you can do functional animation based on it. For example, using the XForms <bind> tag:

<bind infoset="id('img1')" calculatewhen="//pointer[@elementref = 'img1' && @status='active']">
  ./@css:left = "//pointer[@elementref = 'img1']/@screenX;
  ./@css:top = "//pointer[@elementref = 'img1']/@screenY;
</bind>

which once activated, binds img1 to the evolving location of the mouse / stylus.

As you can see, this approach avoids the need to use javascript at all - event handlers and declarative functional animation are enough.

Touch screens are the future

Computer mice have been around for so long that it's tempting to see them as a permanent fixture in computing. But actually they're pretty uninuitive - remember seeing someone using a mouse for the first time?

As touch screens spread, web developers will be faced with an interesting set of challenges, which are best overcome using a few simple CSS tags, and a declarative approach.

The iPhone is a Tablet PC

No one would have predicted it, but the two biggest IT innovations of the last 6 months have been new user interfaces - Nintendo's Wii, and Apple's iPhone.

It seems the old metaphors of mouse and keyboard are getting some competition.

I particularly like the iPhone's 'pinching' feature - you can use two fingers to squeeze a window down from both sides. It's a natural instinct, yet it would require using two mice simultaneously on a desktop PC!

Actually, the keyboard is safe, for now. It appears, albeit in software form, in the iPhone. It's the mouse that's under pressure, and not before time - there are serious mental gymnastics in moving an object across a horizontal surface, in order to direct another object some distance away at a different scale on a vertical surface.

Touch-screen interfaces are not new - the iPhone is really a small Tablet PC with phone functionality. Steve Jobs made this clear when he boasted that it gets rid of the mouse - most phones don't come with mice! The real innovation is in merging a Tablet PC with a phone.

So, three predictions - first, Microsoft will take the opportunity to remind everyone about the Tablet PC, and who invented it. Sometimes, Apple learns from Microsoft, not the other way round!

Second - sales of Tablet PCs will finally take off once people get used to the iPhone. They will want a larger version that opens their Word Documents and allows them to synchronize with corporate applications. I don't believe anyone has found laptop nipples or trackpads perfect, and the Tablet PC will provide a solution. Microsoft will benefit from this (unless Apple releases a high-end iTablet? ...)

Third - I'd be very surprised if we didn't see a Tablet variant of Windows Mobile coming soon. After all, Microsoft has had the software ready for a couple of years.

The end of the mouse is nigh, and user interfaces are evolving rapidly.