Bill Gates on Personal Computing

You can tell Microsoft still don't understand the web, because they make comments like this:

IBM is no longer at the center of the computer industry, he asserted, for two reasons. First, the industry is now centered on personal computing. "As much as IBM created the IBM PC, it was never their culture, their excellence," he said. "Their skill sets were never about personal computing."

Actually, personal computing dates from the 1990s. We're now entering a different era - call it "web computing" or "social computing", where the focus is on collaboration and networks of people.

Facebook and Flickr are not personal computing - the world doesn't sit on your C: drive any more.

Times have changed, and you have to ask whether Microsoft's "culture", "excellence" and "skill sets" are keeping pace.

Multi-touch and DOM events

The iPhone has highlighted a weakness with the HTML DOM - it has keyboard events and mouse events, but nothing for touch screens.

You can try to kludge touch screens using the mouse interface, or if you're simply looking to code drag-and-drop or resize, you can wait until HTML 5 comes along and use its new attributes "draggable" and "resize".

But if this doesn't work, you're stuck. And for other user interfaces, such as accelerometers in the Wii, the situation is even worse.

Sensors

So it's time to resuscitate my sensors proposal, which provides a framework for any sensor, not just a touch screen.

What it takes is a single new HTML DOM javascript function - document.sensors() - which returns an XML document of all sensors that the page has access to, e.g.:

<sensors xmlns="http://sensors.org/namespace">
<keyboard shift="" ctrl="" alt="" ins="" value="ab"/>
<mouse x="20px" y="30px" left="down" right="none" middle="none"/>
<touch pressure="30" x="150px" y="50px"/>
<temperature value="23C"/>
<video src="file://c/program%20files/webcam/webcam.mov"/>
<accel x="2" y="0" z="0"/>
<location latitude="37.386013" longitude="-122.082932"/>
</sensors>

This document is constantly being updated with the latest values - here, you can see there is a keyboard, mouse, touch screen, thermometer, webcam, accelerometer, and GPS. The namespace would define a standard basic set of sensors, but I'm sure that extra ones could evolve over time.

So, any web designer can access the latest status of the environment, in an extensible, standard way, using javascript.

In the case of a touch screen, you could place an onchange() listener on the element, to be notified of any movement. You could also watch out for multiple elements, if several fingers hit the screen.

Security

Some of the sensory information is private, so various security measures should be in place to protect prying eyes from getting access to it. I've listed a few ideas:

• Only pages with focus can see the sensory data
• Browsers allow site-specific permission settings for each sensor
• Webcam and background sound files can't be posted back to web servers

Conclusion

The simple suggestion above provides a much more comprehensive approach to human computer interaction than today's HTML DOM events. It handles not just keyboards, mice and touch screens, but all types of sensors - from thermometers to accelerometers to webcams - in a standard, extensible way. And it brings the web into many new areas - from security cameras to production line control to satellite navigation.

CSS and XPath selectors

Many people have noticed the similarities between CSS selectors and XPath - and it's fair to say that XPath is far more powerful.

In fact, XPath can do pretty much anything that CSS can, plus much more.

select the parents of all paragraphs:

//p..

select alternate list items:

//ul/li[position() mod 2 = 0]

select table cells with a value less than 10

//tr[number(td) < 10]/td

Yet another CSS wish - XPath stylesheet selectors

No existing browsers have XPath enabled inside stylesheets. But wouldn't it be great if they did?

All it takes is a new CSS selector - XPath(string), where string is an XPath expression.

For example, selecting paragraphs containing the word 'Chris':

XPath("//p[contains(., 'Chris')]") {border: 1px solid black}

Of course, XPath doesn't itself handle pseudo-elements or pseudo-classes like :hover - but we can mix and match the XPath function with other CSS (e.g. colouring any hovered paragraphs containing two hyperlinks):

XPath("//p[count(.//a) = 2]") :hover {background-color:red}

And it can work with other selectors too, e.g. finding <ul>s directly underneath any <p> element, and shading them alternate colours:

p XPath("ul/li[position() mod 2 = 0]") {background-color: white}
p XPath("ul/li[position() mod 2 = 1]") {background-color: silver}

Following the usual CSS rules, if the XPath function contained mis-formed XPath, the style could be ignored. Also, it would be ignored if it returned anything other than element nodes (i.e. no attribute nodes or strings). And the default starting node of the XPath query is the document element, unless clarified by preceding CSS selectors.

Why not implement it?

As you can see, just by adding a single new selector, the CSS language is extended in so many ways.

XPath has already been agreed as a W3C recommendation, and has already been implemented in the major browsers.

I therefore see no good reason, other than inertia, why such a powerful new feature can't be added as soon as possible!

Browser User Interfaces

You can tell we've really begun to understand the web in the last five years, because while browsers have got more powerful, their user interfaces are now simpler and clearer.

This isn't always the case; think back to the office suite wars in the 1990s, when Microsoft and others added endless cluttered options, menus and functions to every release of their word processors and spreadsheets.

See the screenshots below of IE4 versus IE7 - the later release is much more streamlined.

The next version of Firefox, v3.0, is simplifying even further by merging History and Bookmarks into a much more powerful, unified interface: Places.

How far can this go?

Even further! I've listed some ideas below which would simplify and extend whole areas of browser design:

Calendar

Show the current time in the browser bar. When clicked, it opens your pre-defined calendar site (e.g. google calendar), and you can also drag text onto the clock from a webpage (e.g. "meet in Canary Wharf tonight at 7pm") to add events to your calendar.

Location

For devices with GPS. Show the current location in the browser bar (e.g. "Canary Wharf"). When clicked, it opens your pre-defined map site (e.g. google maps), and you can also drag text onto it from a webpage (e.g. "Canary Wharf") to view that place in a map.

Style

Subscribe to an external CSS stylesheet to control default and override settings like font name, font size, link underline, and audio volume. The stylesheet is cached locally and can be edited in a user-friendly manner via the subscribed website. Mozilla could set up their own stylesheet site as a start.

Page Analysis

Display page file size, security information, "view source", "view HTTP headers", error messages, and spelling or grammer checks. This could be done by posting the page to a subscribed analysis website, if security allows.

Tabs and Windows

Drag hyperlinks onto the tab bar to open them in a new tab, or drag them outside the browser window to open them in new window.

The goal is to reduce the browser interface down to a very few, powerful functions.

There's a common theme with the ideas above: functionality that used to be part of the browser - e.g. default fonts, page information - is now provided by a website. You enter the website details in the browser, and the browser hands over the appropriate information.

Websites are likely to be much better than the browser at certain tasks, because of the speed of application development and deployment on the web, the power of HTML and mashups, and the funding of Silicon Valley. It also allows browser makers to concentrate on page rendering, their core competency.

Security Considerations

Of course, the problem with handing features over to websites is security.

The most obvious example is browsing history - this would be much more powerful if it was integrated with your search engine. The problem is, uploading personal information to a web server is arguably even more insecure than storing it on your local computer.

Until privacy is improved on the internet, browsers will have to retain certain functions, like browsing history.

Conclusion

Experience has taught me that a few, simple, deep principles are always far better than many shallow ones. When I see something very complicated, I know it's just as likely to be a weakness in the design as in my understanding.

So it's great that browser makers are able to simplify their products, while extending their features.

Folders and File Categorisation

Microsoft's Vista, like XP, comes pre-built with folders called "My Documents", "My Photos", "My Music", "My Videos", and even "My Scans".

They're useful for about 10 mins, and then you run into issues - it stops you from categorising your files any other way, for example by project or by customer. Something is very wrong.

The best solution is multi-dimensional categorisation - being able to view your files by document type, project, customer, or any other cut.

However, since WinFS was removed from Vista's feature list, it isn't possible to do this. But for online storage, like Google Apps, these features should be straightforward - right?

Google gets it wrong too

Wrong. There's Google Docs & Spreadsheets, Google Photos (a.k.a. Picasa), Google Videos (a.k.a. YouTube), Blogger, and many other tools.

But there's still no way to group your files together into a "project folder".

They've made a few stabs in that direction - for example, you can attach Picasa images to Blogger entries, and Google Search in the US now works across file types. But in general, it's even further behind Microsoft.

How it should work

The obvious technology for combining all these file types is HTML.

Imagine creating an online project homepage, with links to the appropriate photos, videos, blogs, emails, spreadsheets, or even calendars. You could either upload new files, or link to ones previously uploaded to sites like YouTube.

The key point is that every perspective - project, file type, author, etc - should have its own homepage, allowing you to view or edit appropriate files. Many of these files will appear in several different places - the author's homepage, plus the project homepage, plus YouTube - but that's ok, because it reflects life!

Why WinFS didn't work

Looked at this way, it's obvious why WinFS failed, and Vista still has those pre-built folders like "My Photos". You simply can't categorise files without web technology - the URL, the hyperlink, the homepage, mashups, even the Wiki.

It's also a reminder that we're still at the foundation stages of computing - categorising files is a basic requirement that no one has truly accomplished yet.

Data mismatches

Over the years, each of the traditional system tiers - database, web server, browser - has grown more powerful, reliable and manageable.

The problem is, they still don't work with each other well!

I still haven't seen an elegant way to create object oriented code from queries of relational tables (although LINQ comes closest).

And using objects to manage semi-structured, hierarchical HTML doesn't work nicely either - that's why the DOM is so ugly.

Finally, placing (X)HTML neatly in a relational database is very awkward, although that hasn't stopped vendors from attempting it.

Different forms of data

That's because they all use different approaches to model data - table, object, and document.

Each approach has many advantages, each requires a different technical skill and personality type to use, and each works best in different circumstances. Unfortunately, they don't work together particularly well.

Can this last?

At the moment there are a few creaks, but no cracks. The creaks are

• The success of scripting languages like PHP in managing documents, rather than formal object oriented code
• Buzz around REST, which uses URLs and HTTP to store (and even edit) data, hiding the underlying relational database
• Relational databases increasingly outputting XML, rather than proprietary data

Another approach - REST, XQuery

It is now (just) possible to use the document approach throughout every tier. It makes code enormously easy to write and maintain, and it fits perfectly into the web.

You wouldn't want to do this for data-intensive applications, such as handling financial market data. But for document-intensive web applications, such as social networking, blogging and photo-sharing, it's perfect.

The idea is to follow the REST approach:

• carefully construct a URL for every resource important to your site
• decide which resources require create, review, update, and delete (CRUD) permissions
• enable HTTP PUT, GET, POST, and DELETE commands against these URLs

Even if these resources are eventually stored in a relational database, this approach totally shields the relational viewpoint in favour of the document.

You can even write most of the server-side code in XQuery. The advantage of doing this is that it fits perfectly with (X)HTML and REST - you can GET documents, extract the relevant parts using XPath, and insert them into page markup using straightforward inline code. No object orientation in sight!

After all, server-side code does four things:

Action	Technology
read / update a database	REST, HTTP
get / set HTTP headers	XQuery functions
manage sessions	XQuery functions
construct HTML output	XQuery, HTML

It's very useful if you've got a huge number of URIs (as per REST) - you just have a central application that parses the URI and returns the appropriate mashed-up resources. XQuery is good at this parsing and returning.

One for the future

Unfortunately, the technologies behind REST and XQuery are still very immature and there isn't much support from libraries, documentation, or tools.

And given the immense standing base of relational databases and object-oriented code, and their use in so many different areas, I can't see their support diminishing soon.

That's ok - the point is that new ideas are still bubbling forward for improving developer productivity. SQL and OOP both pre-date the web; they have survived well, but it's always worth taking a step back and asking if there's a better approach.

Harry Potter technology: animated paintings

There's a great scene in the film Harry Potter and the Order of the Pheonix when Hogwart's caretaker, Argus Filch, is taking down an old tudor oil painting. As he twists the painting to remove it, the men in ruffs get angry, shaken from side to side, and eventually fall off the bottom.

It's visually stunning, but also emotionally engaging - it gives the viewer a real connection with the men in the painting. Imagine owning a photo frame that did this!

In fact, it must already be possible to create this effect for real, perhaps using the iPhone (since it has tilt sensors).

Surely picture animation is the next huge area for art - a way to break out of the static image and into lifelike, arresting motion. Why shouldn't the next Lucian Freud create animated paintings?

It's also one of the first digital art forms that isn't a direct copy of an analog one - unlike photography or film cartoons, you simply can't do it using paper. And it seems an even better idea than that other Harry Potter gem, the whereabouts clock.

All we need now is an open standard that describes such animation - I don't think SMIL really cuts the mustard...

From financial services to financial management

I don't think banks really help me manage my money. They might store it securely, give me interest, enable transactions, and offer statements, but that's a short list of services - it's not the whole picture.

For example: what about allowing tagging and categorizing of each statement entry, so I can see how much I spend in the supermarkets or on the gas bill each month?

What about helping me with my tax? Or offering reporting and trending? Or providing financial planning tools? Or customized email / SMS alerts (or even customized financial transfers) when I reach various thresholds?

Drill-down and comparison

I've got online accounts with various different utilities and retailors, but I've forgotten many passwords and don't get around to visiting them very often.

So, imagine if I could click an entry in my statement, and be taken to the relevant website - e.g. click on an entry from the water company, to see my online account there. It just requires my bank to know the URL for common providers (or even store my usernames and passwords).

That way, my bank is truly helping me organize my finances and track my expenditure. It's moving from offering basic banking services to helping me manage my bills.

I could even pick a utility or shop, and view a timeline with all the money I'd spent there. It could link to a shopping comparison site, to help me understand the competition.

Financial Management is more than Financial Services

I don't think they understand this, but banks are a long way from truly enabling people to manage their financials. They're stuck in a decades-long cycle of improving efficiencies, and it's time to open up new capabilities for their customers.