How we make websites

Designing and building data driven dynamic web applications the one web, domain driven, RESTful, open, linked data way

For the past few months I've been touting a presentation around the BBC entitled 'How we make websites'. It's a compendium of everything our team has learned from long years developing /programmes, the recent work on /music and the currently in development /events.

As a warning there's very little original thinking in here. For those familiar with the concept of one web, the importance of persistent URIs, REST, Domain Driven Design and Linked Open Data it'll probably be old news. Possibly it's interesting to see all these threads tied up in one place!?! Maybe it's interesting to see them all from a user experience point of view?!? Anyway, as ever, it's built on the thinking and achievements of many clever people over many years who are too numerous to mention here. Although obviously I'll make an exception for Paul Clifford and TimBL. :)

The presentation is here and the (slightly) expanded text is below for the sake of accessibility and Google.

Explore the domain

This should be clear from the business requirements - it might be food or music or gardening or...

Employ a domain expert. Get them to sketch their world and sketch back at them. Concentrate on modelling real (physical and metaphysical) things not web pages - try to blank from your mind all thoughts of the resulting web site. This work should never stop - you need to do this through the lifetime of the project as you refine your understanding.

Identify your domain objects and the relationships between them

As you chat and sketch with your domain expert you should build up a picture of the types of things they're concerned with. Make a list of these objects.

As your knowledge of the domain increases you'll build up a picture of how your objects interlink. You can sketch basic entity relationship diagrams with your domain expert and keep sketching until the picture clears. Bear in mind you're trying to capture the domain ontology - this isn't about sketching database schemas. The resulting domain model will inform the rest of your project and should be one of the few artifacts your project ever creates.

Check your domain model with users

Run focus groups and speak to users. Get them to sketch their understanding of the domain and again sketch back at them. After several round trips you should be able to synthesise the expert model and the user model. User-centric design starts here - if you choose to model things and relationships between those things that users can't easily comprehend no amount of wireframes or personaes or storyboards will help you out.

Check to see if your website already deals with some of your domain objects

If it does then reuse this functionality by linking to these pages - you don't want to mint new URIs for existing objects. Having more than one page per thing confuses users and confuses Google. Try to think of your website as a coherent whole; not as a collection of individual products. And as ever, don't expose your internal organisational structures through your website. Users don't care about departments or reporting lines.

The glory will always come from building skyscrapers - the real challenge lies in decent town planning. It's more difficult to build new services that stitch into your site and stitch into the web than build shiny, shrink wrapped, self contained products.

Design your database

Translate your domain model into a physical database schema.

Source your data

Check if there are business systems in your organisation able to populate your schema. Check if there are existing websites outside your organisation you can use to populate your schema. Give preferential treatment to any websites that offer their data under a liberal licencing agreement - you can buy in data to help you slice and dice your own data but if you do this you might not be able to provide an open data API without giving away the 3rd party's business model. If your organisation AND an open data website can provide the data, consider the danger in minting new identifiers for your own data - can you easily link out / can you easily get links in?

Data licensing is one of those areas that often gets ignored in project planning. If you fail to consider it or get it wrong it can severely curtail your plans further down the line.

Pipe in your data

Whether you choose to use your business data or buy data or use open data you'll need a way of piping it into your database schema. You'll probably have to reshape it to make it suitable for publishing.

Make your models

In an MVC framework your models should contain all your business logic. This mean they should capture all the constraints of your database schema plus all the extra constraints implied by your domain model.

Design your URI schema

Your URI schema should follow naturally from your domain model. As an example if you're dealing with books and a book can have many authors then ../:book/authors should list all the authors of that book. At Audio and Music we tend to use large walls and lots of post-its to design our URIs. Add some string to show links and journeys and there's no need to ever draw another site map.

This isn't just about designing URIs for resources you link to - sometimes your pages will be made up of other transcluded resources - all of these subsidiary resources should be addressable too. It means you can easily change your user experience layer by taking out transcluded resources and linking to them instead or removing links and transcluding.

By making every nugget of content addressable you allow other sites to link to it, improve your bookmarkability and increase your SEO - cf. an individual 'tweet'. Bear in mind that some representations (specifically mobile) will need smaller, more fragmented representations with lower page weight - designing your subsidiary resources to be addressable allows you to easily deal with this requirement - transclude the content on a desktop machine, link to it on a mobile.

This is where we begin to talk about one web and REST. Each thing should be one resource with one URI - the representation you get back (whether desktop HTML or mobile XHTML MP or RDF or YAML or JSON) should depend on what your user agent asks for via content negotiation. It means I can send a link to a friend from a desktop machine, they can click on that link from a mobile and they'll get back a representation appropriate to their device. Or vice versa. One web with no mobile ghetto.

It's important not to confuse URI design with site structure and user journeys. If you're used to working on hierarchical silo sites then the URI structure often determines the navigation. This isn't true here. Think of the individual resources as tent poles - the user journeys are the canvas that gets draped over later.

It's nice if URIs are human readable. It's also nice if they're hackable. It's an absolute prerequisite that they're persistent.

Don't sacrifice persistence for the sake of prettiness or misguided SEO. URIs are your promise to the web and your users - if you change them or change their meaning you break that promise - links break, bookmarks break, citations break and your search engine juice is lost.

Remember: Cool URIs don't change.

Make hello world pages for your primary domain objects

For now all they need is an h1 with the title of the object.

Make hello world pages for your primary aggregations

For now all they need is an h1 with the title of the aggregation and a linked list of things aggregated.

Define the data you need to build each of your pages

Traditional wireframes lump together data requirements (via annotations), page layout and (by implication) document design. It's best to split these out into 3 distinct tasks. The first task is to define the data requirements.

For each URI define the data needed to build all representations of the thing. Just because the HTML representation doesn't need to show the updated date doesn't mean the RSS or Atom or RDF don't need it.

Some resources will transclude others. There's no need to define the data required for these - just reference the transcluded resource.

Build up your HTML pages and other representations

Now you know what data you need you can begin to surface this in your representations.

If you're working in HTML make sure you design your document to be semantically correct and accessible. Try not to think about page layout - that's the job of CSS not markup. Document design should be independent of page layout. In general your page should be structured into title, content, navigation - screen readers don't want to fight through calendar tables etc to get to the content.

Add caching and search sitemaps

Knowing what can be cached and for how long is a vital part of designing your user experience. Cache for too long and pages go stale. Don't cache for long enough and you send unnecessary traffic across the wires and place extra strain on your application.

Cached pages will also be faster and smoother to render in a browser. And if your users are paying for data on a mobile every extra connection means bigger bills, which is definitely a user experience issue.

An example: if you're creating a schedule page for today's TV you want to cache for performance reasons but you don't want to cache it for too long since schedules are subject to change. But you can cache yesterday's schedule more aggressively and last week's schedule more aggressively still.

Creating XML search sitemaps helps search engines know which bits of your site have been updated. Which helps them to know which bits to re-index. Which helps to make your content more findable.

Apply layout CSS

Add layout CSS to your HTML pages. Experiment with different layouts for your markup by moving elements around the page. You're wireframing!

Test and iterate

You should be testing with real users at every stage of development but it's particularly important to conduct usability AND accessibility tests now. It's like testing traditional wireframes but you're testing on the real application with real application behaviours and real data (no lorum ipsum nonsense).

Sometimes the results of your testing will require changes to layout CSS, sometimes to markup, sometimes to the data you need to surface and sometimes to the underlying domain / data model. Bear in mind if you're using data from existing business systems there may need to be heavy investment to make changes to that data model and employ the staff to admin those changes. Occasionally it might even mean renegotiating contracts with outside data providers. All design and usability issues are fixable - some just need more lawyers than others : )

Apply decor CSS

Over the top of your wireframe application you can now start to add visual design and branding. This is exactly the same process as taking a paper wireframe and applying design treatments over the top except you're mainly working in CSS.

Experiment with different treatments - see how far you can stretch the design with the markup given. Sometimes you'll need to add additional markup to hook your CSS off.

Now's the time to add background imagery for headers, dividers, buttons, list items etc so best to open Photoshop / Illustrator to make your design assets.

And test and iterate

Never stop testing.

Remember that personas are just abstractions of people - it's always better to use real people.

Ideally you should be able to adjust your code / markup / CSS to respond to user requests. If you can afford the recruitment / developer time there's no better way to test than with a user sitting alongside a developer - the developer can react to user requests, tweak the application and gain instant feedback without the ambiguity that sometimes comes from test reports.

Again you should accessibility test - some of the design / decor changes may affect font sizes etc - make sure your users can still read the page.

Add any JavaScript / AJAX

By designing your browsable site first and adding in Javascript / AJAX over the top you stand a better chance of making an accessible web site - one that degrades gracefully.

As ever Google et al are your least able users - search bots don't like forms or JavaScript - sites that degrade well for accessibility also degrade well for search engines.

Making every subsidiary resource addressable and providing these resources serialised as XML or JSON makes adding AJAX relatively trivial.

You'll probably need to tweak your CSS to adjust to life with JavaScript / AJAX.

And test and iterate

Again test your site for accessibility and usability with JavaScript turned on and off.

Continue

Follow the same steps for each development cycle. Some development cycles will just be about surfacing new views of the existing domain model; some will require expanding your domain model.

Now you know your domain model and have made each domain object addressable layering over new views and more subtle user journeys should be trivial.

And keep testing!