The details of how fragment navigation (also hash navigation or hash fragment navigation) works have been covered extensively by others, and have been abstracted into various frameworks and toolkits. A number are available for jQuery, such as:

• History
• BBQ (tutorial)
• Address

However, fragment navigation comes with a new set of challenges that we found ourselves having to address, so these will be the focus of this post.

Full page alternatives

Rather than changing all links to the form <a href="#!fragment/path/for/ajax">Link</a> and then detecting and acting upon fragment changes in javascript, we wanted to ensure that our links worked when they were copied and pasted into a new browser window, shared by email, or used with javascript disabled. This meant making them into real URL links, and then progressively enhancing using a Javascript onclick handler to cancel the normal navigation action:

<a href="/path/works/as/fragment/or/normal/page" class="fragnav">Link</a>

And some jQuery to hook the onclick handler on these fragment-navigation-compatible links, and convert them so that rather than navigating to the specified href, the browser changes it’s hash instead:

$("a.fragnav").click(function() {
  var href = this.href.replace(/^https?:\/\/[a-z0-9\.]+\/(.*\#\!)?/, '');
  location.hash = this.href;
  return false;
});

Now you can either click the link with javascript enabled and get a dynamic AJAX behaviour, or right click then ‘Open in new window’ (or click with javascript disabled) to get the full URL of the link to load normally.

Caching

When your user presses ‘back’, you can detect the fragment change and reload appropriate content – great. But this is not as good as what you get from browsers’ normal back button behaviour. Normally, the previous page is cached, so it reappears almost instantly. In the AJAX version, unless your fragment is just switching between different versions of content that are all preloaded, you may fire an AJAX request to load the content again.

It shouldn’t be necessary to reimplement caching yourself, since AJAX requests are subject to the same browser caching as normal page loads, but it’s easy to forget that you should include appropriate cache headers with your AJAX responses. There are four HTTP headers that generally modify a browser’s caching behaviour:

• Cache-Control (Docs)
• Expires (Docs)
• ETag (Docs)
• Last-Modified (Docs)

Cache-Control and Expires are cache directive headers, telling the browser whether and for how long it may cache the resource. Typically we set only Cache-Control, as it is more powerful and flexible, and Expires is generally unnecessary.

ETag and Last Modified are validators. These allow the browser to make a conditional request to the server to find out if the resource has changed, and allow the server to respond with a simple 304 Not Modified if it hasn’t. It’s often the case that your web server will add these automatically, and we prefer to avoid them entirely, relying on a single Cache-Control header to determine browser behaviour.

Cache-Control: max-age:60, public

A nice trick for content that is personalised to an authenticated user is to include the private directive in your Cache-Control header, which allows the page to be cached, but only by the browser, not by any proxies along the way.

Scroll position

One of the things we found to be most difficult was managing scroll position. Consider the default behaviour of a browser when navigating normally. If you scroll down a page and click a link somewhere near the bottom, the browser loads the new page and displays it starting at the top. However, if you then press the back button, it redisplays the last page you viewed and restores your scroll position for that page. You may not have even noticed this, but if it stopped behaving this way you’ get pretty annoyed soon enough!

So, if a user clicks one of your fragment links low down on your page, and the change you make to the page as a result would appear to the user to constitute a new page (obviously the point at which perception of ‘a new page’ is triggered is subjective), consider scrolling the browser window back to the top.

But, you should not do this if the back button is used, because the browser will restore the user’s previous scroll position all by itself. However, this becomes more complex if navigating forwards has considerably shortened the page content, and restoring the scroll position on back would require you to restore the content first in order for the necessary scroll offset to actually exist. This is a bit confusing. Here’s an illustration:

So the solution we use is to remember the scroll position on every navigation action, and then restore it after repopulating the content if it looks like a backwards step. You need to ensure you have got your caching rules right to support this otherwise the delay in refetching the content will make the browser reposition the scroll position twice – once by itself immediately, and again triggered by your JavaScript after the content has loaded.

Loading pause

Normally, the process of clicking a link and navigating to a new page involves the current page remaining on screen while the new page is being requested. The browser only blanks it out when content starts to arrive for the new page. The browser does provide some progress feedback immediately though, in the form of a wait mouse cursor or spinner in the browser chrome.

We aimed to replicate this experience for our AJAX-loaded views. This means avoiding what seems like an obvious solution – empty the container when the link is clicked, and populate it when the AJAX response is received – because you’ll get a flash of blankness between the old content disappearing and new content replacing it. Instead, the old content should remain, and the new content should simply replace it when the AJAX completes.

But this doesn’t provide progress feedback. The risk is, the user will think nothing’s happened and will click the link again. This tends to happen after about 2-3 seconds for most users, but AJAX calls normally don’t take anywhere near that long. So we implemented a timeout that, after 250ms, would blank out the container, replacing the old content with a loading spinner. If the AJAX completes within that time, it cancels the timer.

So, in most cases the user clicks a link and sees an almost immediate cut from old content to new with no flash of blankness. In some cases they see the old content vanish and a loading spinner to reassure them that something is happening while we wait for the new content to come back.

If you regularly have edge cases where content takes more than a few seconds to load, you should consider moving on to a different kind of progress feedback, to avoid hitting that “it hasn’t worked” perception boundary. The aim is to keep pushing that boundary back, keeping the user’s expectations in line with the performance that they’re getting from the site. I like to think of this like a timeline:

By providing better progress feedback, you can expand the ‘Expectation’ phase and avoid the ‘Frustration’ phase. However, it’s also equally important that you don’t display a big piece of progress feedback for the entire operation to then complete in under 250ms – the feedback will be on screen for such a short period that the user will potentially be unsure about what happened and get anxious. So be aware of the ‘instant’ phase, when you should present no feedback at all.

Inline Javascript

Sometimes, it’s convenient to get both new HTML code and new Javascript when your user clicks a fragment link. In our case, we wanted to set some Javascript globals in the response to allow javascript code already loaded on the page to better understand the content that had just been loaded. Say the user has requested a page with a fragment that loads some search results. We also want to tell the browser the search query that produced the results so that it can cache them, or populate the query into a search field that’s outside the fragment container, or something.

Normally, if you load content from a server using AJAX and append it to the DOM using innerHTML, any <script> sections in the HTML are not parsed by the browser’s JavaScript engine. However, to make it do this is as simple as searching the returned source for <script> sections, and evaling them. Make sure you properly filter and escape end user input before allowing it to be evaled though, otherwise you’re opening up your site to XSS attacks.

$('#main script').each(function() { eval(this.text); });

Command, Control and Shift modifier keys

Power users like to open links in new tabs or windows. In fact when faced with a list of links I often hold down Control and hit each link to turn to start preloading all the results into tabs which I can then review in turn without having to wait for each one to load. To avoid annoying users, you need to ensure that whereever possible, control-click (or command-click) and shift-click (normally ‘new tab’ and ‘new window’ respectively) still work.

It’s easy to get lulled into a false sense of security by right clicking on your links and using the context menu to select ‘Open in new tab’. This won’t run your onclick handler, so most likely will work if your link has a non-JavaScript href. But using the keyboard CTRL key while clicking the link will fire the onclick event, and if you are cancelling the normal link navigation action by returning false, you’ll also be cancelling the new tab or new window request. Ensuring that you don’t is quite straightforward (remember to include e.metaKey to detect the Command key on Macs):

if (e.shiftKey || e.ctrlKey || e.metaKey) return true;

Focus rectangles

In most browsers, when you click a link, you get a small dotted rectangle around it. This focus rectangle also appears if you tab through the actionable items on a page, to indicate which one would be followed if you pressed enter. The problem is that if your link fires an AJAX action, and the link itself remains on screen after the action has completed, you’re left with a focus rectangle that you probably don’t want.

A bad way of dealing with this is to simply use CSS to set outline:none on all A tags. This will certainly solve your problem, but kill all hope of keyboard navigation of your site. Better is to study the way that this interaction happens normally, and then mimic it for your Ajax actions. This is actually fairly easy to do generically:

(function() {

  // Keep track of which link element last had the focus (if browser does not support document.activeElement)
  if (!document.activeElement) {
    $("a").live('focus', function() {
      document.activeElement = this;
    });
  }

  // When any AJAX operation completes, blur any focused link
  $.ajaxSetup({complete:function(xhr, textStatus) {
    if (document.activeElement && document.activeElement.is('a')) document.activeElement.blur();
  }});
}());

Drop the above snippet into your javascript and you should find that focus rectangles still appear and stay visible while the ajax is running, but then vanish once it completes – perfect replica of the effect the user has learned to expect. If your links don’t perform any AJAX, you’ll have to deal with those separately.

Conclusion

With all this to think about, you’d be forgiven for concluding that AJAX navigation simply isn’t worth the bother. But bear with it – the benefits of being able to trivilally maintain state and load small fragments of content really makes a difference, both to the quality of the user experience and the load on your servers.

The site Assanka recently launched for FT Tilt uses a variety of faceting strategies to give the most appropriate suggestions in each category

Most implementations of faceting always show the same suggestions in any given category, but may change the filters available based on the search the user has done so far. For example, if you’re searching a computer supplies retailer for a new hard disk, most of the products matching your search will have a ‘capacity’ property, so the retailer will probably offer you a capacity filter with all available capacities listed. It probably won’t offer you a ‘pages per minute’ filter, because resolution is not relevant to hard disks, and your existing search keywords have probably eliminated any product (like a monitor) for which a resolution choice would make sense.

So it’s pretty easy to choose which filters to display. The more difficult problem is choosing which facets to display within each filter.

Sometimes it’s impractical to show all possible facets in a filter. eBay, for example, allows you to restrict your search by seller, and of course there are millions of those. So there are a number of possible strategies:

1. Display every option available
2. Display just a few hard coded options, such as aggregate options designed to always match something (eg eBay’s ‘Top rated sellers’ option), or editorially chosen ‘top picks’. Either way, the point is that the suggestions are a subset of the full range available, and are not sensitive to the context created by the user’s search query
3. Use the list of options as per (1) or (2), but hide any that, if selected, would produce no results. This refinement makes the suggestions context-sensitive in a very rudimentary way, in that they are at least reacting to the current state of the user’s search, but still do not surface any long-tail options when they become more relevant.
4. Generate options based on running the query the user has put together so far, including any facets selected, and analysing the resultset for the top facet refinements in each filter category.
5. As per (4), but for each filter category, run the search excluding any options already selected in that category.
6. As per (4) or (5), but where the values are all numeric, determine the facets not from the frequency of occurrence of specific values, but by analysing the distribution of values within the resultset and constructing boundaries that provide a sensible number of divisions with approximately the same number of results in each division.

There are arguments for and against all of these, depending on the distribution of the metadata within your document index.

An e-commerce site will typically have a set of properties on a product, where each property has one value, and won’t use all the available property names. The values available for each property will also typically be a range set, and will efficiently cover the whole available range. For example a hard disk will have a ‘capacity’ property, where the options might include 200GB, 300GB, 500GB and 1TB. Only one of these can apply to any given hard disk. A hard disk, as previously discussed, will also not make use of other available properties, such as resolution, that might apply to other types of product, such as monitors. A property like ‘pages per minute’ is really only going to be used on a very small subset of your product catalogue (only printers). ‘Capacity’ might get a bit more limelight as it applies to USB sticks, RAM and storage appliances as well as hard disks (though consider that the option values required in these types of products might be in a different range), and some properties like ‘manufacturer’ would apply to virtually the entire product catalogue.

Sometimes, a product might have more than one applicable value in the same category. Take an example category “Special offers available”. A single product might qualify for “Buy one get one free” as well as “Free delivery”. This kind of thing actually happens more in the non-retail world, and a better example would be a film library, where a film may have more than one actor, more than one screenwriter, more than one content advisory. Where this is the case, filtering on one actor does not necessarily mean you’ve excluded all the others from the resultset.

The distribution is also relevant. In the ‘capacity’ example, we could expand the number of values to include more granularity below 50GB to allow for solid state devices, and above 1TB to allow for storage appliances, but in any given search, results will tend to form an unequal distribution with a peak, or multiple peaks, in particular capacities. Across a range of hard disks, at time of writing 100-500GB would likely be the most popular value. On the other hand take a category like ‘Actor’ in the case of a film library, and the distribution looks a lot flatter. An actor can only do so many films, and there are a lot of actors, so there isn’t a strong head to this distribution – it’s all tail.

Looking at each filter category in turn, the logic for deciding how to choose suggested values therefore comes down to a number of questions about how the category is used to classify your content:

1. Are there few enough values that you could display them all together?
2. Do the values form a continuous range (like capacity) or are they discrete options (like actor)?
3. Is it possible for any single item of content to have more than one value from the same category?
4. Is there a ‘head’ of a few values (few enough to display all of them together) which, combined, apply to a majority of your content?

It’s important to stress that these are not decisions to take for your site as a whole – they need to be applied to each filter category individually.

There is one final consideration. Is your faceting feature intended to help users narrow their search, change it, or broaden it?

Going back to the possible strategies for determining facets, displaying every option available works for small categories, and using hard coded options groups like ‘top 100 sellers’ is basically a solution for displaying every option available by consolidating many options into one. Doing this where necessary (and then also hiding any options that would result in no matches) gives you about the best solution you are likely to get without going context sensitive.

The DVD search on Amazon.co.uk displays all possible options in the BBFC rating category, and hides any that don’t apply to the results. In this case, the results found by the search only contain films rated PG, 15 and 12.

It starts to get interesting when you have a lot of possible values, in a flattish distribution, and want to present specific, context sensitive options. The sense of ‘context’ depends on whether you want to help broaden or narrow the search.

Take a search for the location “UK” and animal “Dog”, which will give you results referring to dogs in the UK. One of the facet categories might be location, in which there is one option that is already a term in the search – UK. Determining the facets to suggest in the location category by only looking at the results returned in the existing search would only yield locations that co-exist on items tagged with UK and dogs, so your filter refinements would be places like “Manchester”, “Birmingham”, “Liverpool”, “London”, “Cardiff”, “Edinburgh”. This helps the searcher to refine their search.

The film search on Lovefilm presents facets that narrow your search. It also arranges the facets in a hierarchy, though this is an illusion

However, using strategy 5, you run the search once on “UK Dogs” to produce the results to show the user, but you run it again on “Dogs”, excluding the term from the locations category, to generate location facets. This time you get locations globally that co-exist with Dogs, and the suggestions for facets in any given category do not change if you choose to add a facet from that category to your search. In this case the suggestions would be more likely to be “Paris”, “New York”, “France”, “Boston”, “London”, “United States”. This helps to broaden the search by showing the user good suggestions for other locations that also have lots of dogs that they may not have considered.

Finally, where the values are numeric, it may be appropriate to produce dynamic facets as range boundaries, based on an analysis of the values that exist within the resultset. You still follow one or other of the above strategies to get a resultset that either narrows or broadens the search, but then analyse the list of values and construct divisions that evenly partition the data into a small number of ranges. Doing this with a search for say the product type “Hard disks” and the capacity “100-150GB”, the suggested capacity facets would subdivide the selected range, with narrow boundaries covering the most popular capacities, and wider ranges where there are fewer results.

Ebuyer’s product search contains lots of faceting of numeric categories, some of which are presented in dynamic ranges, and some are treated as standard terms

For numeric categories like this, it’s also worth considering the sort order of the facet list. For categories of non-numeric values and sometimes even for discrete numeric values (say ‘film speed’ or ‘quantity per box’), it’s generally best to present them in decreasing order of popularity within the search context you’ve chosen. For numeric categories where you’re constructing range facets (eg. ‘capacity’, ‘pixel density’, ‘brightness’), they should instead be presented in value order.

Conclusion

There are many ways of faceting search results. Take some time to choose the one that suits your application best, and provides the best search experience for your users. We use the open source search engine Xapian, which is excellent at doing all the faceting described in this post, and I’d like to publicly acknowledge Richard Boulton for his excellent advice when we were designing faceting strategy for FT Tilt.

Try scrolling either of the panels in this example:

You should find it scrolls madly and unpredictably until you press stop.

The first problem is that onscroll is not simply fired when you finish scrolling. It fires like a machine gun continuously while the mouse cursor is moving on the scroll handle. The solution to this is a watchdog. A watchdog is a timer that will execute action A if action B does not happen within X seconds. So every time onscroll fires, we reset the timer, and if it gets to zero we know that the user has finished scrolling (or at least has paused for long enough for us to do something about it).

You’ll find that although it’s not quite as crazed, the panels do keep scrolling, one after the other.

The problem now is that when you scroll DIV 1, and this causes an automatic scroll of DIV 2, that automatic scroll ALSO triggers the onscroll event and so we then scroll DIV 1 again. The solution is to have a variable that flags whether we are currently paying attention to scroll events, and turn it off when we don’t want to detect scrolling (ie just before the auto-scroll) and then on again after the scroll has completed (I’m using an animation library so the scroll takes around half a second to complete). Try this (don’t click the button yet, just scroll the panels):

Success. However, this approach is a bit short-sighted.

You also need to turn off scroll detection when adding or removing content from either DIV, because changing the height of the content within the element can also fire the onscroll event. So if this is a chat window, and we’re adding and removing content in the DIVs, we have to disable scroll detection while we’re doing that and then turn it on again afterwards.

Sadly, our scroll detection flag is crude – it’s a sheer yes/no, and if it’s already set to no (say in order to execute a lengthy scrolling animation), and in the meantime we need to do something quick (say adding a line to one of the DIVs) then that quick operation is going to disable scroll detection (unnecessarily, as it’s already disabled) and then crucially enable it again before the scroll animation has finished. Click the button in the example above to demonstrate this (and then scroll). You should, intermittently, get the unwanted cascade effect you saw in example 2.

What we need is a list, not a flag. Enter the blocker list – an object that collates tokens from each procedure in the script that is currently blocking scroll detection. So if a procedure wants to disable scrolling for a period of time, it adds its token to the blocker list, and then removes its token when it’s done. When a scroll event fires, we now only need to know whether there are any items in the blocker list, and then we can work out if it’s ok to process the event.

It’s also worth noting that there’s no need to actually count the number of items in the blocker list – it’s enough simply to know that there is at least one item in there. And as a further optimisation, we don’t actually need to compute this when scroll events fire (frequently), because the value will only change when some function wants to add or remove its token from the list (less frequent). So we can have enableScrollDetection and disableScrollDetection functions that deal with the blocker list, and which ultimately simply change the old scrolldetection flag to true or false.

There might be a neater way of achieving this, but this certainly works for me. I’d welcome any comments or suggestions. The sources for all these demos are available in the iframes above.

But this was clearly not my problem, and various sources suggested that this had in any case been fixed for Firefox 3. I discovered that my problem was rather simpler. If you disable an element that has focus, and then re-enable it, you don’t get the cursor back.

Solution: blur (remove focus from) the element before you disable it. Those sources that do refer to this specific problem tend to suggest that you focus a different element before you disable the text box. But this is not necessary – you can just blur the element that has focus and leave the page with nothing focused.

Problem test case

Try clicking and typing in the field below – if your browser exhibits this problem, the text cursor will not appear. If it does, then your browser does not have this problem.

This demo has a single text field which, when you click into it, is briefly disabled, and then enabled again. The disabling of the field while it has focus triggers this problem, so the text cursor does not appear (but you can still type into the field!).

Solution test case

Now try typing in this field – the cursor should appear consistently.

This demo still briefly disables and re-enables the text field when it gets focus, but this time it blurs it right before it’s disabled, then focuses it again after re-enabling it. The result is that when you place your mouse cursor in the field to manually give it focus, the text cursor appears as normal.

First, in case anyone has just arrived from Mars, or even more unlikely, isn’t familiar with Facebook, the auto-growing textarea is a text box that gets bigger as you type into it, so that you never see a scroll bar (unless you’re typing war and peace).

There are lots of ways of doing each part of this. The parts I’ll look at are (a) what triggers a review of the text box’s size, (b) how to determine whether a resize is required, and (c) how to perform a resize.

Triggering a review

There are four ways this is typically done:

1. On an interval, say every 50ms
2. On keypress
3. On change
4. On keyup

I’m staggered by how many scripts insist on doing this kind of thing on an interval. Setting up interval timers for no good reason is a shortcut to terrible performance and memory leaks galore. So we won’t be doing that. onkeypress and onchange events are triggered before the box is updated with the latest keypress, so we want to avoid those, as the latest keypress might have been the one to bring us onto a new line. That leaves onkeyup, which is fired after the box is updated with the new character, and allows us to inspect it and decide whether to increase its size.

To resize or not to resize

How to determine whether to resize the box comes in three flavours:

1. Count the number of newline characters in the textarea’s value, and see if that matches the number of rows the textarea has.
2. Create a ’shadow’ DIV which is off screen (eg. margin-left: -10000px) and has no height declared, but otherwise has the same style properties as the textarea, fill it with the textarea’s content, measure the subsequent height of the DIV, then see if that height matches the current height of the textarea.
3. Check whether scrollHeight (the height of the scollable content of the box) > clientHeight (the height of the box itself).

In this case I was suprised at the number of implementations that favoured counting newlines. This only works if combined with counting the number of characters in the textarea’s value, AND a monospaced font, AND knowing the number of columns in the textarea. In short, er, it’s mad.

Second option, and the one favoured by a lot of the framework plugins due to the ease with which you can create shadow elements in the likes of jQuery, is to create a shadow DIV. This has the advantage of telling you the actual pixel height of the text, even where it is LESS than the height of the textarea box. Otherwise, you’re limited to measuring clientHeight and scrollHeight, which are exactly the same if the textarea isn’t scrolling, regardless of how much space you’ve got to spare at the bottom. My issue with this method is that it basically requires use of a framework to not be painful, and even then it’s non-trivial amounts of code, and adds needless pollution to the DOM.

So that leaves relying on scrollHeight and clientHeight. These are well supported and efficient to read, so provided that you work around the issue of scrollHeight always being at least equal to clientHeight, this offers a very lightweight solution. The features that you can achieve with a shadow DIV that you can’t do by simply reading scrollHeight and clientHeight are (a) you can ensure there is always a blank line at the bottom of your textarea, and (b) you can shrink the textarea if the user deletes text from it. I’m personally of the view that neither of these is actually particularly desirable. There’s potentially an argument for leaving a blank line at the bottom, but equally the uer might feel like they’re just being pressured to write more.

How to resize the box

OK, so if you’ve concluded that the user is writing chapter and verse and the textarea is in need of a bit more space, how do we go about doing it?

1. Animate the CSS height property
2. Add some height via the CSS height property
3. Add 1 to the rows attribute

A fair few of the framework plugins use their framework’s animation capabilities to animate the grow effect. I don’t like this. Just because you have an animation effect available doesn’t mean it’s appropriate to use it, and there are many situations where the end user just doesn’t want to wait 300ms for the privilege of using a fresh line.

Simply adding height to the CSS property is a fair way of doing it, but unless you do some maths on the user’s line height (or hard code some magic numbers), you can’t necessarily guarantee that the resulting height will be a multiple of the textarea’s line height.

Easiest, most efficient solution: add one to the rows attribute of the textarea. rows is part of XHTML as well as HTML 4.01, and has universal support going back yonks.

Pasting

Watch out for pastes. If the user pastes in a large quantity of text, they will trigger only one keyup event, but will have added many lines to the textarea. Make sure that if a resize is required, you trigger another review after the resize is complete.

Max height: the war and peace scenario

If the user really does seem to be writing a novel, we probably should call it a day on growing the textarea at some stage, and certainly before it gets to be taller than the viewport. You can check the height against the viewport height, though I typically just restrict it to an arbitrary height, say 30 rows.

The code

You’ll need a textarea element with an ID of mytextarea to make this code sample work, and obviously you can easily modify it to use selectors from your favourite framework rather than the native document.getElementById.

document.getElementById('mytextarea').onkeyup = function() {
	var ta = document.getElementById('mytextarea');
	var maxrows = 30;
	var lh = ta.clientHeight / ta.rows;
	while (ta.scrollHeight > ta.clientHeight && !window.opera && ta.rows < maxrows) {
		ta.style.overflow = 'hidden';
		ta.rows += 1;
	}
	if (ta.scrollHeight > ta.clientHeight) ta.style.overflow = 'auto';
}