Probably because touchscreen devices weren’t nearly as popular in 1995 as they are today, JavaScript DOM events tend to reflect the mouse ‘click’ paradigm, where each event neatly corresponds to a single, deliberate click of the button. Take the onClick event, for example. How does that translate to a touchscreen device? Not easily, as it turns out.

Treating a ‘tap’ as a ‘click’ is the practical approach. On iOS at least, there’s no onTap event – onClick has been repurposed for that role. But in order to properly handle multiple-touch gestures like pinching or even double-taps, some compromises have to be made. One of these is the roughly 300ms delay between a tap and the firing of a click event, which can make your apps feel laggy and unresponsive even when it’s not technically so.

The technique we use to get around this problem is to track all TouchStart events in our app and fire a click event as soon as we receive a TouchEnd event (unless some application-specific exception is satisfied).

As we refined our fast-clicking code, we turned it into a small and efficient library, which we call FastClick. This code has been tried and tested by hundreds of thousands of users, and so far has proved to be very robust. We’d love to know what others are doing to address this challenge and find ways of improving our own approach, and to help kick off that process, we’re open-sourcing FastClick today. We’d encourage you to try it out, and let us know what you think.

To use FastClick, instantiate it on the layer you’d like to be fast-clickable – we use document.body because we want all our buttons and links to receive fast clicks. In your event listeners, ‘click’ events synthesised by FastClick will have the forwardedTouchEvent property set to true.

If you use buttons and iOS-style menus in your app then there’s a good chance your interface feels unresponsive on touchscreen devices. Here’s a simple example of how we solve that problem for a single button by instantiating FastClick on it:

<button class=”fastclick” onclick=”someHandler()”>Fast click</button>
<button onclick=”someHandler()”>Slow click</button>
<script type='text/javascript'>
var button = document.querySelector(".fastclick");
new FastClick(button);
</script>

In the example, the button with a fastclick class will have its click handler called as soon as your finger is lifted off the screen, while tapping on any other buttons on the page will trigger the same handler after a noticeable delay. Try the live demo.

Unfortunately, the select element doesn’t behave normally when receiving (synthesised) programmatic clicks, so if you apply FastClick to an element that contains selects, FastClick will ignore clicks on the select and allow the normal click event to fire.

If you want any other elements (besides selects) in your FastClick layer to receive non-programmatic clicks, you’ll need to use one of two classes: clickevent or touchandclickevent. For any clickable element in a FastClick layer, tapping the element will cause different effects depending on how you use these classes:

• No class: The element will receive only a programmatic click from FastClick. The default click event triggered by the user will be suppressed.
• clickevent: The element will receive only the default click event, and will be ignored by FastClick
• touchandclickevent: The element will receive both the default click event AND a programmatic click from FastClick (the FastClick one will be triggered first). This is only safe if your handler’s action is idempotent.

Here’s an example of all three:

<div class="fastclick">
	<button onclick="someHandler()">
		Will receive programmatic click event
	</button>
	<button class="clickevent" onclick="someHandler()">
		Will receive non-programmatic click event
	</button>
	<button class="touchandclickevent" onclick="someHandler()">
		Will receive both click events
	</button>
</div>
<script type='text/javascript'>
var button = document.querySelector(".fastclick");
new FastClick(button);
</script>

When is this useful? Try the other live demo we’ve built using a click event handler that attempts to trigger focus on an input element. iOS will only allow focus to be triggered on other elements within a handler function if the event that triggered it was non-programmatic.

We’ll be posting updates and answering questions on this blog. If the interest reaches a stage where FastClick could benefit from the developer community, we’ll move to a public repository host. But for now, concentrate on giving your users the best response time they can get – download FastClick (or minified) and give it a go. Its free to use in all your apps, and licensed under the MIT license.

The details of how fragment navigation (also hash navigation or hash fragment navigation) works have been covered extensively by others, and have been abstracted into various frameworks and toolkits. A number are available for jQuery, such as:

• History
• BBQ (tutorial)
• Address

However, fragment navigation comes with a new set of challenges that we found ourselves having to address, so these will be the focus of this post.

Full page alternatives

Rather than changing all links to the form <a href="#!fragment/path/for/ajax">Link</a> and then detecting and acting upon fragment changes in javascript, we wanted to ensure that our links worked when they were copied and pasted into a new browser window, shared by email, or used with javascript disabled. This meant making them into real URL links, and then progressively enhancing using a Javascript onclick handler to cancel the normal navigation action:

<a href="/path/works/as/fragment/or/normal/page" class="fragnav">Link</a>

And some jQuery to hook the onclick handler on these fragment-navigation-compatible links, and convert them so that rather than navigating to the specified href, the browser changes it’s hash instead:

$("a.fragnav").click(function() {
  var href = this.href.replace(/^https?:\/\/[a-z0-9\.]+\/(.*\#\!)?/, '');
  location.hash = this.href;
  return false;
});

Now you can either click the link with javascript enabled and get a dynamic AJAX behaviour, or right click then ‘Open in new window’ (or click with javascript disabled) to get the full URL of the link to load normally.

Caching

When your user presses ‘back’, you can detect the fragment change and reload appropriate content – great. But this is not as good as what you get from browsers’ normal back button behaviour. Normally, the previous page is cached, so it reappears almost instantly. In the AJAX version, unless your fragment is just switching between different versions of content that are all preloaded, you may fire an AJAX request to load the content again.

It shouldn’t be necessary to reimplement caching yourself, since AJAX requests are subject to the same browser caching as normal page loads, but it’s easy to forget that you should include appropriate cache headers with your AJAX responses. There are four HTTP headers that generally modify a browser’s caching behaviour:

• Cache-Control (Docs)
• Expires (Docs)
• ETag (Docs)
• Last-Modified (Docs)

Cache-Control and Expires are cache directive headers, telling the browser whether and for how long it may cache the resource. Typically we set only Cache-Control, as it is more powerful and flexible, and Expires is generally unnecessary.

ETag and Last Modified are validators. These allow the browser to make a conditional request to the server to find out if the resource has changed, and allow the server to respond with a simple 304 Not Modified if it hasn’t. It’s often the case that your web server will add these automatically, and we prefer to avoid them entirely, relying on a single Cache-Control header to determine browser behaviour.

Cache-Control: max-age:60, public

A nice trick for content that is personalised to an authenticated user is to include the private directive in your Cache-Control header, which allows the page to be cached, but only by the browser, not by any proxies along the way.

Scroll position

One of the things we found to be most difficult was managing scroll position. Consider the default behaviour of a browser when navigating normally. If you scroll down a page and click a link somewhere near the bottom, the browser loads the new page and displays it starting at the top. However, if you then press the back button, it redisplays the last page you viewed and restores your scroll position for that page. You may not have even noticed this, but if it stopped behaving this way you’ get pretty annoyed soon enough!

So, if a user clicks one of your fragment links low down on your page, and the change you make to the page as a result would appear to the user to constitute a new page (obviously the point at which perception of ‘a new page’ is triggered is subjective), consider scrolling the browser window back to the top.

But, you should not do this if the back button is used, because the browser will restore the user’s previous scroll position all by itself. However, this becomes more complex if navigating forwards has considerably shortened the page content, and restoring the scroll position on back would require you to restore the content first in order for the necessary scroll offset to actually exist. This is a bit confusing. Here’s an illustration:

So the solution we use is to remember the scroll position on every navigation action, and then restore it after repopulating the content if it looks like a backwards step. You need to ensure you have got your caching rules right to support this otherwise the delay in refetching the content will make the browser reposition the scroll position twice – once by itself immediately, and again triggered by your JavaScript after the content has loaded.

Loading pause

Normally, the process of clicking a link and navigating to a new page involves the current page remaining on screen while the new page is being requested. The browser only blanks it out when content starts to arrive for the new page. The browser does provide some progress feedback immediately though, in the form of a wait mouse cursor or spinner in the browser chrome.

We aimed to replicate this experience for our AJAX-loaded views. This means avoiding what seems like an obvious solution – empty the container when the link is clicked, and populate it when the AJAX response is received – because you’ll get a flash of blankness between the old content disappearing and new content replacing it. Instead, the old content should remain, and the new content should simply replace it when the AJAX completes.

But this doesn’t provide progress feedback. The risk is, the user will think nothing’s happened and will click the link again. This tends to happen after about 2-3 seconds for most users, but AJAX calls normally don’t take anywhere near that long. So we implemented a timeout that, after 250ms, would blank out the container, replacing the old content with a loading spinner. If the AJAX completes within that time, it cancels the timer.

So, in most cases the user clicks a link and sees an almost immediate cut from old content to new with no flash of blankness. In some cases they see the old content vanish and a loading spinner to reassure them that something is happening while we wait for the new content to come back.

If you regularly have edge cases where content takes more than a few seconds to load, you should consider moving on to a different kind of progress feedback, to avoid hitting that “it hasn’t worked” perception boundary. The aim is to keep pushing that boundary back, keeping the user’s expectations in line with the performance that they’re getting from the site. I like to think of this like a timeline:

By providing better progress feedback, you can expand the ‘Expectation’ phase and avoid the ‘Frustration’ phase. However, it’s also equally important that you don’t display a big piece of progress feedback for the entire operation to then complete in under 250ms – the feedback will be on screen for such a short period that the user will potentially be unsure about what happened and get anxious. So be aware of the ‘instant’ phase, when you should present no feedback at all.

Inline Javascript

Sometimes, it’s convenient to get both new HTML code and new Javascript when your user clicks a fragment link. In our case, we wanted to set some Javascript globals in the response to allow javascript code already loaded on the page to better understand the content that had just been loaded. Say the user has requested a page with a fragment that loads some search results. We also want to tell the browser the search query that produced the results so that it can cache them, or populate the query into a search field that’s outside the fragment container, or something.

Normally, if you load content from a server using AJAX and append it to the DOM using innerHTML, any <script> sections in the HTML are not parsed by the browser’s JavaScript engine. However, to make it do this is as simple as searching the returned source for <script> sections, and evaling them. Make sure you properly filter and escape end user input before allowing it to be evaled though, otherwise you’re opening up your site to XSS attacks.

$('#main script').each(function() { eval(this.text); });

Command, Control and Shift modifier keys

Power users like to open links in new tabs or windows. In fact when faced with a list of links I often hold down Control and hit each link to turn to start preloading all the results into tabs which I can then review in turn without having to wait for each one to load. To avoid annoying users, you need to ensure that whereever possible, control-click (or command-click) and shift-click (normally ‘new tab’ and ‘new window’ respectively) still work.

It’s easy to get lulled into a false sense of security by right clicking on your links and using the context menu to select ‘Open in new tab’. This won’t run your onclick handler, so most likely will work if your link has a non-JavaScript href. But using the keyboard CTRL key while clicking the link will fire the onclick event, and if you are cancelling the normal link navigation action by returning false, you’ll also be cancelling the new tab or new window request. Ensuring that you don’t is quite straightforward (remember to include e.metaKey to detect the Command key on Macs):

if (e.shiftKey || e.ctrlKey || e.metaKey) return true;

Focus rectangles

In most browsers, when you click a link, you get a small dotted rectangle around it. This focus rectangle also appears if you tab through the actionable items on a page, to indicate which one would be followed if you pressed enter. The problem is that if your link fires an AJAX action, and the link itself remains on screen after the action has completed, you’re left with a focus rectangle that you probably don’t want.

A bad way of dealing with this is to simply use CSS to set outline:none on all A tags. This will certainly solve your problem, but kill all hope of keyboard navigation of your site. Better is to study the way that this interaction happens normally, and then mimic it for your Ajax actions. This is actually fairly easy to do generically:

(function() {

  // Keep track of which link element last had the focus (if browser does not support document.activeElement)
  if (!document.activeElement) {
    $("a").live('focus', function() {
      document.activeElement = this;
    });
  }

  // When any AJAX operation completes, blur any focused link
  $.ajaxSetup({complete:function(xhr, textStatus) {
    if (document.activeElement && document.activeElement.is('a')) document.activeElement.blur();
  }});
}());

Drop the above snippet into your javascript and you should find that focus rectangles still appear and stay visible while the ajax is running, but then vanish once it completes – perfect replica of the effect the user has learned to expect. If your links don’t perform any AJAX, you’ll have to deal with those separately.

Conclusion

With all this to think about, you’d be forgiven for concluding that AJAX navigation simply isn’t worth the bother. But bear with it – the benefits of being able to trivilally maintain state and load small fragments of content really makes a difference, both to the quality of the user experience and the load on your servers.

The site Assanka recently launched for FT Tilt uses a variety of faceting strategies to give the most appropriate suggestions in each category

Most implementations of faceting always show the same suggestions in any given category, but may change the filters available based on the search the user has done so far. For example, if you’re searching a computer supplies retailer for a new hard disk, most of the products matching your search will have a ‘capacity’ property, so the retailer will probably offer you a capacity filter with all available capacities listed. It probably won’t offer you a ‘pages per minute’ filter, because resolution is not relevant to hard disks, and your existing search keywords have probably eliminated any product (like a monitor) for which a resolution choice would make sense.

So it’s pretty easy to choose which filters to display. The more difficult problem is choosing which facets to display within each filter.

Sometimes it’s impractical to show all possible facets in a filter. eBay, for example, allows you to restrict your search by seller, and of course there are millions of those. So there are a number of possible strategies:

1. Display every option available
2. Display just a few hard coded options, such as aggregate options designed to always match something (eg eBay’s ‘Top rated sellers’ option), or editorially chosen ‘top picks’. Either way, the point is that the suggestions are a subset of the full range available, and are not sensitive to the context created by the user’s search query
3. Use the list of options as per (1) or (2), but hide any that, if selected, would produce no results. This refinement makes the suggestions context-sensitive in a very rudimentary way, in that they are at least reacting to the current state of the user’s search, but still do not surface any long-tail options when they become more relevant.
4. Generate options based on running the query the user has put together so far, including any facets selected, and analysing the resultset for the top facet refinements in each filter category.
5. As per (4), but for each filter category, run the search excluding any options already selected in that category.
6. As per (4) or (5), but where the values are all numeric, determine the facets not from the frequency of occurrence of specific values, but by analysing the distribution of values within the resultset and constructing boundaries that provide a sensible number of divisions with approximately the same number of results in each division.

There are arguments for and against all of these, depending on the distribution of the metadata within your document index.

An e-commerce site will typically have a set of properties on a product, where each property has one value, and won’t use all the available property names. The values available for each property will also typically be a range set, and will efficiently cover the whole available range. For example a hard disk will have a ‘capacity’ property, where the options might include 200GB, 300GB, 500GB and 1TB. Only one of these can apply to any given hard disk. A hard disk, as previously discussed, will also not make use of other available properties, such as resolution, that might apply to other types of product, such as monitors. A property like ‘pages per minute’ is really only going to be used on a very small subset of your product catalogue (only printers). ‘Capacity’ might get a bit more limelight as it applies to USB sticks, RAM and storage appliances as well as hard disks (though consider that the option values required in these types of products might be in a different range), and some properties like ‘manufacturer’ would apply to virtually the entire product catalogue.

Sometimes, a product might have more than one applicable value in the same category. Take an example category “Special offers available”. A single product might qualify for “Buy one get one free” as well as “Free delivery”. This kind of thing actually happens more in the non-retail world, and a better example would be a film library, where a film may have more than one actor, more than one screenwriter, more than one content advisory. Where this is the case, filtering on one actor does not necessarily mean you’ve excluded all the others from the resultset.

The distribution is also relevant. In the ‘capacity’ example, we could expand the number of values to include more granularity below 50GB to allow for solid state devices, and above 1TB to allow for storage appliances, but in any given search, results will tend to form an unequal distribution with a peak, or multiple peaks, in particular capacities. Across a range of hard disks, at time of writing 100-500GB would likely be the most popular value. On the other hand take a category like ‘Actor’ in the case of a film library, and the distribution looks a lot flatter. An actor can only do so many films, and there are a lot of actors, so there isn’t a strong head to this distribution – it’s all tail.

Looking at each filter category in turn, the logic for deciding how to choose suggested values therefore comes down to a number of questions about how the category is used to classify your content:

1. Are there few enough values that you could display them all together?
2. Do the values form a continuous range (like capacity) or are they discrete options (like actor)?
3. Is it possible for any single item of content to have more than one value from the same category?
4. Is there a ‘head’ of a few values (few enough to display all of them together) which, combined, apply to a majority of your content?

It’s important to stress that these are not decisions to take for your site as a whole – they need to be applied to each filter category individually.

There is one final consideration. Is your faceting feature intended to help users narrow their search, change it, or broaden it?

Going back to the possible strategies for determining facets, displaying every option available works for small categories, and using hard coded options groups like ‘top 100 sellers’ is basically a solution for displaying every option available by consolidating many options into one. Doing this where necessary (and then also hiding any options that would result in no matches) gives you about the best solution you are likely to get without going context sensitive.

The DVD search on Amazon.co.uk displays all possible options in the BBFC rating category, and hides any that don’t apply to the results. In this case, the results found by the search only contain films rated PG, 15 and 12.

It starts to get interesting when you have a lot of possible values, in a flattish distribution, and want to present specific, context sensitive options. The sense of ‘context’ depends on whether you want to help broaden or narrow the search.

Take a search for the location “UK” and animal “Dog”, which will give you results referring to dogs in the UK. One of the facet categories might be location, in which there is one option that is already a term in the search – UK. Determining the facets to suggest in the location category by only looking at the results returned in the existing search would only yield locations that co-exist on items tagged with UK and dogs, so your filter refinements would be places like “Manchester”, “Birmingham”, “Liverpool”, “London”, “Cardiff”, “Edinburgh”. This helps the searcher to refine their search.

The film search on Lovefilm presents facets that narrow your search. It also arranges the facets in a hierarchy, though this is an illusion

However, using strategy 5, you run the search once on “UK Dogs” to produce the results to show the user, but you run it again on “Dogs”, excluding the term from the locations category, to generate location facets. This time you get locations globally that co-exist with Dogs, and the suggestions for facets in any given category do not change if you choose to add a facet from that category to your search. In this case the suggestions would be more likely to be “Paris”, “New York”, “France”, “Boston”, “London”, “United States”. This helps to broaden the search by showing the user good suggestions for other locations that also have lots of dogs that they may not have considered.

Finally, where the values are numeric, it may be appropriate to produce dynamic facets as range boundaries, based on an analysis of the values that exist within the resultset. You still follow one or other of the above strategies to get a resultset that either narrows or broadens the search, but then analyse the list of values and construct divisions that evenly partition the data into a small number of ranges. Doing this with a search for say the product type “Hard disks” and the capacity “100-150GB”, the suggested capacity facets would subdivide the selected range, with narrow boundaries covering the most popular capacities, and wider ranges where there are fewer results.

Ebuyer’s product search contains lots of faceting of numeric categories, some of which are presented in dynamic ranges, and some are treated as standard terms

For numeric categories like this, it’s also worth considering the sort order of the facet list. For categories of non-numeric values and sometimes even for discrete numeric values (say ‘film speed’ or ‘quantity per box’), it’s generally best to present them in decreasing order of popularity within the search context you’ve chosen. For numeric categories where you’re constructing range facets (eg. ‘capacity’, ‘pixel density’, ‘brightness’), they should instead be presented in value order.

Conclusion

There are many ways of faceting search results. Take some time to choose the one that suits your application best, and provides the best search experience for your users. We use the open source search engine Xapian, which is excellent at doing all the faceting described in this post, and I’d like to publicly acknowledge Richard Boulton for his excellent advice when we were designing faceting strategy for FT Tilt.

We chose to use Drupal – having successfully built Drupal sites for the likes of News International, we knew we could hit the ground running and end up with something that would be able to serve the CIPR for several years.

The challenges

The existing site had 30,000 static HTML files, with widely varying markup and no style consistency.  Because the site was really difficult to edit, many of the most active areas had been spun out into microsites with an asp.net based proprietary content management system.  These microsites were editable by their respective content owners, but with no oversight, and with complete content control available to the editors, so styles began to diverge even more.

A number of third party services and applications needed to be integrated into the new site. Eventbrite for event ticketing, ezProxy for access to academic journal services, and the CIPR’s own Ladder CPD system (another Assanka project).  For Ladder and ezProxy, we needed to be able to authenticate CIPR members outside of Drupal, so we would need to have some kind of single sign on system centered on the Drupal site.

Details about the members themselves are held in the CIPR’s membership system, a client-server application with a Windows front end and a Microsoft SQL Server database.  On the old site, synchronising this with the database the website was using for authentication required a long manual process and, for some reason, a number of Microsoft Word mail merges. The membership system is here to stay, so our solution would need to vastly improve on this process to get member profile updates into the website far faster than was possible before.

Finally, it’s worth mentioning the workflow challenges. Under the old site, the institute’s staff had to email their requested changes to the web team using a Microsoft Word based form. The web team would implement them “within 5 working days”, by editing static HTML files manually, and could not provide any content oversight, only the technical support to put the content online. For a new site to be successful, the management and oversight of the institute’s web output would need to change completely.

Build process

Our build process followed a fairly typical sequence:

1. Site map workshops: We gathered together and worked with the main content creators and heads of department. Over a number of sessions, and using a lot of Post-it notes and a very large wall, we produced a site map.  The site map was transferred to Mindmeister, where it continued to be collaboratively edited and refined.
2. Wireframing: We produced wireframes of a new home page and typical article pages, to explore the organisation and prioritisation of content.  We needed a number of different, flexible content positions to accommodate a variety of content and media.
3. Creative concepts: The wireframes were used as a basis for producing creative concepts to establish a new stylish look and feel for the site.  Three concepts were presented, from which the CIPR chose one.
4. Standardisation: From the concept mock-ups, we derived a set of design standards. This is where we sought out any inconsistencies in the design and attacked them to produce a documented design standard that would be used as a rule book for developing the remaining layouts. The standards cover column layouts, margins, font and colour palettes, image aspect ratios and sizes, paragraph formats, heading levels, link and button styles and all interaction styling.  It’s easy to forget interaction styles – for every button, for example, you need a standard, hovered, pressed and disabled style.
5. Layout development: Using the design standards, the concepts were expanded into the dozen or so different layouts that would be used in the final site.
6. Rendering: With all layouts complete, HTML templates and an optimised stylesheet could be produced to serve all the layouts.
7. Drupal build: Skinning Drupal with the new templates, installing and configuring all the content types and modules we would need, and writing our own bespoke modules for our special requirements.
8. UI behaviours and trackers: JavaScript libraries and loaders for all the interface behaviours, with graceful degradation for users with JavaScript disabled.

Special features

Much of the functionality of the new site came out of the box with Drupal, but we added some of our own modules to enable some particular functionality:

Node image

Our design called for image based teasers for content pages.  You can see examples of these on the homepage of the site, under ‘Features’.  The image used for the feature teaser cannot be required to be part of the content of the page itself, so it has to be separately mapped to the node.  Additionally, our design standards mandated a specific aspect ratio for all images on the site, and four specific valid sizes, so all images, whether used for feature teasers or within the content of the node, must be made to fit these design rules.

To achieve this, we wrote a new module, which we call Node Image, and a TinyMCE plugin called Node Image Chooser, to replace the standard TinyMCE Image insert/edit dialog.  The Node Image module also alters the node edit form to add the ability to attach images outside the content body (to serve as feature teasers).

The form for uploading and editing node image nodes provides a cropper tool to allow the user to preview the original uploaded image (often a digicam photo at a far higher resolution than desired), and crop the four valid image sizes out of it, using a draggable frame with a constrained aspect ratio.  This allows editors to quickly create the four variants of each image, so for any image we can always offer the choice of all four possible sizes, without requiring editors to be trained in image manipulation software.

Restricted content

We needed to be able to protect some pages so that they were only accessible by CIPR members or even a smaller subset of the membership who were perhaps members of a particular regional or sectoral group.  However, access to this information is one of the most important and valuable benefits of membership, so we wanted to ensure that non-members could see that this content existed, even if they couldn’t access it themselves.

Many newspaper sites handle this situation with a concept called a content barrier.  The user is shown a teaser of the page content, and a message indicating that in order to read the remainder of the content, they must pay a fee or register.  We started with the existing Restricted Content module, but completely rewrote it to support tokens and offer completely different views to anonymous users versus those who were logged in but simply did not have the required role.  We’ve released these changes back to the community – our modified version is available on the Drupal project page.

We married this module with a bespoke (and very simple) robots login module, which identifies legitimate search engine crawlers from Google, Yahoo and Microsoft, and allow them to view all protected content as if they were members, but with a NOARCHIVE restriction.  This further contributes towards our goal of making this member only content visible to non-members by allowing them to find it via a general web search, though they still need to join in order to view the full page.

Single sign on

We didn’t just need to authenticate users for the benefit of Drupal.  We also need them to be logged in for other sites as well, notably Ladder, the CIPR’s Continuing Professional Development system.  We wanted to ensure that users did not have to log in several times to access these services.  So we turned Drupal into a single sign on hub.

Our solution starts with the Login Cookie module, modified to support tokens and shared secret based signing.  When users log in, we use this module to set a cookie containing various particulars about their user account, and a signature that can be verified by co-operating apps on the same root domain.

The second part of the solution is the need to redirect users to a querystring-specified destination on login and logout.  This is to enable users initiating login directly from an external app to be seamlessly redirected back to it after their login has been processed.  We started with the Login Destination module, but found this to be over-engineered for our needs in many ways, while it also missed a key requirement – redirect on logOUT.  So we ended up writing a bespoke module for this.

Finally, the services module gave external apps that read the SSO cookie the ability to query the user’s full Drupal user account to get the details that we did not include in the cookie.

Social sharing

Our requirements included the usual need to allow social sharing of content via Digg, Facebook, Twitter and so on, as well as the ability to send pages by email to friends, and bookmark pages in the browser.  All these were easily solved with AddThis which also adds analytics into the bargain.

Inline audio and video

The CIPR needs to be able to include videos from Youtube and Vimeo, and publishes its own video content via Vzaar, all of which have straightforward embed codes, but there’s no simple way to insert these into TinyMCE in a way that allows the designer (not editorial users) to control their dimensions and style.

We wrote a new TinyMCE plug-in to replace the media plug-in, which allows CIPR editors to find an online video and just paste the URL into the plug-in dialog.  The plug-in recognises the URL format, queries the API of the appropriate video service provider, and displays the video title to the user to confirm they have the right one.  The user can then simply choose which of our standard image sizes they wish to use for the video player – ensuring that video players are shown at exactly the same size as embedded images.

This continues a theme of replacing complex functionality with alternatives that separate content decisions from style at the admin level.  So if an editor wants to insert a video, they should not have the choice of what width and height to assign to the player.  That is a decision made by the designer, not user.  So our plug-ins don’t offer the choice.  As a result all the CIPR’s videos are presented in the same way.

Audio files are embedded using the same plug-in, recognising the file as an audio format, and embedding the excellent Google Reader inline mp3 player.

Forms

The CIPR needs loads of forms.  Membership applications, renewals, applications to join groups, entries for awards competitions, and so on.  Drupal’s capabilities to create arbitrary user defined forms and aggregate the results did not really do the job, so we turned to the service provided by Formassembly.  This third party service allows a non-specialist end user to create a wide variety of forms, and process results in many different ways, as well as providing useful connectors such as Salesforce integration and one-off Paypal payments.

We created a TinyMCE plug-in to insert verified Formassembly form IDs into a specially designed machine tag in the content of any page.  We paired this with an output filter that converted the machine tag into the full HTML of the form, making a few changes to it on the way through – tweaking Formassembly’s HTML mark-up slightly allowed it to be styled by our CSS without having to include a second set of form styles.

Search

Drupal does offer user facing search via various means, but we felt that we needed something really awesome to make up for the years of having no search on the CIPR site at all.  We wanted something that offered a great relevance engine to surface the best matching content.  In short, we wanted something like Google on a small scale.

Fortunately, for a fee, that’s exactly what we got.  The search engine on the CIPR’s new site is powered by Google Site Search, a paid-for API-driven service that builds and uses a dedicated index on Google’s servers, but is delivered and branded entirely within your own site.

Course finder and member directory: searchable views

The old site already had a directory of members, and we wanted to maintain that, and add directories of PR suppliers and courses as well.  The Better Exposed Filters module came in handy here, providing an excellent checkbox list interface to options such as level, cost and type of course.

Events calendar and Eventbrite integration

With the CIPR’s events all managed on Eventbrite, we needed some close integration between the Eventbrite accounts and the CIPR site.  Using Eventbrite’s XML feeds, we perform a regularly scheduled import of event data and use it to populate Event nodes within Drupal.  Eventbrite already provides a lot of meta data – and we add a machine tag in the Eventbrite body text to link the events into courses within the CIPR course finder.

This also enables features such as listing all available dates for a course on the course page, and allowing users to navigate to alternative dates from an event page.

Meta data

The existence of content types like courses and events made us think that it would be really useful to have the ability to assign arbitrary key-value data to any node, and display it in a table alongside the content, in a similar way to Wikipedia’s boxes.  We used CCK fields to collect the meta data and a custom block to display it.

Content types

Our full set of Drupal content types are:

• Standard page: A flexible content page
• Blog post: Automatically bylined and aggregated under an author or topic.
• Course: Anything that can loosely be described as a training course is created using the Course node type, which has additional CCK fields for properties such as length, level, type, cost and accreditation.  This then powers the course finder.
• Course provider: Used to record details of an institution that provides training courses.  A course is linked relationally to a course provider which allows course provider pages to display a list of their available courses.
• Event: A single event, normally an instance of a training course, imported from Eventbrite, linked to a course
• Forum topic: Using the forums module, forums provide discussion areas for groups
• Node Image: Our bespoke node type for constrained-size images

Optimisation

Finally, we thought we would share a few tips on optimising delivery and measurement.

• Third party JavaScript loading: We use JavaScript from a number of third parties, such as AddThis and Google.  We developed a simple scheduled task to download these libraries and cache them on our servers so that we could deliver all the JavaScript to power the site’s UI behaviours in as few downloads as possible.
• JavaScript grouping: We group all our JavaScript files into two downloads – one in the <head>, and one just before </body>.  A script in our theme concatenates the script files, and also returns 304 Not Modified responses whenever it can.
• Tracking 404s: It’s really easy when using Drupal to make the mistake of including your tracking JavaScript on your 404 error page.  Not only does this incorrectly inflate your traffic figures, but it’s a lost opportunity to create a useful broken links report.  We use Google’s recommended method for tracking 404s, and then we filter them out of the website profile we use for reporting traffic numbers.
• Caching of twitter / facebook feeds: Updates posted to the CIPR’s twitter and Facebook accounts are downloaded on a schedule and cached on our servers as fully rendered HTML to allow them to be included in our pages at the final stage of assembly.
• Feedburner: Using a simple user-agent and request-uri based redirect in an Apache rewrite rule, all end user requests for RSS feeds are diverted to Feedburner, which provides excellent analytics to let us know how many readers are consuming each of our RSS feeds, using which devices or clients.  It also reduces the load on our servers to have over 95% of our feed requests answered by Feedburner.

The logic for implementing native animations in the browser is a no-brainer, particularly if you try and use a site that implements scrolling tickers by incrementing an element’s left or margin-left properties every few milliseconds. Watch your interactions with the rest of the page slow down to a sluggish crawl and realise that animating the positioning of DOM elements in JavaScript can sometimes be pretty slow.

With animation increasingly important to creating rich and usable user interfaces, the chance to start optimising the performance of animation in the browser via native implementation is a great step forward. Unfortunately, this is currently only supported by Webkit based browsers. In developing the ticker, I wanted to see if support in Firefox would be possible.

From version 1.9.3 of Gecko, the -moz-transition family of CSS properties have been added, so I tried grabbing the latest version of Minefield (the pre-release codename for Firefox) and running the ticker plugin.

It doesn’t work. Specifically, it doesn’t work for these reasons:

1. It doesn’t seem to be possible to act on position properties. left, right, top and bottom do not appear to animate. It might be possible to work around this by switching to animating margins rather than absolute positioning, but hopefully it will be possible to animate these before the Firefox 3.7 release.
2. It doesn’t appear to fire the mozTransitionEnd event, which you’d expect from a full implementation of the standard. As a result, my ticker can’t loop.
3. It doesn’t seem to be possible to change animation properties (such as the duration) from JavaScript. I need this to disable animation while repositioning the ticker for the next loop.

This is clearly an early pre-release version, and it’s encouraging to see support starting to emerge from the Firefox camp. I think I might have retired by the time this kind of feature hits Internet Explorer!

In the process, I was pointed at the gorgeous Panic status board. Our objectives were similar and I was attracted to the idea of adding the scrolling tickers (at least, we assume they are scrolling from Panic’s photograph), which run along the bottom of their board. I was reminded of the kind of tickers you get on 24 hour news channels (here’s one from BBC World):

Tickers in web browsers are difficult for a number of reasons. First, continuously animating the left or margin-left properties of an element in JavaScript is computationally expensive, particularly if you want to reposition the element often enough to create a properly smooth animation. Sites that use Javascript for tickers in this manner tend to hog your computer’s CPU and feel slow and laggy as a result. Many sites give up on the traditional ticker and go for something that’s easier for the browser to animate, like a ‘type in’ reveal (eg the BBC News homepage ticker), or a vertical ’scroll up’ ticker.

Another issue with continuous tickers is making them loop properly. When your animation reaches the end of the text, you have to scroll it all the way off the left edge of the screen before it can be repositioned and re-enter the frame from the right. This leaves an ugly gap in your ticker.

Finally, if your ticker is updating with new information in real time, making changes that are noticeable in the visible portion of the ticker is amateur hour stuff.  You rarely see TV tickers suddenly change mid-scroll, and yet stories are seamlessly added and removed as necessary, outside of the visible frame.

Since Panic didn’t release any code (boo), I set out to produce a jQuery plugin that would enable news tickers to be easily added to a page, using CSS3 animation effects.  If it’s working for you, you should see a couple of tickers scrolling away right here:

If it’s not working, you’re probably not using a browser that supports CSS3 transitions.  Try it in Chrome, or Safari.  Support should be added to Firefox pretty soon.  But the objective here is not to get universal compatibility, since I can easily dictate which browser our status board will use.

With CSS3 transitions, we substantially improve the performance issue of tickers.  And this will improve further with browsers implementing these effects at an increasingly lower level.  This leaves the looping and updating issues to resolve before we can claim a properly news-grade ticker.

Looping

There are two possible solutions to the looping problem.  The first is to target segments of the ticker just after they have left the frame on the left, and move them to the end of the ticker on the right.  The second is to make several copies of your ticker content, and when you get to the end of the second copy, reposition the entire ticker element so that the first copy lines up with where the second copy had got to, then scroll on into the second copy again and repeat.

I chose the second approach, because you need several copies of the ticker anyway if it’s too short to fill the width of the frame with enough left over to sort the looping out.

Updates

In order to enable the ticker to be updated dynamically, and yet not suffer any noticeable changes in the visible portion, I added some public methods – addMessage() and removeMessage(), which allow you to pass a list element and have it queued for addition or removal at an appropriate moment.

This does require breaking a rather holy tenet of jQuery, namely returning a reference to the ticker rather than returning the jQuery chain, so you have an object on which you can call the public methods.

Every time the ticker hits the end of the text, it is repositioned with the penultimate copy in view, all copies of the ticker text are updated to match the copy to their right, and the final copy (the original set of elements) is updated with the queued additions and removals (out of view).   As a result, when you call the add or remove methods, you may have to wait some time before you see the effect, but the ticker will never judder or change while it is visible to the user.

How to use

Your ticker must be a block element containing a UL. This is because the ticker needs a frame to mask the content as it slides through.

<div id='ticker1'>
  <ul>
    <li>Headline one</li>
    <li>Headline two</li>
  </ul>
</div>
<script type="text/javascript">
  var ticker = $('#ticker1').ticker();
  var newid = ticker.addMsg('Headline three');
  setTimeout(function() {
    ticker.removeMsg(newid);
  }, 60000);
</script>

Settings and methods

The following settings are available, which can be passed to the ticker() function in a standard json object:

• pxpersec: Speed of the ticker in pixels per second (optional; default is 30)

Calls to the ticker() function return a reference to the ticker, not the jQuery chain. The ticker reference exposes the following methods:

• string addMsg(msg): Add a new message to the ticker. Can be a jQuery object or raw list item element (which will be moved from its current location in the DOM), or a plain text string. Returns a string identifier for the message (to enable you to remove it later). If you pass an element that already has an ID attribute, that ID is used (and returned).
• void removeMsg(msg): Remove a message from the ticker. Can be a jQuery object or raw list item element (which must be in the ticker already), or a string identifier returned from the addMsg call that added the message to the ticker. Returns nothing.

Get the code

• Download development (commented, 7KB)
• Download production (minified, 4KB)

Try scrolling either of the panels in this example:

You should find it scrolls madly and unpredictably until you press stop.

The first problem is that onscroll is not simply fired when you finish scrolling. It fires like a machine gun continuously while the mouse cursor is moving on the scroll handle. The solution to this is a watchdog. A watchdog is a timer that will execute action A if action B does not happen within X seconds. So every time onscroll fires, we reset the timer, and if it gets to zero we know that the user has finished scrolling (or at least has paused for long enough for us to do something about it).

You’ll find that although it’s not quite as crazed, the panels do keep scrolling, one after the other.

The problem now is that when you scroll DIV 1, and this causes an automatic scroll of DIV 2, that automatic scroll ALSO triggers the onscroll event and so we then scroll DIV 1 again. The solution is to have a variable that flags whether we are currently paying attention to scroll events, and turn it off when we don’t want to detect scrolling (ie just before the auto-scroll) and then on again after the scroll has completed (I’m using an animation library so the scroll takes around half a second to complete). Try this (don’t click the button yet, just scroll the panels):

Success. However, this approach is a bit short-sighted.

You also need to turn off scroll detection when adding or removing content from either DIV, because changing the height of the content within the element can also fire the onscroll event. So if this is a chat window, and we’re adding and removing content in the DIVs, we have to disable scroll detection while we’re doing that and then turn it on again afterwards.

Sadly, our scroll detection flag is crude – it’s a sheer yes/no, and if it’s already set to no (say in order to execute a lengthy scrolling animation), and in the meantime we need to do something quick (say adding a line to one of the DIVs) then that quick operation is going to disable scroll detection (unnecessarily, as it’s already disabled) and then crucially enable it again before the scroll animation has finished. Click the button in the example above to demonstrate this (and then scroll). You should, intermittently, get the unwanted cascade effect you saw in example 2.

What we need is a list, not a flag. Enter the blocker list – an object that collates tokens from each procedure in the script that is currently blocking scroll detection. So if a procedure wants to disable scrolling for a period of time, it adds its token to the blocker list, and then removes its token when it’s done. When a scroll event fires, we now only need to know whether there are any items in the blocker list, and then we can work out if it’s ok to process the event.

It’s also worth noting that there’s no need to actually count the number of items in the blocker list – it’s enough simply to know that there is at least one item in there. And as a further optimisation, we don’t actually need to compute this when scroll events fire (frequently), because the value will only change when some function wants to add or remove its token from the list (less frequent). So we can have enableScrollDetection and disableScrollDetection functions that deal with the blocker list, and which ultimately simply change the old scrolldetection flag to true or false.

There might be a neater way of achieving this, but this certainly works for me. I’d welcome any comments or suggestions. The sources for all these demos are available in the iframes above.

But this was clearly not my problem, and various sources suggested that this had in any case been fixed for Firefox 3. I discovered that my problem was rather simpler. If you disable an element that has focus, and then re-enable it, you don’t get the cursor back.

Solution: blur (remove focus from) the element before you disable it. Those sources that do refer to this specific problem tend to suggest that you focus a different element before you disable the text box. But this is not necessary – you can just blur the element that has focus and leave the page with nothing focused.

Problem test case

Try clicking and typing in the field below – if your browser exhibits this problem, the text cursor will not appear. If it does, then your browser does not have this problem.

This demo has a single text field which, when you click into it, is briefly disabled, and then enabled again. The disabling of the field while it has focus triggers this problem, so the text cursor does not appear (but you can still type into the field!).

Solution test case

Now try typing in this field – the cursor should appear consistently.

This demo still briefly disables and re-enables the text field when it gets focus, but this time it blurs it right before it’s disabled, then focuses it again after re-enabling it. The result is that when you place your mouse cursor in the field to manually give it focus, the text cursor appears as normal.

This problem is often solved with complex classes or functions in PHP that are designed to strip out the nasty stuff while allowing as much useful formatting as possible. We realised that these functions are pretty much just reinventing the wheel, because there is already a pretty good mechanism for parsing and validating XML syntax: libxml, which has PHP bindings and can be accessed using SimpleXML.

What’s more, libxml can parse an XML document for conformance to a DTD, so if you include an XHTML Transitional DTD in your XML code string, you can check that the markup is valid XHTML.

Here’s the PHP to do this. This is tested on PHP 5.3 with libxml2-2.6.26-2.1.2.8.

function isXML($str) {
	libxml_use_internal_errors(true);
	libxml_clear_errors();
	$options = (strpos($str, '<!DOCTYPE') !== false) ? (LIBXML_DTDLOAD + LIBXML_DTDVALID) : 0;
	simplexml_load_string($str, 'SimpleXMLElement', $options);
	$errors = libxml_get_errors();
	return (empty($errors) or $errors[0]->level == LIBXML_ERR_WARNING) ? true : false;
}

You could of course use the contents of $errors to feed back to the user, or potentially deal with a validation failure more intelligently, but for now true or false will do.

So the markup submitted by a user is valid. Excellent. But just because the markup is valid doesn’t mean it’s safe to output to the browser. You’ll also want to ensure it contains no <script type="text/javascript"> sections or event handlers, and may want to restrict the set of elements available. This is where you can start getting creative with your own DTD spec. Just start with the standard you want to conform to for the whole page (say XHTML) and strip out anything you don’t like.

We’ll start by removing the HEAD tag and all its contents. Our users will not be writing entire documents, just fragments of body markup, so we don’t want a HEAD, TITLE, or any META tags, etc.

You can continue, removing things like SCRIPT, OBJECT, forms, frames, and so on. Be careful where elements are defined using presets, which often contain the nasties, for example the %event set of attributes grants an element the ability to fire event handlers. Fortunately this is almost exclusively used as part of %attrs, so we can just remove it from that superset.

We’ll also define a new root element fragment_under_test to ensure that we don’t cause any confusion and lead anyone to believe that they’re writing a normal <html> or <body>.

Once we’re done, we can then wrap the isXML function in a convenience function that adds our new custom DTD.

function isXHTMLFragment($str) {
	return isXML("<!DOCTYPE fragment_under_test system \"http://www.example.com/dtds/xhtml-content-restrictive.dtd\"><fragment_under_test>".$str."</fragment_under_test>");
}

If you want, feel free to download the DTD I created for this article.

Now you can use the fast libxml to validate user input in a fairly bulletproof way.

Finally, and very importantly, make sure you cache the schemas on your server in an XML catalog file. If you don’t do this, libxml will make an external HTTP request for the DTD schema file every time you call the function. In fact, since most web documents cite W3C DTDs, they are having enormous problems with software making repeated requests for the standard XHTML, HTML 4 etc DTDs which haven’t changed in years. Be a good net citizen, and cache your schemas. In this case we’re writing and hosting our own anyway, but if you’re using a public schema you may as well save yourself the pointless HTTP traffic, and it’ll speed up the validation as well.

Recently I needed to use this method to serialise some data in a JavaScript library that might be used in ‘foreign’ web pages. My own library was nicely encapsulated, and didn’t interfere with any other JavaScript that might be running on the page, and it included Douglas Crockford’s JSON2.js implementation.

But on one of our clients’ sites, it didn’t work. I got this:

{"key":"val",[\{\"key\":\"val\"\},\{\"key\":\"val\"\}]}

What’s happened here is that any arrays in my data structure have been stringified twice. This didn’t happen in my dev environment. I narrowed down the differences and realised what was causing this effect. They’re using Prototype. We’re not.

Prototype modifies a number of JavaScript’s native objects, including the Array object, and… you guessed it, adds a toJSON() method to it. Unfortunately it does not return what Crockford’s JSON implementation is expecting. From the docs for json2:

A toJSON method does not serialize: it returns the value represented by the name/value pair that should be serialized, or undefined if nothing should be serialized.

Prototype’s toJSON() is serialising. There don’t seem to be any sensible solutions to this online, but it’s actually relatively simple to solve using a replacer function, allowed for in the json2 API:

var reqdata = JSON.stringify(req, function(key, value) {
	if (typeof this[key] == 'object' && Object.prototype.toString.apply(this[key]) === '[object Array]') {
		return this[key];
	} else {
		return value;
	}
});

Essentially this says ‘for each key in the data structure, if the value is an array, use the raw value, otherwise use the value you gave me’. This makes sense when you look at the sequence of steps that stringify() goes through for each key it encounters:

1. If the value has a toJSON() method, call it.
2. If a replacer function has been given, call it.
3. If the remaining value is a scalar, return it.
4. If the remaining value is an object, stringify each member, then concatenate keys and values within braces {key:val,key:val}
5. If the remaining value is an array, stringify each element, then concatenate values within brackets [val,val,val]

So, stringify() has already called Prototype’s toJSON() method by the time it executes the replacer function, but we can use the replacer function to restore the original value, allowing stringify() to then deal with the array by calling itself recursively.

The result is that we can ensure that even if a toJSON() method does exist on the Array object, its output is ignored, and we then get the JSON string that we wanted. ]]> http://assanka.net/content/tech/2009/09/02/json2-js-vs-prototype/feed/ 2