We chose to use Drupal – having successfully built Drupal sites for the likes of News International, we knew we could hit the ground running and end up with something that would be able to serve the CIPR for several years.

The challenges

The existing site had 30,000 static HTML files, with widely varying markup and no style consistency.  Because the site was really difficult to edit, many of the most active areas had been spun out into microsites with an asp.net based proprietary content management system.  These microsites were editable by their respective content owners, but with no oversight, and with complete content control available to the editors, so styles began to diverge even more.

A number of third party services and applications needed to be integrated into the new site. Eventbrite for event ticketing, ezProxy for access to academic journal services, and the CIPR’s own Ladder CPD system (another Assanka project).  For Ladder and ezProxy, we needed to be able to authenticate CIPR members outside of Drupal, so we would need to have some kind of single sign on system centered on the Drupal site.

Details about the members themselves are held in the CIPR’s membership system, a client-server application with a Windows front end and a Microsoft SQL Server database.  On the old site, synchronising this with the database the website was using for authentication required a long manual process and, for some reason, a number of Microsoft Word mail merges. The membership system is here to stay, so our solution would need to vastly improve on this process to get member profile updates into the website far faster than was possible before.

Finally, it’s worth mentioning the workflow challenges. Under the old site, the institute’s staff had to email their requested changes to the web team using a Microsoft Word based form. The web team would implement them “within 5 working days”, by editing static HTML files manually, and could not provide any content oversight, only the technical support to put the content online. For a new site to be successful, the management and oversight of the institute’s web output would need to change completely.

Build process

Our build process followed a fairly typical sequence:

1. Site map workshops: We gathered together and worked with the main content creators and heads of department. Over a number of sessions, and using a lot of Post-it notes and a very large wall, we produced a site map.  The site map was transferred to Mindmeister, where it continued to be collaboratively edited and refined.
2. Wireframing: We produced wireframes of a new home page and typical article pages, to explore the organisation and prioritisation of content.  We needed a number of different, flexible content positions to accommodate a variety of content and media.
3. Creative concepts: The wireframes were used as a basis for producing creative concepts to establish a new stylish look and feel for the site.  Three concepts were presented, from which the CIPR chose one.
4. Standardisation: From the concept mock-ups, we derived a set of design standards. This is where we sought out any inconsistencies in the design and attacked them to produce a documented design standard that would be used as a rule book for developing the remaining layouts. The standards cover column layouts, margins, font and colour palettes, image aspect ratios and sizes, paragraph formats, heading levels, link and button styles and all interaction styling.  It’s easy to forget interaction styles – for every button, for example, you need a standard, hovered, pressed and disabled style.
5. Layout development: Using the design standards, the concepts were expanded into the dozen or so different layouts that would be used in the final site.
6. Rendering: With all layouts complete, HTML templates and an optimised stylesheet could be produced to serve all the layouts.
7. Drupal build: Skinning Drupal with the new templates, installing and configuring all the content types and modules we would need, and writing our own bespoke modules for our special requirements.
8. UI behaviours and trackers: JavaScript libraries and loaders for all the interface behaviours, with graceful degradation for users with JavaScript disabled.

Special features

Much of the functionality of the new site came out of the box with Drupal, but we added some of our own modules to enable some particular functionality:

Node image

Our design called for image based teasers for content pages.  You can see examples of these on the homepage of the site, under ‘Features’.  The image used for the feature teaser cannot be required to be part of the content of the page itself, so it has to be separately mapped to the node.  Additionally, our design standards mandated a specific aspect ratio for all images on the site, and four specific valid sizes, so all images, whether used for feature teasers or within the content of the node, must be made to fit these design rules.

To achieve this, we wrote a new module, which we call Node Image, and a TinyMCE plugin called Node Image Chooser, to replace the standard TinyMCE Image insert/edit dialog.  The Node Image module also alters the node edit form to add the ability to attach images outside the content body (to serve as feature teasers).

The form for uploading and editing node image nodes provides a cropper tool to allow the user to preview the original uploaded image (often a digicam photo at a far higher resolution than desired), and crop the four valid image sizes out of it, using a draggable frame with a constrained aspect ratio.  This allows editors to quickly create the four variants of each image, so for any image we can always offer the choice of all four possible sizes, without requiring editors to be trained in image manipulation software.

Restricted content

We needed to be able to protect some pages so that they were only accessible by CIPR members or even a smaller subset of the membership who were perhaps members of a particular regional or sectoral group.  However, access to this information is one of the most important and valuable benefits of membership, so we wanted to ensure that non-members could see that this content existed, even if they couldn’t access it themselves.

Many newspaper sites handle this situation with a concept called a content barrier.  The user is shown a teaser of the page content, and a message indicating that in order to read the remainder of the content, they must pay a fee or register.  We started with the existing Restricted Content module, but completely rewrote it to support tokens and offer completely different views to anonymous users versus those who were logged in but simply did not have the required role.  We’ve released these changes back to the community – our modified version is available on the Drupal project page.

We married this module with a bespoke (and very simple) robots login module, which identifies legitimate search engine crawlers from Google, Yahoo and Microsoft, and allow them to view all protected content as if they were members, but with a NOARCHIVE restriction.  This further contributes towards our goal of making this member only content visible to non-members by allowing them to find it via a general web search, though they still need to join in order to view the full page.

Single sign on

We didn’t just need to authenticate users for the benefit of Drupal.  We also need them to be logged in for other sites as well, notably Ladder, the CIPR’s Continuing Professional Development system.  We wanted to ensure that users did not have to log in several times to access these services.  So we turned Drupal into a single sign on hub.

Our solution starts with the Login Cookie module, modified to support tokens and shared secret based signing.  When users log in, we use this module to set a cookie containing various particulars about their user account, and a signature that can be verified by co-operating apps on the same root domain.

The second part of the solution is the need to redirect users to a querystring-specified destination on login and logout.  This is to enable users initiating login directly from an external app to be seamlessly redirected back to it after their login has been processed.  We started with the Login Destination module, but found this to be over-engineered for our needs in many ways, while it also missed a key requirement – redirect on logOUT.  So we ended up writing a bespoke module for this.

Finally, the services module gave external apps that read the SSO cookie the ability to query the user’s full Drupal user account to get the details that we did not include in the cookie.

Social sharing

Our requirements included the usual need to allow social sharing of content via Digg, Facebook, Twitter and so on, as well as the ability to send pages by email to friends, and bookmark pages in the browser.  All these were easily solved with AddThis which also adds analytics into the bargain.

Inline audio and video

The CIPR needs to be able to include videos from Youtube and Vimeo, and publishes its own video content via Vzaar, all of which have straightforward embed codes, but there’s no simple way to insert these into TinyMCE in a way that allows the designer (not editorial users) to control their dimensions and style.

We wrote a new TinyMCE plug-in to replace the media plug-in, which allows CIPR editors to find an online video and just paste the URL into the plug-in dialog.  The plug-in recognises the URL format, queries the API of the appropriate video service provider, and displays the video title to the user to confirm they have the right one.  The user can then simply choose which of our standard image sizes they wish to use for the video player – ensuring that video players are shown at exactly the same size as embedded images.

This continues a theme of replacing complex functionality with alternatives that separate content decisions from style at the admin level.  So if an editor wants to insert a video, they should not have the choice of what width and height to assign to the player.  That is a decision made by the designer, not user.  So our plug-ins don’t offer the choice.  As a result all the CIPR’s videos are presented in the same way.

Audio files are embedded using the same plug-in, recognising the file as an audio format, and embedding the excellent Google Reader inline mp3 player.

Forms

The CIPR needs loads of forms.  Membership applications, renewals, applications to join groups, entries for awards competitions, and so on.  Drupal’s capabilities to create arbitrary user defined forms and aggregate the results did not really do the job, so we turned to the service provided by Formassembly.  This third party service allows a non-specialist end user to create a wide variety of forms, and process results in many different ways, as well as providing useful connectors such as Salesforce integration and one-off Paypal payments.

We created a TinyMCE plug-in to insert verified Formassembly form IDs into a specially designed machine tag in the content of any page.  We paired this with an output filter that converted the machine tag into the full HTML of the form, making a few changes to it on the way through – tweaking Formassembly’s HTML mark-up slightly allowed it to be styled by our CSS without having to include a second set of form styles.

Search

Drupal does offer user facing search via various means, but we felt that we needed something really awesome to make up for the years of having no search on the CIPR site at all.  We wanted something that offered a great relevance engine to surface the best matching content.  In short, we wanted something like Google on a small scale.

Fortunately, for a fee, that’s exactly what we got.  The search engine on the CIPR’s new site is powered by Google Site Search, a paid-for API-driven service that builds and uses a dedicated index on Google’s servers, but is delivered and branded entirely within your own site.

Course finder and member directory: searchable views

The old site already had a directory of members, and we wanted to maintain that, and add directories of PR suppliers and courses as well.  The Better Exposed Filters module came in handy here, providing an excellent checkbox list interface to options such as level, cost and type of course.

Events calendar and Eventbrite integration

With the CIPR’s events all managed on Eventbrite, we needed some close integration between the Eventbrite accounts and the CIPR site.  Using Eventbrite’s XML feeds, we perform a regularly scheduled import of event data and use it to populate Event nodes within Drupal.  Eventbrite already provides a lot of meta data – and we add a machine tag in the Eventbrite body text to link the events into courses within the CIPR course finder.

This also enables features such as listing all available dates for a course on the course page, and allowing users to navigate to alternative dates from an event page.

Meta data

The existence of content types like courses and events made us think that it would be really useful to have the ability to assign arbitrary key-value data to any node, and display it in a table alongside the content, in a similar way to Wikipedia’s boxes.  We used CCK fields to collect the meta data and a custom block to display it.

Content types

Our full set of Drupal content types are:

• Standard page: A flexible content page
• Blog post: Automatically bylined and aggregated under an author or topic.
• Course: Anything that can loosely be described as a training course is created using the Course node type, which has additional CCK fields for properties such as length, level, type, cost and accreditation.  This then powers the course finder.
• Course provider: Used to record details of an institution that provides training courses.  A course is linked relationally to a course provider which allows course provider pages to display a list of their available courses.
• Event: A single event, normally an instance of a training course, imported from Eventbrite, linked to a course
• Forum topic: Using the forums module, forums provide discussion areas for groups
• Node Image: Our bespoke node type for constrained-size images

Optimisation

Finally, we thought we would share a few tips on optimising delivery and measurement.

• Third party JavaScript loading: We use JavaScript from a number of third parties, such as AddThis and Google.  We developed a simple scheduled task to download these libraries and cache them on our servers so that we could deliver all the JavaScript to power the site’s UI behaviours in as few downloads as possible.
• JavaScript grouping: We group all our JavaScript files into two downloads – one in the <head>, and one just before </body>.  A script in our theme concatenates the script files, and also returns 304 Not Modified responses whenever it can.
• Tracking 404s: It’s really easy when using Drupal to make the mistake of including your tracking JavaScript on your 404 error page.  Not only does this incorrectly inflate your traffic figures, but it’s a lost opportunity to create a useful broken links report.  We use Google’s recommended method for tracking 404s, and then we filter them out of the website profile we use for reporting traffic numbers.
• Caching of twitter / facebook feeds: Updates posted to the CIPR’s twitter and Facebook accounts are downloaded on a schedule and cached on our servers as fully rendered HTML to allow them to be included in our pages at the final stage of assembly.
• Feedburner: Using a simple user-agent and request-uri based redirect in an Apache rewrite rule, all end user requests for RSS feeds are diverted to Feedburner, which provides excellent analytics to let us know how many readers are consuming each of our RSS feeds, using which devices or clients.  It also reduces the load on our servers to have over 95% of our feed requests answered by Feedburner.

The logic for implementing native animations in the browser is a no-brainer, particularly if you try and use a site that implements scrolling tickers by incrementing an element’s left or margin-left properties every few milliseconds. Watch your interactions with the rest of the page slow down to a sluggish crawl and realise that animating the positioning of DOM elements in JavaScript can sometimes be pretty slow.

With animation increasingly important to creating rich and usable user interfaces, the chance to start optimising the performance of animation in the browser via native implementation is a great step forward. Unfortunately, this is currently only supported by Webkit based browsers. In developing the ticker, I wanted to see if support in Firefox would be possible.

From version 1.9.3 of Gecko, the -moz-transition family of CSS properties have been added, so I tried grabbing the latest version of Minefield (the pre-release codename for Firefox) and running the ticker plugin.

It doesn’t work. Specifically, it doesn’t work for these reasons:

1. It doesn’t seem to be possible to act on position properties. left, right, top and bottom do not appear to animate. It might be possible to work around this by switching to animating margins rather than absolute positioning, but hopefully it will be possible to animate these before the Firefox 3.7 release.
2. It doesn’t appear to fire the mozTransitionEnd event, which you’d expect from a full implementation of the standard. As a result, my ticker can’t loop.
3. It doesn’t seem to be possible to change animation properties (such as the duration) from JavaScript. I need this to disable animation while repositioning the ticker for the next loop.

This is clearly an early pre-release version, and it’s encouraging to see support starting to emerge from the Firefox camp. I think I might have retired by the time this kind of feature hits Internet Explorer!

In the process, I was pointed at the gorgeous Panic status board. Our objectives were similar and I was attracted to the idea of adding the scrolling tickers (at least, we assume they are scrolling from Panic’s photograph), which run along the bottom of their board. I was reminded of the kind of tickers you get on 24 hour news channels (here’s one from BBC World):

Tickers in web browsers are difficult for a number of reasons. First, continuously animating the left or margin-left properties of an element in JavaScript is computationally expensive, particularly if you want to reposition the element often enough to create a properly smooth animation. Sites that use Javascript for tickers in this manner tend to hog your computer’s CPU and feel slow and laggy as a result. Many sites give up on the traditional ticker and go for something that’s easier for the browser to animate, like a ‘type in’ reveal (eg the BBC News homepage ticker), or a vertical ’scroll up’ ticker.

Another issue with continuous tickers is making them loop properly. When your animation reaches the end of the text, you have to scroll it all the way off the left edge of the screen before it can be repositioned and re-enter the frame from the right. This leaves an ugly gap in your ticker.

Finally, if your ticker is updating with new information in real time, making changes that are noticeable in the visible portion of the ticker is amateur hour stuff.  You rarely see TV tickers suddenly change mid-scroll, and yet stories are seamlessly added and removed as necessary, outside of the visible frame.

Since Panic didn’t release any code (boo), I set out to produce a jQuery plugin that would enable news tickers to be easily added to a page, using CSS3 animation effects.  If it’s working for you, you should see a couple of tickers scrolling away right here:

If it’s not working, you’re probably not using a browser that supports CSS3 transitions.  Try it in Chrome, or Safari.  Support should be added to Firefox pretty soon.  But the objective here is not to get universal compatibility, since I can easily dictate which browser our status board will use.

With CSS3 transitions, we substantially improve the performance issue of tickers.  And this will improve further with browsers implementing these effects at an increasingly lower level.  This leaves the looping and updating issues to resolve before we can claim a properly news-grade ticker.

Looping

There are two possible solutions to the looping problem.  The first is to target segments of the ticker just after they have left the frame on the left, and move them to the end of the ticker on the right.  The second is to make several copies of your ticker content, and when you get to the end of the second copy, reposition the entire ticker element so that the first copy lines up with where the second copy had got to, then scroll on into the second copy again and repeat.

I chose the second approach, because you need several copies of the ticker anyway if it’s too short to fill the width of the frame with enough left over to sort the looping out.

Updates

In order to enable the ticker to be updated dynamically, and yet not suffer any noticeable changes in the visible portion, I added some public methods – addMessage() and removeMessage(), which allow you to pass a list element and have it queued for addition or removal at an appropriate moment.

This does require breaking a rather holy tenet of jQuery, namely returning a reference to the ticker rather than returning the jQuery chain, so you have an object on which you can call the public methods.

Every time the ticker hits the end of the text, it is repositioned with the penultimate copy in view, all copies of the ticker text are updated to match the copy to their right, and the final copy (the original set of elements) is updated with the queued additions and removals (out of view).   As a result, when you call the add or remove methods, you may have to wait some time before you see the effect, but the ticker will never judder or change while it is visible to the user.

How to use

Your ticker must be a block element containing a UL. This is because the ticker needs a frame to mask the content as it slides through.

<div id='ticker1'>
  <ul>
    <li>Headline one</li>
    <li>Headline two</li>
  </ul>
</div>
<script type="text/javascript">
  var ticker = $('#ticker1').ticker();
  var newid = ticker.addMsg('Headline three');
  setTimeout(function() {
    ticker.removeMsg(newid);
  }, 60000);
</script>

Settings and methods

The following settings are available, which can be passed to the ticker() function in a standard json object:

• pxpersec: Speed of the ticker in pixels per second (optional; default is 30)

Calls to the ticker() function return a reference to the ticker, not the jQuery chain. The ticker reference exposes the following methods:

• string addMsg(msg): Add a new message to the ticker. Can be a jQuery object or raw list item element (which will be moved from its current location in the DOM), or a plain text string. Returns a string identifier for the message (to enable you to remove it later). If you pass an element that already has an ID attribute, that ID is used (and returned).
• void removeMsg(msg): Remove a message from the ticker. Can be a jQuery object or raw list item element (which must be in the ticker already), or a string identifier returned from the addMsg call that added the message to the ticker. Returns nothing.

Get the code

• Download development (commented, 7KB)
• Download production (minified, 4KB)

Try scrolling either of the panels in this example:

You should find it scrolls madly and unpredictably until you press stop.

The first problem is that onscroll is not simply fired when you finish scrolling. It fires like a machine gun continuously while the mouse cursor is moving on the scroll handle. The solution to this is a watchdog. A watchdog is a timer that will execute action A if action B does not happen within X seconds. So every time onscroll fires, we reset the timer, and if it gets to zero we know that the user has finished scrolling (or at least has paused for long enough for us to do something about it).

You’ll find that although it’s not quite as crazed, the panels do keep scrolling, one after the other.

The problem now is that when you scroll DIV 1, and this causes an automatic scroll of DIV 2, that automatic scroll ALSO triggers the onscroll event and so we then scroll DIV 1 again. The solution is to have a variable that flags whether we are currently paying attention to scroll events, and turn it off when we don’t want to detect scrolling (ie just before the auto-scroll) and then on again after the scroll has completed (I’m using an animation library so the scroll takes around half a second to complete). Try this (don’t click the button yet, just scroll the panels):

Success. However, this approach is a bit short-sighted.

You also need to turn off scroll detection when adding or removing content from either DIV, because changing the height of the content within the element can also fire the onscroll event. So if this is a chat window, and we’re adding and removing content in the DIVs, we have to disable scroll detection while we’re doing that and then turn it on again afterwards.

Sadly, our scroll detection flag is crude – it’s a sheer yes/no, and if it’s already set to no (say in order to execute a lengthy scrolling animation), and in the meantime we need to do something quick (say adding a line to one of the DIVs) then that quick operation is going to disable scroll detection (unnecessarily, as it’s already disabled) and then crucially enable it again before the scroll animation has finished. Click the button in the example above to demonstrate this (and then scroll). You should, intermittently, get the unwanted cascade effect you saw in example 2.

What we need is a list, not a flag. Enter the blocker list – an object that collates tokens from each procedure in the script that is currently blocking scroll detection. So if a procedure wants to disable scrolling for a period of time, it adds its token to the blocker list, and then removes its token when it’s done. When a scroll event fires, we now only need to know whether there are any items in the blocker list, and then we can work out if it’s ok to process the event.

It’s also worth noting that there’s no need to actually count the number of items in the blocker list – it’s enough simply to know that there is at least one item in there. And as a further optimisation, we don’t actually need to compute this when scroll events fire (frequently), because the value will only change when some function wants to add or remove its token from the list (less frequent). So we can have enableScrollDetection and disableScrollDetection functions that deal with the blocker list, and which ultimately simply change the old scrolldetection flag to true or false.

There might be a neater way of achieving this, but this certainly works for me. I’d welcome any comments or suggestions. The sources for all these demos are available in the iframes above.

But this was clearly not my problem, and various sources suggested that this had in any case been fixed for Firefox 3. I discovered that my problem was rather simpler. If you disable an element that has focus, and then re-enable it, you don’t get the cursor back.

Solution: blur (remove focus from) the element before you disable it. Those sources that do refer to this specific problem tend to suggest that you focus a different element before you disable the text box. But this is not necessary – you can just blur the element that has focus and leave the page with nothing focused.

Problem test case

Try clicking and typing in the field below – if your browser exhibits this problem, the text cursor will not appear. If it does, then your browser does not have this problem.

This demo has a single text field which, when you click into it, is briefly disabled, and then enabled again. The disabling of the field while it has focus triggers this problem, so the text cursor does not appear (but you can still type into the field!).

Solution test case

Now try typing in this field – the cursor should appear consistently.

This demo still briefly disables and re-enables the text field when it gets focus, but this time it blurs it right before it’s disabled, then focuses it again after re-enabling it. The result is that when you place your mouse cursor in the field to manually give it focus, the text cursor appears as normal.

This problem is often solved with complex classes or functions in PHP that are designed to strip out the nasty stuff while allowing as much useful formatting as possible. We realised that these functions are pretty much just reinventing the wheel, because there is already a pretty good mechanism for parsing and validating XML syntax: libxml, which has PHP bindings and can be accessed using SimpleXML.

What’s more, libxml can parse an XML document for conformance to a DTD, so if you include an XHTML Transitional DTD in your XML code string, you can check that the markup is valid XHTML.

Here’s the PHP to do this. This is tested on PHP 5.3 with libxml2-2.6.26-2.1.2.8.

function isXML($str) {
	libxml_use_internal_errors(true);
	libxml_clear_errors();
	$options = (strpos($str, '<!DOCTYPE') !== false) ? (LIBXML_DTDLOAD + LIBXML_DTDVALID) : 0;
	simplexml_load_string($str, 'SimpleXMLElement', $options);
	$errors = libxml_get_errors();
	return (empty($errors) or $errors[0]->level == LIBXML_ERR_WARNING) ? true : false;
}

You could of course use the contents of $errors to feed back to the user, or potentially deal with a validation failure more intelligently, but for now true or false will do.

So the markup submitted by a user is valid. Excellent. But just because the markup is valid doesn’t mean it’s safe to output to the browser. You’ll also want to ensure it contains no <script type="text/javascript"> sections or event handlers, and may want to restrict the set of elements available. This is where you can start getting creative with your own DTD spec. Just start with the standard you want to conform to for the whole page (say XHTML) and strip out anything you don’t like.

We’ll start by removing the HEAD tag and all its contents. Our users will not be writing entire documents, just fragments of body markup, so we don’t want a HEAD, TITLE, or any META tags, etc.

You can continue, removing things like SCRIPT, OBJECT, forms, frames, and so on. Be careful where elements are defined using presets, which often contain the nasties, for example the %event set of attributes grants an element the ability to fire event handlers. Fortunately this is almost exclusively used as part of %attrs, so we can just remove it from that superset.

We’ll also define a new root element fragment_under_test to ensure that we don’t cause any confusion and lead anyone to believe that they’re writing a normal <html> or <body>.

Once we’re done, we can then wrap the isXML function in a convenience function that adds our new custom DTD.

function isXHTMLFragment($str) {
	return isXML("<!DOCTYPE fragment_under_test system \"http://www.example.com/dtds/xhtml-content-restrictive.dtd\"><fragment_under_test>".$str."</fragment_under_test>");
}

If you want, feel free to download the DTD I created for this article.

Now you can use the fast libxml to validate user input in a fairly bulletproof way.

Finally, and very importantly, make sure you cache the schemas on your server in an XML catalog file. If you don’t do this, libxml will make an external HTTP request for the DTD schema file every time you call the function. In fact, since most web documents cite W3C DTDs, they are having enormous problems with software making repeated requests for the standard XHTML, HTML 4 etc DTDs which haven’t changed in years. Be a good net citizen, and cache your schemas. In this case we’re writing and hosting our own anyway, but if you’re using a public schema you may as well save yourself the pointless HTTP traffic, and it’ll speed up the validation as well.

Recently I needed to use this method to serialise some data in a JavaScript library that might be used in ‘foreign’ web pages. My own library was nicely encapsulated, and didn’t interfere with any other JavaScript that might be running on the page, and it included Douglas Crockford’s JSON2.js implementation.

But on one of our clients’ sites, it didn’t work. I got this:

{"key":"val",[\{\"key\":\"val\"\},\{\"key\":\"val\"\}]}

What’s happened here is that any arrays in my data structure have been stringified twice. This didn’t happen in my dev environment. I narrowed down the differences and realised what was causing this effect. They’re using Prototype. We’re not.

Prototype modifies a number of JavaScript’s native objects, including the Array object, and… you guessed it, adds a toJSON() method to it. Unfortunately it does not return what Crockford’s JSON implementation is expecting. From the docs for json2:

A toJSON method does not serialize: it returns the value represented by the name/value pair that should be serialized, or undefined if nothing should be serialized.

Prototype’s toJSON() is serialising. There don’t seem to be any sensible solutions to this online, but it’s actually relatively simple to solve using a replacer function, allowed for in the json2 API:

var reqdata = JSON.stringify(req, function(key, value) {
	if (typeof this[key] == 'object' && Object.prototype.toString.apply(this[key]) === '[object Array]') {
		return this[key];
	} else {
		return value;
	}
});

Essentially this says ‘for each key in the data structure, if the value is an array, use the raw value, otherwise use the value you gave me’. This makes sense when you look at the sequence of steps that stringify() goes through for each key it encounters:

1. If the value has a toJSON() method, call it.
2. If a replacer function has been given, call it.
3. If the remaining value is a scalar, return it.
4. If the remaining value is an object, stringify each member, then concatenate keys and values within braces {key:val,key:val}
5. If the remaining value is an array, stringify each element, then concatenate values within brackets [val,val,val]

So, stringify() has already called Prototype’s toJSON() method by the time it executes the replacer function, but we can use the replacer function to restore the original value, allowing stringify() to then deal with the array by calling itself recursively.

The result is that we can ensure that even if a toJSON() method does exist on the Array object, its output is ignored, and we then get the JSON string that we wanted. ]]> http://assanka.net/content/tech/2009/09/02/json2-js-vs-prototype/feed/ 2 http://assanka.net/content/tech/2009/09/01/personal-data-in-cookies/ http://assanka.net/content/tech/2009/09/01/personal-data-in-cookies/#comments Tue, 01 Sep 2009 16:12:40 +0000 Andrew Betts http://assanka.net/content/tech/?p=9 Personal data in cookies is bad. I accepted this for quite some time, until I actually stopped to think about it, and realised how much simpler our scaling would be if we could store a lot more state information in cookies.

So… conventional wisdom. This is bad:

myCookie=Andrew,andrew@trib.tv,123456,TE51 7NG,4,2009-05-22 17:43:06,138cb381a04dba00340ab450deea934;

This imaginary cookie contains my display name, email address, user identifier, postcode, user level, cookie generation time, and security signature.

In my experience the first argument people have against the data I have in my cookie is that ‘the user could just change it! You’ve got their permissions level in there and everything. Are you mad!?’.

The ‘user could just change it’ argument

Yes, they could. So in order to solve this problem you only need to make it impossible for a user to write a legitimate cookie value, or edit a cookie such that it remains legitimate. This is trivially simple to do and validate, much more so than implementing a scalable session store at the server end.

All you do is include, within your cookie, two additional items – the generation time of the cookie, and a signature string. The signature string is the result of running the rest of the cookie (excluding the signature), AND a constant value NOT included in the cookie, through a hashing algorithm, such as md5, crc32 or sha1 (use sha1 for preference).

Here’s an example, using the cookie above. The constant value not included in the cookie (normally a string of gibberish) will for our purposes be ‘password’ (obviously you should use something a lot more random than that). Sticking all the parameters of the cookie together, and adding the super-secure, super secret constant on the end, we get:

Andrewandrew@trib.tv123456TE51 7NG42009-05-22 17:43:06password

Then we run this through our hashing algorithm of choice, and end up with:

138cb381a04dba00340ab450deea934

Which we pop on the end of the cookie before it is sent to the user in a Set-Cookie header. When that cookie is then received in a subsequent request (to a different server), the server only needs to know the super-secret constant and recalculate the hash to be able to verify that whoever wrote the cookie ALSO knows the super secret constant. The user doesn’t know it, so the user can’t write a cookie that we’re going to believe. No matter how much the user wants to get to user level 10, they’re just gonna have to earn it.

But, wail the doubters, you can see all the data on the wire in plain text!

The ‘you can see it on the wire’ argument

When you put together the web page your user has requested – your shopping cart or social network or Tom Jones fan page or whatever – you are probably going to print a nice friendly message in the top right of the page that says ‘Hello Andrew!’ with links to exciting places such as ‘My Account’, ‘Preferences’ and ‘Log out’. When the user goes to the my account page, you’re going to include loads more details as well, like their email address, phone number, postcode, and so on. And of course because your app has a ton of JavaScript making life easy for the user, you’ve probably got their email address in a JavaScript variable somewhere so you can prepopulate ‘email a friend’ dialogue boxes and overlays.

So all the content that you object to including in a cookie, you are including in the HTML responses that you send back to the user. If it’s OK in the response, why not allow it in the request?

There is a possible sub argument here: using DNS cache poisoning an attacker could intercept a request more easily than they could intercept a response. But if your site is at risk from attempts at DNS poisoning (which is a lot more difficult than it used to be), then you probably have bigger worries than just leaking some personal data.

Conclusion

In many cases, storing personal data in cookies is a great idea, and is not a privacy issue. If you’re storing something that you’re not intending to use in JavaScript, and want to make it less readable (though not actually secure) you could easily obfuscate it, or make it actually secure using a cheap-to-decrypt symmetric encryption with a key known to all your servers, which still avoids the need to store the data in a server-side session store.

First, in case anyone has just arrived from Mars, or even more unlikely, isn’t familiar with Facebook, the auto-growing textarea is a text box that gets bigger as you type into it, so that you never see a scroll bar (unless you’re typing war and peace).

There are lots of ways of doing each part of this. The parts I’ll look at are (a) what triggers a review of the text box’s size, (b) how to determine whether a resize is required, and (c) how to perform a resize.

Triggering a review

There are four ways this is typically done:

1. On an interval, say every 50ms
2. On keypress
3. On change
4. On keyup

I’m staggered by how many scripts insist on doing this kind of thing on an interval. Setting up interval timers for no good reason is a shortcut to terrible performance and memory leaks galore. So we won’t be doing that. onkeypress and onchange events are triggered before the box is updated with the latest keypress, so we want to avoid those, as the latest keypress might have been the one to bring us onto a new line. That leaves onkeyup, which is fired after the box is updated with the new character, and allows us to inspect it and decide whether to increase its size.

To resize or not to resize

How to determine whether to resize the box comes in three flavours:

1. Count the number of newline characters in the textarea’s value, and see if that matches the number of rows the textarea has.
2. Create a ’shadow’ DIV which is off screen (eg. margin-left: -10000px) and has no height declared, but otherwise has the same style properties as the textarea, fill it with the textarea’s content, measure the subsequent height of the DIV, then see if that height matches the current height of the textarea.
3. Check whether scrollHeight (the height of the scollable content of the box) > clientHeight (the height of the box itself).

In this case I was suprised at the number of implementations that favoured counting newlines. This only works if combined with counting the number of characters in the textarea’s value, AND a monospaced font, AND knowing the number of columns in the textarea. In short, er, it’s mad.

Second option, and the one favoured by a lot of the framework plugins due to the ease with which you can create shadow elements in the likes of jQuery, is to create a shadow DIV. This has the advantage of telling you the actual pixel height of the text, even where it is LESS than the height of the textarea box. Otherwise, you’re limited to measuring clientHeight and scrollHeight, which are exactly the same if the textarea isn’t scrolling, regardless of how much space you’ve got to spare at the bottom. My issue with this method is that it basically requires use of a framework to not be painful, and even then it’s non-trivial amounts of code, and adds needless pollution to the DOM.

So that leaves relying on scrollHeight and clientHeight. These are well supported and efficient to read, so provided that you work around the issue of scrollHeight always being at least equal to clientHeight, this offers a very lightweight solution. The features that you can achieve with a shadow DIV that you can’t do by simply reading scrollHeight and clientHeight are (a) you can ensure there is always a blank line at the bottom of your textarea, and (b) you can shrink the textarea if the user deletes text from it. I’m personally of the view that neither of these is actually particularly desirable. There’s potentially an argument for leaving a blank line at the bottom, but equally the uer might feel like they’re just being pressured to write more.

How to resize the box

OK, so if you’ve concluded that the user is writing chapter and verse and the textarea is in need of a bit more space, how do we go about doing it?

1. Animate the CSS height property
2. Add some height via the CSS height property
3. Add 1 to the rows attribute

A fair few of the framework plugins use their framework’s animation capabilities to animate the grow effect. I don’t like this. Just because you have an animation effect available doesn’t mean it’s appropriate to use it, and there are many situations where the end user just doesn’t want to wait 300ms for the privilege of using a fresh line.

Simply adding height to the CSS property is a fair way of doing it, but unless you do some maths on the user’s line height (or hard code some magic numbers), you can’t necessarily guarantee that the resulting height will be a multiple of the textarea’s line height.

Easiest, most efficient solution: add one to the rows attribute of the textarea. rows is part of XHTML as well as HTML 4.01, and has universal support going back yonks.

Pasting

Watch out for pastes. If the user pastes in a large quantity of text, they will trigger only one keyup event, but will have added many lines to the textarea. Make sure that if a resize is required, you trigger another review after the resize is complete.

Max height: the war and peace scenario

If the user really does seem to be writing a novel, we probably should call it a day on growing the textarea at some stage, and certainly before it gets to be taller than the viewport. You can check the height against the viewport height, though I typically just restrict it to an arbitrary height, say 30 rows.

The code

You’ll need a textarea element with an ID of mytextarea to make this code sample work, and obviously you can easily modify it to use selectors from your favourite framework rather than the native document.getElementById.

document.getElementById('mytextarea').onkeyup = function() {
	var ta = document.getElementById('mytextarea');
	var maxrows = 30;
	var lh = ta.clientHeight / ta.rows;
	while (ta.scrollHeight > ta.clientHeight && !window.opera && ta.rows < maxrows) {
		ta.style.overflow = 'hidden';
		ta.rows += 1;
	}
	if (ta.scrollHeight > ta.clientHeight) ta.style.overflow = 'auto';
}