This week, the Japanese stock market suffered a serious blow when a trader accidentally sold 610,000 shares in J-Com, a recruitment company, for 1 yen (half a penny) each. The scale of this error becomes apparent when you realise that the broker concerned had intended to sell only one share, for 610,000 yen (£2,893).

Oops doesn’t really cover situations like that, does it. The brokerage responsible for the sale is having to buy back all the mis-sold shares at their market value (since no-one else is prepared to sell them at 1 yen), and suffer a resulting loss of around 27 billion yen (£126 million). All because someone put two numbers in the wrong boxes on a computer screen.

This could be considered a human error, and it’s worth remembering that even if there had been no computer involved at all, the broker might still have written the numbers down in the wrong boxes on an equivalent paper form. However, while a piece of paper is incapable of telling you when you’re writing what amounts to incomprehensible gibberish, a computer is certainly capable of taking a look at your request and checking that it falls within the boundaries of what should be considered rational. So really this is a software failure, in that it is certainly a failure of software to pick up on human mistakes.

The problem is, while computer software is now allowing us to process data and particularly financial transactions at a truly amazing speed, this also means that when technical errors occur, they tend to be large and far-reaching. As systems get ever more efficient and more accurate, we can process even more transactions per second; run trains faster and fly aircraft closer together, keep our cars on the road and even put people in space. We are putting more and more of our personal and financial safety in the hands of computers.

If that makes you shudder with fright and long for the days of manila folders stored in endless file rooms, it shouldn’t. When you’re in a lift on your way to the 50th floor of a skyscraper and the cable snaps, what do you want to rely on to brake your fall? A lift operator manually engaging an emergency brake, or a computer that can recognise the fault and react before you have even had the chance to regret having the chicken tikka for lunch.

Reliance on computer systems can’t be avoided. They’re just far too useful to be sidelined. What must be done instead is to make them absolutely bulletproof, and that’s not cheap.

The need for overwhelming testing and safety has been a staple of the nuclear industry for decades. Instead of being the ultimate source of unlimited clean, cheap power as originally promised, nuclear has become a source of energy no more cost effective than existing techniques. The industry recognised and accepted that the cost benefits of nuclear generation methods were almost entirely swallowed up by the cost of compliance with safety regulations.

Unfortunately many large computer systems are still constructed by the lowest bidder, and resilience is often one of the first things to suffer compromises. Considering the human and financial cost of computer failures should make it easy for firms to see the value in spending money in advance to prevent them. I would argue that any amount up to and including the cost of the worst conceivable failure is a valid investment in ensuring the resilience of a system. Certainly with hindsight Mizuho Securities in Japan would rather have paid the 27 billion yen earlier to develop a way of filtering incomprehensible trading orders, than pay the same amount now to buy back the mis-sold shares and suffer the glare of the media spotlight as well.

When you are considering a new system, calculate the cost of the worst possible failure of the software. Then consider what proportion of that you are willing to invest to prevent the failure from occurring. The system may end up being not as profitable as you’d like, but on the other hand, it may be considerably more profitable, depending on whether the disaster has happened yet.