The Problem with Scheduled Downtime
One of the things which surprises me about IT systems, whether SaaS or not, is the amount of time that upgrades take as systems grow, their often inconvenient timing, and the impact the resulting downtime can have.
Whilst not as damaging as the unscheduled variety, scheduled downtime still has an impact: with international travel it’s not uncommon for your colleagues in distant timezones to need to access and update your data, at what are otherwise unsocial hours. Or, if your system is based in another continent an “out of hours” upgrade can happen at the most inconvenient time (for our UK users, the good news is that we are UK-based so the timing of our scheduled outages is less likely to be a problem). But, regardless of where our users are, we want to reduce the duration and minimise the impact of our scheduled outages.
So we do all we can to reduce scheduled downtime. We have always kept the frequency of system updates to a minimum and considered their timing carefully – for example by adopting a quarterly cycle for major releases – but now we’re taking advantage of Workbooks’ architecture to do things a little differently.
One reason why upgrades are time-consuming is because they involve changes to how data is stored and organised, and the more data that you have to modify at the same time the slower things will get. At Workbooks we’ve taken a different approach from our competitors as to how we store customer data: for security and scalability that data is completely segmented into separate databases for each customer. Amongst other advantages this means we can process upgrades one customer at a time and prioritise the processing of updates for customers who attempt to access the system in the minutes just after the upgrade has been initiated. Because many customers don’t actually access their data at night over the weekend this means that we can reduce scheduled downtime for those that do. We can also run more of the upgrade in parallel further reducing its duration.
I’m pleased that we’ll be using this new scheme for all future upgrades, starting with our next major release.
Your data is safe with us
One of the questions we’re often asked is “how can my data be secure if it’s on the Internet” or some variant of it. Before I give the Workbooks answer (the short version is “at Workbooks we take data security very seriously indeed”), I’d like to run through a little corporate history.
A decade ago (our company then was known as Activis) security became our number one priority – we were responsible for managing and maintaining over four hundred firewalls for a hundred corporate clients from three locations (UK, Germany and the USA). As you can imagine protecting the keys to the firewalls at the heart of our clients’ information security was something we had to do in ways which stood up to scrutiny from our clients – often these were Information Security Officers and the like. Multiple layers of security – “the onion model” – and completely redundant systems including redundant operating centers were central to the solution.
John mentioned BlackSpider in a previous article. BlackSpider was our next venture and it specialised in email security – at its core it was concerned with the detection and blocking of e-mail threats including early-stage viruses which the traditional anti-virus industry was unable to combat effectively. From the beginning we built data security into the core of our systems and we extended Information Security – through the implementation of something called an Information Security Management System – throughout the whole company. So we gained certification (to the “gold standard”: ISO 27001) not only of our SaaS systems but also of Information Security in our HR processes and our CRM processes.
Information Security – as any expert should tell you – isn’t just about keeping your data secret: it’s also about ensuring it is available when you need it, and that it can be believed. In the industry jargon you ensure information’s “Confidentiality, Availability and Integrity”.
With the the advent of Workbooks, information security remains uppermost in our minds. We began with our “clean sheet of paper”; one early decision was to store customer data in completely separate databases – this helps with the Confidentiality bit – for example you no longer need to worry about accidentally picking up some other customer’s data when generating a report; it also helps keep performance consistent because within customer-specific databases you have much smaller indexes (but that’s probably for a future blog article).
The “onion model” at Workbooks looks a little like this:
- Physical security of our servers is achieved through their being located in inconspicuous buildings with 24×7 manned security, CCTV-monitored data centres with biometric systems and certified entry procedures.
- Availability is achieved through using buildings with redundant power and airconditioning systems and through the use of two physically-separate locations with a high-speed network connecting them. Our policy is to implement systems with no single points of failure. All hardware has remote-management capability.
- Network security is achieved through the application of multiple layers of protection, including packet filters/ACLs, firewalls, and other techniques which are confidential. External specialist organisations are used to vulnerability scan at the network level and do more involved penetration testing. All data transfer happens under strong encryption; all access to Workbooks secure website uses 256-bit SSL together with an Extended-Validation certificate.
- Careful design. All systems are built on the principle of ‘least privilege’ such that processes run with the minimum set of capabilities and software is not present on the operational systems unless it is specifically required. The operating system is under tight version control and we monitor for reports of security vulnerabilities in the OS and its components.
- Our Development and QA processes are geared towards a controlled release cycle with a focus on avoiding security vulnerabilities and data corruption. The processes are extensive and include both automated and manual testing at many levels: unit, integration, system and functional. System changes are only permitted under a full Change Control process with signoff by senior Workbooks management.
- Data is accessed and copied only over strongly-encrypted connections. We implement separate databases for each customer to add an additional layer of security above an extensive Permissions/Capabilities model which allows functions to be limited to specific roles or groups of users. Underpinning all data storage is a row-level security model which allows users to hold private data securely and permits control to read, modify, write, change access or change ownership on a record-by-records basis.
- As described above, each customer’s data is held separately which enables us to offer customers the ability to receive a file export of all of their data; this could be used to recover customer data in the complete absence of the Workbooks service.
- Workbooks’ development and customer support staff do not have access to live customer data without the customer’s prior permission which the customer grants through the use of the Workbooks service itself.
Finally, our customers are encouraged to help themselves. By default we set password security options for all our customers to enforce secure passwords. There is no substitute for keeping those passwords secure and hard to guess!
This is a public blog article so I’ve had to be a little circumspect with some of the details but hopefully I’ve answered the question; if you want to know more we’re always happy to discuss things in a little more detail privately.
The buck stops here
As with IT generally, there are a lot of facets to “cloud computing”. Some companies deliver cloud services for end-users (facebook or Microsoft’s hotmail for example). Some, like Workbooks, deliver services for other companies. And some companies deliver services to support other companies’ delivery of cloud services.
Amazon Web Services is perhaps the leading provider to such services – it is a division of Amazon which many organisations use to power their IT services by providing them with computer and storage capacity “on demand”. If you take Amazon’s Web Services’ main offering – EC2 – then you are renting “virtual machine instances”. These are systems on which you can run your own operating system and they share a portion of the underlying hardware’s resources. You are of course sharing hardware with other EC2 customers and it’s down to chance whether your fellow virtual machine users are resource hogs or not. So there are now cloud services which help you manage your Amazon Web Services and it’s perhaps unsurprising that some of them report variations in EC2’s performance.
This last category poses an interesting question if you believe in the power of the cloud computing model: if you are going to deliver cloud services yourself, should you rely on other cloud services to deliver some of your underlying components? Certainly a model where Workbooks was built upon cloud services from other providers would have its advantages: we’d only pay for exactly what we used, and it would be very easy and very fast to add capacity: simply order some more and turn it on, on demand.
When things go according to plan, everyone’s happy. But if things become unreliable it’s Workbooks which our customers would see failing – and those customers would be right to expect Workbooks to stand behind the service and remedy it. Excuses such as “it’s our supplier’s fault” just wouldn’t cut it. When you have very large third-parties delivering components of the service it’s unlikely that Workbooks’ top priority would be their top priority.
Basically, we think that the cloud model is something which works – brilliantly – where there’s a simple customer/supplier relationship but that it can break down when there are hierarchies of services unless you think very carefully about how you will deal with the contingencies. It’s a little different from traditional business relationships where you have the luxury of at least a little time to sort out most issues: we need our infrastructure to be always available and reliable. We don’t want to be involved in trying to diagnose a third-party infrastructure (like EC2) and having the responsibility to sort out issues within it without having the ability to do so.
So we took a different route. We built our own infrastructure and we are responsible for its management – right down to the hardware. Although we do use third parties for some of the components, there is always redundancy: multiple networks, multiple locations. If a provider fails to deliver a service we can call on an alternative so we can be certain we can deliver the service levels we commit to in our SLA. Maybe we will use some Amazon services in the future but if we do they’ll be non-core and we’ll be sure to have a backup plan.
The reinvigorated web browser
At Workbooks we are very interested in the evolution of the web browser – it has only recently become possible to deliver a compelling rich User Interface like the Workbooks Desktop without requiring all manner of plugins. The good news is that the evolution is accelerating and with it the browser’s speed and capability.
Eighteen months ago Google launched Chrome which was dramatically faster than any other browser available at the time. A browser performance and functionality war has been underway since then. In those eighteen months, and despite some hiccups, Chrome has become the world’s third most popular browser – after Microsoft’s Internet Explorer and Mozilla’s Firefox and ahead of Apple’s Safari and Opera’s eponymous browser. It’s well documented that Internet Explorer’s market share has been declining at an increasing pace since its peak in 2004 – that decline accelerating further since Google’s entrance into the market and likely further still with the advent of the European Union’s “browser ballot” screen. Google have also announced Google Chrome Frame, a plugin to Internet Explorer, which brings Chrome’s features and performance to Microsoft’s browsers for websites that are modified to use it. Finally there is now a proliferation of new ways of accessing the Internet using things other than PCs where there is nothing Microsoft in sight: from PS3s, iPhones and iPads for example.
Internet Explorer was first released nearly fifteen years ago; the other week Microsoft unveiled a “Platform Preview” of the next version, dubbed Internet Explorer 9. It’s not even “alpha” software – we have no idea when it will be ready – and yet it has excited a lot of commentary. My view is that it signals a radical departure for Microsoft and they should be applauded for it.
As is Microsoft’s way they are still a little picky about which open standards they embrace but IE9 includes much that other browsers are also adopting so eventually if you choose a modern browser from any mainstream vendor it will be capable of rendering video, storing data to work with while offline, and showing beautiful charts which scale smoothly as you zoom your browser – all without requiring any external software (“plugins”). As an added benefit it will get easier for developers to write web sites which work consistently across all major browsers without major compromise through a “lowest common denominator” approach: the lowest common denominator is now getting a lot higher.
IE9 is notable also for the speed it promises. It has become very fashionable – and easy – to criticise the performance of Microsoft’s browsers. It’s refreshing that with IE9 Microsoft has ditched its denial of there being a performance issue with IE and decided to address the issue: now it’s going to be “crazy fast“.
Depending on the benchmark IE9 shows upwards of a sixfold improvement over IE8 which itself is twice as fast as IE7. Although it’s unlikely IE9 will take the speed crown for most tasks from current leaders Opera and Chrome or overtake Safari it does at least mean we will be able to stop focusing quite so much on IE performance as an issue.
Prior to IE9’s announcement it was looking increasingly likely that Microsoft’s market share decline would only increase however now it’s clear that Microsoft have decided not to abdicate from the browser market; Chrome Frame will likely be marginalised to be a tool for those IE users who are unable or unwilling to migrate to the latest version of IE.
But more significant than any of the above is that I think IE9 is going to be fundamentally different from previous versions of IE: it delivers most of what it does using open standards. It’s well documented that many organisations are still stuck on IE6 – even though IE6 has had its funeral – because that browser included a raft of proprietary extensions that did not make it into IE7 or any subsequent browser. Open standards are easier for developers and allow users to avoid lockin.
This is all great news for the user: whichever browser you use it is getting faster, more stable and more functional.
Made in the UK
One of the decisions we took first at Workbooks was to build our own software, and to do so here in the UK.
We’d looked around at a number of open source software packages on which we could build a service, some with licences which we could live with but none met enough of our requirements: it had to support multiple customers efficiently, support a powerful security model, be easy to use and allow us to deliver a service that wasn’t just another CRM solution. Workbooks was to deliver a complete ‘Prospect to Cash’ solution and it became obvious the only route to this was to create our own.
Having decided that we also got to choose to deliver a few more goodies, such as a completely flexible system to generate reports, and a Windows-like user interface (implemented using a Javascript framework running in all widely-deployed modern web browsers without plugins or compromise). We’re particularly proud of the user interface: comparing a traditional Web UI with the Workbooks Desktop is a bit like comparing DOS with Windows (if you’re old enough, can you remember when you could only see one form at a time?).
The next decision was that we’d build it here in the UK. This is deeply unfashionable (schoolchildren aren’t even offered the option to learn basic programming skills in most UK schools) and there is a widespread view that pretty soon all software development will be offshore. But we wanted the flexibility to react quickly as our understanding of the requirements solidified and to build a capability to respond quickly to customer requirements in the future. Plus, it gave us the opportunity to work with some very skilled developers with whom we’d built great things in the past.
Thankfully the UK government’s R&D tax credit scheme (just about to reach its tenth birthday) made things a little less daunting than otherwise. And although it still costs us more than our offshored competitors to produce software we believe it is worth it for the flexibility and responsiveness it delivers to us.
Another unfashionable thing we did was to commit to a UK-based customer support function: one of the things we have set out to do is to differentiate ourselves from our competitors through the quality of our support team.
So we’re proud to report that Workbooks was born in the UK, continues to be built in the UK and is supported by our completely UK-based team.
Traditional software companies won’t fly in the clouds
Microsoft’s recent launch of its Azure platform made me consider again whether they are likely to succeed in moving their whole business to a new model. Something that has interested me for quite a while is how hard it seems to be for traditional software companies to succeed with their ‘Cloud Computing’ efforts. There are at least two problems.
The first problem is that simply building the technology and getting the whole company (and their partners!) committed to the model is always going to take a while: educating technical staff in new architectures that can scale massively with all the implications that has, delivering applications on a new platform which was foreign to those staff, building systems for which over-the-web delivery isn’t just a bolt-on – it’s the whole point. Significantly being a player in the new world seems to require a conversion to open principles – whether these are the adoption of open standards for data transfer or open architectures. Compare, for example, Google’s approach to that of Microsoft: Google even have a team endearingly called “the data liberation front” whose role it is to make it easy to move data away from their software.
I think a second problem is more significant: that a move to the cloud brings with it a change of business model – from licensing software for the duration of that software’s useful life to a different model based on paying for what is used. Those companies in the ‘traditional’ camp have large direct and indirect sales forces which are comfortable (and paid to succeed in) selling software on a single, up-front payment. Contrast that with cloud computing where the model is that you pay for what you use: this plays havoc with those sales teams’ traditional pay structures. Incenting those sales staff to move to a new model without cannibalising their existing installed base and deferring valuable cashflow may be just too hard: effectively these companies have to build a whole new business in parallel with their existing one and manage the inevitable conflict between the two businesses and channels.
It will be interesting to see if any of Microsoft, Oracle or IBM ever get to the point where a majority of their revenue is from software delivered as a service. Of the three I suspect IBM has the best chance given their history in renting Mainframe operating systems and software. It seems more likely to me that other players, led by Google but including a host of new entrants which are unencumbered by adherence to a traditional software model, will dominate the cloud computing space.
Workbooks was founded on a conviction that applications are best delivered from the cloud so that for our customers there is “no hardware, no software, no hassle”. Who will our competitors be in five years time?