Join our accounting partner program to win new business and add value to your existing clients.
All The Books Are Back Up
Sorry
To all our faithful users, we are truly sorry for any inconvenience caused on Tuesday 31st August 2010. Believe me, there is no one more frustrated than the team at Clear Books and Fubra right now. Our reputation and our business is dependent on providing a fantastic service and we fell short of the mark on Tuesday.
All The Books Are Down
On my return from a relaxing bank holiday weekend, the one brief smile I allowed myself was when someone tweeted:
@spyou unfortunately @clearbooks is down too. so I can’t invoice the people I know not on #facebook. All the #books are down.
Not only was Clear Books down, but so was FaceBook. No matter the size and clout, all web services are susceptible to unexpected downtime.
Admittedly our downtime was longer and where we monumentally went wrong was not having a satisfactory recovery procedure. We thought we did, but we didn’t. It took too long to implement and due to a number of complications, we had to recover data from Saturday.
It was unfortunate that the server failed, but we should expect servers to fail and be prepared accordingly.
Judge Us On How We respond
Please judge us on how we respond to this incident and the steps we take to significantly improve the recovery procedure. This is what we will be doing:
- To regain your confidence we will start documenting regular recovery drills and announcing the results.
- To be completely transparent about our recovery procedure we will set up a thread on Get Satisfaction to outline and discuss our intended setup.
Thank you for your continued support and once again, sorry!
Tim Fouracre
—- Update 2 September 2010 —-
Action Plan
Whereas previously we were using a dual redundancy system (RAID + off site backups on Amazon) we will be putting into place a 5 level redundancy system:
1) RAID on all servers
2) Master / Slave replication – slave will replicate in realtime from master
3) Cold spare database server – will provide a full set of spare parts in the event of a failure of master / slave
4) On site daily backups (dedicated box in our main DC for faster restore in the event of problem with 1 & 2)
5) Off site daily backups (on Amazon EC2 / S3)
For transparency and to show evidence that our new recovery procedures are in place and working we will publish reports of recovery drills in which we replicate a server failure. Additionally, for your peace of mind, we will improve the tools that let you get hold of your own data:
1) We will implement a tool that sends you your backed up data on a regular basis. There will also be the ability to resurrect your own backups so that you can use these as restore points.
2) We will improve our CSV export tools so that you can export the entire Clear Books system to CSV and then import these back into Clear Books.
About the Author - Tim Fouracre
I've always been interested in computers and finance. I started out as a PHP developer and later qualified as a Chartered Accountant at KPMG.
Programming and accounting came together when I co-founded Clear Books online accounting software back in July 2008.
Showing 18 Comments
-
Good job I didn’t try and do the books! Some of us didn’t even notice!
These things happen but what is good about you guys is that you are honest and you are puting things in place should it happen again.
Adam
2 years ago
-
Sheesh, whilst Adam, above, might be happy … I’m not! As we were month end this weekend myself and my wife spent 3 hours on Sunday doing my August book-keeping … I had a couple of queries with my accountant who called just now to inform me of the message you guys had posted. We are now faced with trying to find another 3 hours from somewhere to re-enter this information this month.
I have to ask, although I suspect what the answer is likely to be, is there no way we can recover this data?
I’m also a bit pipped off because one of the reasons I chose Clear books was the premise that I did not need to back up my accounting. As I work from home having off-site back-ups is not feasible!
I’m glad that you guys have put your hands up about this error, but it still has knocked my confidence in your product, I trust it’s not going to happen again …
Allister
2 years ago
-
Unfortunately, some of us did do our books over the weekend and have lost LOADS of data! Maybe we should be able to do backup our own data, so in future this never happens again and some of us do not lose hours of work!
Not at all happy with this outcome!
Kim (YS Tunisia)
2 years ago
-
Adam, Allister and Pete, thank you for your comments.
Allister, you may be interested to read more here but unfortunately the answer to your first question is no.
In terms of happening again, absolutely not. Hopefully our response will persuade you and anyone else of that as we formulate, role out and announce changes to our processes.
2 years ago
-
Google Analytics was down for 16 hrs last week – as you say, it can happen to anybody. Sympathies to Allister but thanks to Tim for the note.
2 years ago
-
Hear, hear. The right approach is total honesty about these things – owning up to mistakes and then putting them right – so they never, ever happen again.
Well done.
Pete2 years ago
-
Tim
Thank-you for taking the time to respond to our comments. I do appreciate your company’s openness and indeed frankness in the face of your adversity … after calming down a bit after this morning’s spat I’m just glad that it’s only two days and not the entire lot down the drain!
Good luck with the rebuilding of everything at your end!
Allister
2 years ago
-
Dear Tim,
Like the others I sympathise with you and well done for coming clean straight away. I have a client who has some corrupted data (I know he has emailed support) so I will keep you informed about this one.
I understand Kim’s (and others’) frustration at loosing data, and it is a fair point that the moment you removed the ability for the individual to make backups of their data that is the moment when you need to be able to guarantee that the data will never be lost or corrupted. It is a huge responsibility and you might like to try and implement a way of doing a database dump so that individuals can take a back up for their own peace of mind.
Good luck in the rewriting of your disaster recovery programme!
Chris Caspell
http://www.oxfordtaxsolutions.com2 years ago
-
It’s beginners luck for me really that the first time I set aside a day to enter my first round of outgoings your server crashes the next day.
It is highly frustrating to loose this data, but fortunately since you provide an excellent product which is continually being tuned I have the faith that you will respond to this and it will not happen again.
I also appreciate your openness, it is very refreshing. Good to see the site back online today.
Matt
2 years ago
-
This is completely out of order.
Having only just come off my trial period I spent a good number of hours into the early hours entering masses of data and now you tell me you have lost it.
I have not read all the above comments (as I dont have the time now!) but have you never heard of daily back ups !
2 years ago
-
Hi Tim,
Every time I have entered a piece of data since I started with clearbooks I have winced a little; “How do I know these guys do have proper back up and DR?”
While I sympathise with your position it’s very hard to build trust in the first place and then restore it after an incident. My life would be a night mare if there was a serious loss of data.
Is there any possibility of organising an external audit of your new backups and procedures? This would reassure new clients as well who may think twice about trusting all there data to a website in the first place.
Also an “export everything” option would be helpful even if it meant paying a small bandwidth based surcharge. Or perhaps lodging a customers data in an accessible way with another service provider. This would only require incremental backups and could be billed as an extra service for the paranoid.
David
2 years ago
-
“Google Analytics was down for 16 hrs last week – as you say, it can happen to anybody. Sympathies to Allister but thanks to Tim for the note.”
If this was the case, and Google was down for 16 hours that’s obviously a blot on their copy book, but, as they’re a free service I’m not sure too many people can have a problem.
Clearbooks on the other hand is not a free service, it is a for-cost service that should come with for-cost accountability.
Andrew
2 years ago
-
I can appreciate that for those of you that did enter data this is a right pain in the proverbial. You have to find time to enter it again. None of us have free time but you will find the time and you have all the information so it can be re input.
Clearbooks buggered up. However painful for people it happens. Apple have been known to wipe peoples emails in error and not recover them. Often it’s the case that a business doesn’t realise something won’t work until they need it to. But I believe that Clearbooks will put something in place so it doesn’t happen again.
At least they have held their hands up. That is rare these days. And ultimately they will pay a price for this
2 years ago
-
I too spent the weekend completing our accounts for my VAT return. Having closed off invoices and entered purchases and invoiced clients I now lost all data that did take me pretty much all of Sunday and Monday to input – wasn’t happy to say the least yesterday…
I now have to go through my paper accounts and input all again – and I hate receipts.
The one saving grace with Clearbooks is that I have spoken with Tim on many occasions about different things the site does and does not do, and he’s always taken it on board. If anything they’ve worked faster than I thought they could on some things and really helped me out. For example being able to provide abbreviated accounts to supply to companies house – Tim added this service efficiently and effectively just at the time i needed it done by.
The procedures that Clearbooks did have in place for their DR did include backups, and John Whyte – you should read what Tim has said before suggesting they don’t do backups at all. It fell down not because of the backups they do but what procedures they have in place to make the backup worth something. Do you on-site and off-site back up John?
Now that the procedure is clear (which is all I asked Tim for) – and that we can now get downloads of the data ourselves that too would make sure that in this situation if to ever happen again, we will all be able to get our accounts back online and untouched.
Venting our anger isn’t going to get our accounts back – so I best shut up and get on… How I love working at the weekends…
Here’s to a transparent future and hopefully an uninterrupted service from Clearbooks.
2 years ago
-
Wow! I was away over the weekend and luckily escaped. It must have damaged confidence in some way; but on the other hand I really am impressed by the open and honest response from ClearBooks. I’m sure they realize that most people will forgive them on this occasion but also realize that if it happened again… With that in mind I’m more than happy to place my trust in them.
As a matter of interest I spent about 6 hours yesterday trying to save and unscramble my G-mail contacts, which went haywire over the weekend.
2 years ago
-
Hi Tim,
Having read your action plan I am satisfied it is a good remedy and will have confidence in you guys going forward.
David
2 years ago
-
Hi Tim,
Very well done for putting together this disaster recovery package so quickly. I was FD for a data recovery company in the late 1990s and we recovered data for some of the biggest names so it goes to show that it can happen to anyone.
The important thing is the lesson learned, which, from what you have said, you have heard the message loud and clear…which, to be honest, is why I recommend Clear Books to my clients – in short, you listen and react quickly.
Great job!
Chris
2 years ago
-
Thanks everyone.
A quick update on the action plan. We are hoping to implement the master/slave replicated servers this Monday. Please see here: http://www.clearbooks.co.uk/blog/2010/09/10/scheduled-maintenance-monday-13-september/
2 years ago
Leave a comment