ForumsNewsAnd we're back! (from a very prolonged outage)
And we're back! (from a very prolonged outage)
| Author | Message |
|---|---|
|
peter |
Fantastic work you guys, you will be all the better for having had to sort out a real bad crash! For me there was actually an upside to this - reading the forum for the first time I now find out there is an Toodledo app for the iPhone!
|
|
Anthony |
Thank you for all your hard work.
Don't beat yourself up too much about this as these things happen. |
|
Bonolono |
No worries... great job, congrats!
As someone else already posted here, I very much prefer a longer outage with full recovery from a quick restore with data missing. Happy regards from Switzerland |
|
fishforpeople |
Fantastic effort, I'm sure all the ToodleDo-ers's appreciate your honesty through this experience.
A very difficult situation handled extremely well. |
|
MFA |
We've all had a million computer crashes in our lives... the great thing about this is that it has happened once in using TD continually for 18 months, and when it did happen it was you (sorry) running around with a bunch of people knowing what they were doing.
Under the old system, it happened every month and it was me, dodgy Microsoft software and no support to call on... You guys do a great job and we all really appreciate it. Well done for getting it back so fast. |
|
Peter Carter |
Like many others. I've just signed up for the Pro subscription because of the thoroughly professional way you handled this incident.
We all make mistakes, but your candid discussion of the issues gives me confidence that this wouldn't happen gain. Keep up the excellent work. |
|
emp |
Great work!
I also work in IT and I know how you can easily get blindsided by something like this. A mechanical failure during the switch to the ISV! ARGH! No one can think of all the possibilities. What makes you stand out above the rest is the open and honest communication during and after this event. Very professional, great work! |
|
Paul |
Well handled, I as a businessman have taken notice of how well you crafted your response to this.
|
|
cschlenk |
We had a public holiday yesterday in Germany. So instead of scrolling through my todos I wen't swimming with the kids ;)
|
|
Matthieu |
Thank you for all your work.
One question : the way you communicate during this crash by using "I" and not "We" seems to indicate that toodledo is a one man company. I'm sure there is plenty of guys around you, but it seems that toodledo is "your baby". I would be happy to read somewhere on the site your story, and a presentation of the team. This episode brought us closer, and we now want to know more about you :+) |
|
Stephen |
Hi Jake
I think that you have communicated well and very openly, thanks for that. I think the up-front offer of free time to be a very customer-centric approach - thanks again. BUT It would make a non-techie, like me, very happy to have confirmed that you have multiple back up, i.e. in a different geographic location and that there is an operational knowledge / ability backup just in case (god forbid) you had a "run over by a bus tomorrow" type of problem. I do like the product and am very likely (99% certain) to be sticking with you. It did just make me consider how much knowledge and trust I place in your product. Thanks |
|
Dave |
Well done!
Your professional and honest reporting is both refreshing and appreciated. Try to relax this coming weekend! |
|
andy |
You've not lost this user.
In fact I'm glad you've gone through this test (and come out successful at the other end I must add) because you've discovered a 1-second improvement that you didn't know before. As the others have already said - go and rest. We're all happy to know our data is being looked after :-) |
|
Guy Iaccarino |
Posted by drylemming:
Thanks for scrambling. Upside for you from the outage is taht people like me may have to buy the iPhone app now if it will let us have some offline access even if you go down for a bit. Thanks Buy the iPhone app, it's totally worth it. And "Here! Here!" to Toodledo for dealing with this issue in a forthright and honest manner! |
|
Evanbalzer |
Hey, this stuff happens. Thanks for your honesty. Like others, you don;t realize what you have til it's gone. So glad to have it back. Get a well deserved rest this weekend.
Thanks for your dedication to what has become an esential service. |
|
ptrav37 |
Thanks for your hard work. I really appreciate the way you managed the situation.
-Trav |
|
Nickolay |
Thanks for the way you handled this!
It's interesting to know that I'm not alone in purchasing the Pro account not because of the features, but as a way of showing support for your openness. Much appreciated, upgraded to Pro Plus now! |
|
Paul |
I have the iPhone app and never even noticed the outage. Heard about it on a blog.
Very well handled. Let me add my thanks to you as well. |
|
OMARY.GONELL |
Thank you for all the clarifications in detail. I'm an EA at JetBlue's IT department and I know first hand how these data centers go down. Back in Sep 08 - JetBlue's Garden City Data Center went out completly due to Squirrels over Labor Day Wknd(go figure that all those poor IT folks had to cancel all vacation plans and work thru the weekend to restore service just so customers could still fly that heavily traveled weekend). Finally by Monday got it all up and then TUESDAY...disaster strikes again with our Salt Lake City Data center down for bad weather....another three days of panic.
We have since changed our strategy and now have an outside vendor taking care of our data center in order for this to not happen again. I must note that I didn't realize how heavily I depended on my list until it went down. I will still be loyal and thank you again for being straight with us. |
|
Jake Toodledo Founder |
Thanks again, everyone, for all the nice remarks.
I just wanted to give you guys an update on what steps we are taking to prevent this from happening again. 1) We have reconfigured our existing master and backup databases in such a way that a power outage is much less likely to cause this type of problem again. 2) We are now running the tape-recorder system (that I described in my first post) on both the master database and the backup database. This will protect us in the event that the master database has a hard drive failure at the same time that the backup database gets corrupted. It would take several simultaneous failures for this to happen, but you can't be safe enough right? 3) Our worst case scenario has always been losing 24 hours of data, since we run nightly offsite backups. We are now running these offsite backups every 12 hours. 4) We are exploring options to get a battery backup and we are thinking about different database configurations that might be more robust and rebuild faster in the event of another similar issue. |
You cannot reply yet
U Back to topic home
R Post a reply
To participate in these forums, you must be signed in.