The death of Distribute IT, backups and back online
As Posted on the ABC Tech Site: June 22, 2011
They say a day is a long time on the internet. Watch a meme unfold and you’ll see what I mean. Image taking your blog, website and business offline for 9 days. Not just 9 days where you know what is happening, but 9 days where there is scant information and most of the information that comes isn’t from the source.
I am talking about what happened to not just my domain, but 4800 others that have gone and a huge number that disappeared for quite some time. I was until recently a very happy customer of Distribute IT. Up until the start of this year anyway. Sure there had been the odd outage here and there, but nothing too long and too extreme. In the last few months things had gone seemingly downhill. A visit to Whirlpool will show the increasing dissatisfaction that people had been experiencing.
Yet all of this was nothing compared to what happened at 5:50pm on Saturday June 11. Whilst the clients of Distribute knew something was up within seconds of this outage quickly updating Whirlpools thread from the previous weekends outage, it wasn’t until Sunday, 12 June 2011 2:35:12 AM that Distribute were to say anything.
“Engineers are at DC working on restoring services asap. Very deliberate, coordinated & malicious attack on DIT network the cause. No ETA yet “ Via their Twitter account.
It felt early on that Distribute had taken the “Sony” approach to corporate communications telling as little as possible and certainly not highlighting what possibly could have gone wrong.
So I lost my email, my little personal blog and really not a great deal. On Whirlpool I would watch as people told stories of massive advertising campaigns locked in for the long weekend which were vaporised by this outage. I watched as people talked about the fact all Distributes phones were IP based and where taken out as well. I swore and cursed that I’d lost my personal email, grateful I wasn’t running a business off my website.
After a number of emails and 9 days my worst fears were confirmed, it was all gone. Everything that I had on that website was lost. Every email from what will be nearly 2 weeks gone into the ether.
Sure, everyone says, have a backup. Guess what, I do. But a backup wouldn’t have helped those businesses that had advertising tied to their website over the long weekend. A backup wouldn’t ensure that you could find a new host over a long weekend in Australia, and a backup wouldn’t ensure that your settings were in place.
Finally now that Distribute have finally said that the data is gone and that mine and 4800 other websites are gone, I am putting into play my backup plans.
Still I wait 2 days or so for my domain to move to a new host, upload all the FTP content, setup a new version of WordPress, upload my old blog XML document, put in a couple of posts manually that weren’t in my last backup. I’ll then have to find all the email lists that I have dropped off and resign to them. Hope like hell I didn’t receive a really important email on an email account I have had for 12 years now.
Yes I have a backup, and yes I have a plan in play to get things backup by weeks end. But there are a lot of business that may not have, and a lot of business that wouldn’t think to watch Twitter for the only updates for days.
Early on, people said they would leave Distribute not for the “hack” which may well have been out of their control, but simply for their lack of communication. I was at that point last week. Ready to leave, just hoping I’d get something back. Instead the money I paid them two months ago to renew my site has gone much like my site and email… into the ether.
Having backups helps, it will help me, but think about your hosting what would you do on a long weekend with an advertising campaign hooked into a website that just got deleted.
– Update: 26/6/2011
Finally back up and online. Turns out a few things took a lot longer than I thought. I had all but 20 posts, but those missing 20, of which I had the content were a pain. Having them as emails, meant I had to strip HTML, fix errors in the HTML etc. Surprisingly tedious as task as well. I did get the domain back by Thursday, but was another 24 hours for the name servers to catch up as well, so really have only had email for the last day or so.
That means two weeks downtime in total. Yikes…. Well I have automated some more of the backups, but it was all the little tweaks to WordPress that I am still sorting out, but all seems good now.
Those last twenty posts are also broken to Google. All the Post ID’s were out on the new posts. Bugger, so inbound links on those 20 posts ( inc a couple of most my popular posts ) are going to be confusing to some folk. So a few annoyances, but as I said in the orignal post, mine is just a little blog. If this was a companies website I’d be in deep poo as it were :-).
So I’m off to make a backup now I have put it all back together…. have you backed up?
