INTERDUBS status

An otherwise smooth running August was considerably spoiled by the August 17 outage.

Enough that we feel that we have not delivered what our clients would want to pay us for.
In order words: August is free for everybody – should they choose to.

August 17th
At 3:43pm our main systems became in-responsive. Something that will get our immediate attention. We still waited with the switch over to the backup system until we had somebody in the data center. In hind-sight the first of multiple mistakes of that day. Over the next 4 hours I personally was able to amount to a total of 51 (!) minutes of downtime. I spare you the details – but most of it would not have been necessary.

Lessons from August 17th

We found the cause of the initial failure, and removed it.

We revised our backup fail over procedures.

We set a re-design process of the entire system in motion that will have an automatic fail over in the future. This process will span a couple of weeks.

All clients have been informed they that do not need to pay their August invoices. We know why we invoice in the end of the service period: First we like to provide 100% service. And then we ask to get paid for it.

Infrastructure changes:

We moved our online documentation to a new home at doc01.interdubs.com. The change is transparent since all links will automatically forward to the new location.