October 3rd 2011: As of 6:36am PDT parts of our upstream network seems not to be working properly.
Only http but not https traffic is affected by this.
It turns out that the filters were put into place on the network of one of our upstream providers to
mitigate an DDOS attack.
As of 7:03 PDT things are back to normal.
On Wednesday 9/21 between 2011-09-21 16:32:37 and 2011-09-21 16:36:37 INTERDUBS might have been unreachable for some people.
One of our network providers did some configuration changes that were flawed and resulted in overloaded circuits on their end. They addressed the issue less in four minutes and were responsive and forthcoming with details on this matter. Since we have used them for quiet some time with great success in terms of performance, reliability and service we do not see this as a trigger to replace them in our provider mix.
Sorry should you have been impacted by this.
The actual INTERDUBS system was not impacted by this problem in the surrounding network.
Judging by the traffic volume we see that only a small minority of clients has been impacted.
An otherwise smooth running August was considerably spoiled by the August 17 outage.
Enough that we feel that we have not delivered what our clients would want to pay us for.
In order words: August is free for everybody – should they choose to.
August 17th
At 3:43pm our main systems became in-responsive. Something that will get our immediate attention. We still waited with the switch over to the backup system until we had somebody in the data center. In hind-sight the first of multiple mistakes of that day. Over the next 4 hours I personally was able to amount to a total of 51 (!) minutes of downtime. I spare you the details – but most of it would not have been necessary.
Lessons from August 17th
We found the cause of the initial failure, and removed it.
We revised our backup fail over procedures.
We set a re-design process of the entire system in motion that will have an automatic fail over in the future. This process will span a couple of weeks.
All clients have been informed they that do not need to pay their August invoices. We know why we invoice in the end of the service period: First we like to provide 100% service. And then we ask to get paid for it.
Infrastructure changes:
We moved our online documentation to a new home at doc01.interdubs.com. The change is transparent since all links will automatically forward to the new location.
Wednesday 8/17/2011
At 3:43pm 8/17/2011 our main systems in the LA INTERDUBS data center went offline.
On 4:16pm 8/17/2011 the backup systems became operational
At 7:50pm we switched back to the main systems again.
Please see the status link in your admin interface for details.
Obviously an outage like this is not what we have in mind when
we offer 100% uptime. Therefor none of our clients will have to pay for August.
We will re-engineer all necessary parts of the system so that an outage like this
can not happen again. The admin interface link has details about this.
The European INTERDUBS data center has not been affected by this outage.
All data centers performed 100% in July 2011. No outages or service interruptions.
Other service interruptions and events:
On July 19th the droplet processing pipeline got stuck for some segments. Uploaded files did appear in INTERDUBS only after we addressed this issue. Everybody affected by this has been informed and properly credited. No data was lost. The bugfix will prevent future events of this nature.
Infrastructure changes:
We added the ability to host content via a EdgeCast CDN. Choosing the delivery network is a simple setting in the configuration and can be set on a login, look or segment basis.
INTERDUBS works – just as it should.
