INTERDUBS status

December finished quietly what turned out to be an amazing year for INTERDUBS. We had never thought that our concept to rely on word-off-mouth alone for growth would have worked so marvelously.

We would like to express our gratitude to all our clients for spreading the word in 2011.

With a constant increase of clients (and revenue) we were able to expand our system both in terms of features (we are running update #3148 today) as well as infra structure. This year we dove even deeper into the technology side of things: We are looking forward to put our peering agreements, data center additions and hardware upgrades to good use.

Oh – and our up time was 100%.

To put that into perspective: A competing service racked up 2 hours 15 minutes in 55 outages in December. Which results in an up time of 99.7%
But we never set out to have other services as our standard. We simply aim to achieve 100% availability. Each month.

Glad we managed to accomplish this in December. January started 14 hours ago and requires 30.5 more days of work and care.
We are looking forward to it.

Things were running smoothly in November.

We crossed the 3 million file mark. But that number change only got noticed by us human observers. The systems don’t care it seems. As they say: “Once a system scales it scales”.

Another month with 100% uptime and availability.
Which brings our overall and actual uptime since we launched in 2007 it to more than 99.997%

Things were running well in October. We doubled our data center footprint and installed new networking gear in anticipation for future growth.

We focused mostly on management of large data and archive sets and established a solid brackground rendering architecture that will be very handy in the future.

We also were able to lower the price for plain vanilla ftp to $1 per GB.

Our systems were running smoothly in September. System changes resulted in page load times that are almost twice as fast as before. We are still in the process of upgrading our infra structure in a reaction to the August 17th event. Parts of puzzle are in place. Others take longer to test than we had wished for.

September is also the month in that we became an Autonomous System. While this has no direct impact just yet, it is the foundation for very fine grained control over our network traffic in the future. And it makes us the proud actual owner of a healthy range of IPv4 addresses.

We saw a brief and partial network outage on 9/21 and had another issue in the morning of October 3rd. As a result we will try to expedite efforts to bring more of our network filtering under our control.

October 3rd 2011: As of 6:36am PDT parts of our upstream network seems not to be working properly.
Only http but not https traffic is affected by this.
It turns out that the filters were put into place on the network of one of our upstream providers to
mitigate an DDOS attack.

As of 7:03 PDT things are back to normal.

On Wednesday 9/21 between 2011-09-21 16:32:37 and 2011-09-21 16:36:37 INTERDUBS might have been unreachable for some people.

One of our network providers did some configuration changes that were flawed and resulted in overloaded circuits on their end. They addressed the issue less in four minutes and were responsive and forthcoming with details on this matter. Since we have used them for quiet some time with great success in terms of performance, reliability and service we do not see this as a trigger to replace them in our provider mix.

Sorry should you have been impacted by this.

The actual INTERDUBS system was not impacted by this problem in the surrounding network.
Judging by the traffic volume we see that only a small minority of clients has been impacted.

An otherwise smooth running August was considerably spoiled by the August 17 outage.

Enough that we feel that we have not delivered what our clients would want to pay us for.
In order words: August is free for everybody – should they choose to.

August 17th
At 3:43pm our main systems became in-responsive. Something that will get our immediate attention. We still waited with the switch over to the backup system until we had somebody in the data center. In hind-sight the first of multiple mistakes of that day. Over the next 4 hours I personally was able to amount to a total of 51 (!) minutes of downtime. I spare you the details – but most of it would not have been necessary.

Lessons from August 17th

We found the cause of the initial failure, and removed it.

We revised our backup fail over procedures.

We set a re-design process of the entire system in motion that will have an automatic fail over in the future. This process will span a couple of weeks.

All clients have been informed they that do not need to pay their August invoices. We know why we invoice in the end of the service period: First we like to provide 100% service. And then we ask to get paid for it.

Infrastructure changes:

We moved our online documentation to a new home at doc01.interdubs.com. The change is transparent since all links will automatically forward to the new location.

Wednesday 8/17/2011

At 3:43pm 8/17/2011 our main systems in the LA INTERDUBS data center went offline.

On 4:16pm 8/17/2011 the backup systems became operational

At 7:50pm we switched back to the main systems again.

Please see the status link in your admin interface for details.

Obviously an outage like this is not what we have in mind when
we offer 100% uptime. Therefor none of our clients will have to pay for August.

We will re-engineer all necessary parts of the system so that an outage like this
can not happen again. The admin interface link has details about this.

The European INTERDUBS data center has not been affected by this outage.

All data centers performed 100% in July 2011. No outages or service interruptions.

Other service interruptions and events:

On July 19th the droplet processing pipeline got stuck for some segments. Uploaded files did appear in INTERDUBS only after we addressed this issue. Everybody affected by this has been informed and properly credited. No data was lost. The bugfix will prevent future events of this nature.

Infrastructure changes:

We added the ability to host content via a EdgeCast CDN. Choosing the delivery network is a simple setting in the configuration and can be set on a login, look or segment basis.

INTERDUBS works – just as it should.