Dtube's video uploads, why they were bad, how it's my fault & why it works now.

Lets open with a little bit of background;

dtube (you may have heard of it) is a distributed video sharing site with a backend powered by IPFS.

ipfsstore.it provides an endpoint for file uploads in ipfs to dtube.video & long term file storage.

prologue

IPFSstore was intended as a long term file storage location, it provided an ipfs gateway (the things that lets you view a remote servers ipfs content in your browser) and for the most part it was essentially never used, the project looked as though it would fade into obscurity not unlike so many of my other projects before.

I was approached by @heimindanger and we were able to work together with my service providing a file dumping ground and assured long term storage obviously I accepted and a after a few days testing we were in business.

Dtube launched, everyone was happy

"Dtube is bigger than jesus" - anon

I was a small part of that and behind the scenes I was quietly thrilled to see the service taking off.

Problems Brewing

IPFSstore is hosted in google's cloud, for the reason of: I could get lots of storage very cheap and what else mattered?

Google charges for network egress (ipfsstore sending files to anywhere else on the internet) and it turns out when you pump out just shy of 1TB in traffic over 4 days, the costs started to add up.

My service was always intended to run on major thin margins and this change was something I hadn't accounted for.

The resolution was a fairly simple one, move to a service that wouldn't charge for network egress.

Migration

A cheap dedicated service in France was purchased ($15) 1TB HDD, 2.5Gbps bandwith with no limitations (something called a fair use policy applied but you could pay to get out of it, so if it became an issue, I figured I would)

ipfsstore's code was migrated, a new ipfs & bitcoin node were setup using use the keys from the old server and when everything was ready or just before if I'm being honest, dns was switched

It was late ~01am local time and I was looking forwards to watching the dns changes roll over and users migrate gracefully to the new site, literally non-the-wiser. I was going to sit back, puff on my pipe (I don't smoke). I was going to have a few glasses of whiskey (I don't drink whiskey). I was even going to sit in my large armchair in front of the fire (I have neither of these either) and watch the everything become better.

This did not happen.

Users migrated over slowly and reports of failed uploads and very slow uploads came in, network utilisation on the new host was at 100%, the processor was struggling and things were "not good"

with migration done I didn't want to roll-back instantly and call the whole thing a loss (I'd dumped $45 on access to this server for 3 months I didn't want that to be lost and I didn't want to have to return to paying google's prices)

I decided to attempt to implement cloudflares CDN as a method of reducing both network and cpu load on the new server, as after one failing migration the only course of action is clearly to hastily implement another solution.

NS servers were switched and load reduced as expected, however uploads continued to fail

With two failing solutions I decided to revert entirely and move the traffic back to the old server.

Now
This isn't the first time dtube has had issues due to my actions but it was the longest outage by a significant margin.

Firstly I would like to personally apologise to the userbase, after hearing how and why the situation came to be I hope you can empathise with me somewhat and I can assure you that I only want the best for dtube and its users.

I hope you can appreciate my openness and honesty in regards to the outage and understand that this type of outage is not to be considered standard.

What's next?

The gateway is the most significant cause of network utilisation and therefor cost, with the new server now laying dormant I intend to redirect all GET requests from google to the "new" server

For users this means that gateway traffic will come from a difference source meaning uploads should be faster, gateway traffic should be faster and in general the ipfs network will be strengthened by another large node existing upon it.

I anticipate this should reduce network egress by a significant enough margin that the new server will essentially pay for its self

The risks are that the gateway traffic will be slower however with the option to switch gateways in dtube this shouldn't pose an overly large problem

This change will be implemented between 21:00 -22:00 UTC.

TLDR;

• ipfsstore started renting a new server to reduce costs
• The server couldn't handle everything that was being thrown at it
• I tried to wrap everything in cloudflares cdn but this caused further issues
• Everything is back as it was
• I am going to make things better.