AWS is Down - This is Why We Need Decentralised Storage like Storj etc
A Video About Storj - A decentralised Cloud Storage Platform
AWS is Down
In case you haven't heard yet, AWS (Amazon Web Services) is down.
If you are trying to make a post and are having difficultly adding images this is the reason why.
Whilst all the text information for Steemit is held on the blockchain, images take up too much space to do this.
Like many other web companies Steemit currently uses AWS to store images.
Since so many websites use AWS a lot of websites are having similar issues.
The huge success of AWS represents a single point of failure which can bring down a large chunk of the web.
This highlights why we need decentralised storage in general.
Whether it is due to a DDOS attack, server outage, or malware attack we need more resilient systems everywhere.
Further I think blockchain technologies like Steemit which are built on a decentralised ethos should really be embracing this as soon as possible to highlight the advantages.
I know that it is only images that aren't working but to the average punter who has grown up on rich web content it may as well all be gone.
Solutions
Personally I am most familiar with Storj as I invested in their ICO a couple of years back. I have also encountered IPFS.
Storj attempts to decentralise storage by letting people rent out space on their own hard drives in exchange for Storj tokens. IPFS is slightly different.
Juan Benet has an explanation of how IPFS (Inter Planetary File System) would work in the following video:
From what I understand IPFS is a bit more ambitious as it tries to create an entire protocol level of decentralisation but please correct me if I am misunderstanding.
Anyway even without knowing the nitty gritty of it I think it is pretty obvious why we need these kinds of systems.
A huge chunk of our lives is currently stored in centralised servers. Their failure can in certain circumstances bring modern life to a halt and this will only become more problematic as more and more services move online.
IPFS is basically the love child of git and bitorrent.
I've used it, it works well. But my preference right now is for https://zeronet.io/
I find the adjustments they made to the IPFS protocol to be more secure and more scalable.
The big problem with all of these storage schemes though is that your data can age out of the system. In the case of Storjx if you don't pay the bill, you lose your data, simple as that.
This makes it unsuitable for long term storage.
All of them though eventually fall back to AWS S3 or something similar and this leaves us with the cloud problem.
https://xkcd.com/908/
The only solution to that is going to be the adoption of content addressable networks and a change in internet topology to open mesh networks. That will require the realization that unlimited high speed internet is as fundamental to the modern human experience as electricity was to the previous generation and water to the generation before that.
Until we have those things in place, we will keep facing these same problems.
I love how informed you are about this @williambanks
I've never been introduced to ANY of this until I joined Steemit and I've realized on several occasions how little I know.
Thank you so much for your enlightened words @williambanks - I was not aware of those issues. I am very much guided in these sorts of things by more knowledgeable people like yourself. I will check out Zeronet. Also love XKCD!
I personally have a hard time understanding where the extra hd space and bandwidth should come. One problem with a decentralized crypto storage is that many files will eventually be duplicates, but can't be treated as duplicates, but as individual files. Correct me if I'm wrong, but if you encrypt a file once, and someone else encrypts the same file in a different location it will not be recognised as the same file anymore, right?
The way I see it many of these technologies don't account for redundant files that hog up space, where as they could be identified as identical from the get-go and then replicated with the same identifier, so whenever someone needs that file, they don't need to go after their own copy, but they can use the same copy everyone else is. That would free resources for all; storage, bandwidth and the search algorithm.
MaidSafe actually already tackles this problem as can be seen here (https://maidsafe.net/features.html). I quote: "Once a file is uploaded, other users who upload the same file will be automatically referred to the original, limiting the number of copies and reducing the computing resources needed to store it."
Thanks for the info. I need to check out MaidSafe, I wonder if they are out of closed beta yet.
Edit: Aah, I see they are in Alpha. "*Data in alpha 1 may get lost or be wiped" isn't very assuring. Hopefully they'll release a production ready version soon.
Well, AFAIK the project has been in "pre-alpha" for about 10 years now. It's not trivial at all to build such a system and I also hope that they will soon release a beta version.
That makes sense to me but I am not an expert in cryptography - perhaps there is some way around it that a more intelligent and knowledgeable person can think of that doesn't also break the encryption.
Otherwise it would certainly seem to be a problem in terms of duplicating redundant information.
Very intelligent point thank you for bringing it up.
Storj is not ready for primetime IMHO. Please see my post. TY
Thanks yes it is probably not ready yet but I'm sure they are working on those issues. If they don't do it someone else will.
You are right. Another issue to think about is how do you know where to connect to the network? Right now we use DNS to connect us, but that is centrally. This past year we've has DDOS attacks on them.
Yes and I think that was something I was asking about not so long ago - Namecoin was trying to deal with this issue but I have not heard anything from them in ages.
If such a service could be developed, that made us of the Steem blockchain to determine what information is stored on what hard drives.. That could be really great STEEM.
If that's even possible, that is. My developer knowledge is rather abysmal.
It is probably better than mine lol! Yes something like that would be good but I don't know enough about the technicalities of doing it.
I doubt that. You seem pretty tech savvy from what I have seen. You are just being modest again...
I finally managed to beat the bot and give you a 100% UV finally after most of the last 3 weeks. I wonder if I will be flagged by a whale for my generosity.
Lol!
I posted on the Russian Steemit site an hour ago. I am getting sick of the whale nonsense on here.
Я довольно общественность о моей Crypto любовь, поддержку и восхищение Чарли в качестве основного выключателя и Liberty Pioneer для всех нас в отношении Blockchain технологий, Bitcoin и связанных с ними вещей в пространстве Crypto.
YA dovol'no obshchestvennost' o moyey Crypto lyubov', podderzhku i voskhishcheniye Charli v kachestve osnovnogo vyklyuchatelya i Liberty Pioneer dlya vsekh nas v otnoshenii Blockchain tekhnologiy, Bitcoin i svyazannykh s nimi veshchey v prostranstve Crypto.
Take that flaggers and bullies! ghahahahaha
Just what I was thinking storj are attempting to go head to head with AWS. Did you manage to get involved with the equity offering recently?
Not this time around! It wasn't really on my radar - I still have some of my tokens not much but I always keep some just in case something explodes.
I had posted on this topic a while back, all my photos in that post were uploaded to IPFS. During this blackout I went back to that post to see if my photos in that post were accessible and they weren't, surprisingly.
I am not sure if IPFS is the solution. In theory its great and Juan Benets video really makes you enthusiastic but its been a while coming and its still not here in a big way. Storj or Maidsafe or Sia may be better solutions.
That is because Steemit isn't directly linking to photos. It copies them and uploads them to AWS instead. That is probably why it stopped working for you. True IPFS integration or Storj integration would get around that.
That is a interesting and is very restrictive especially when you cant edit old posts.
You will be able to edit with the next hard fork.
Evolution of the net became clearly visible.
Yes and from my (limited) understanding that is what IPFS is aiming to do but until we get it things like Storj are a good alternative/addition.
Only one component of AWS, S3 was down, and I'm not sure how occasional and infrequent downtime dictates some need for a decentralized alternative.
99.9% of the time, S3 is amazing and works great. Decentralization is great but there is a complexity and overhead tradeoff. I think it's the future but a small outage does not suddenly mean we require decentralized storage.
Interesting. So you would prefer to wait until the problem was more serious before you deal with it?
Classic contemporary thinking.
Source: http://gunshowcomic.com/648
Considering that we'd be engineering for the 0.01% case, I think you are being a bit hyperbolic.
If anything, with each outage S3 becomes more robust and reliable, so the problem shrinks over time.
Maybe but I think it is something that is important . I understand that it is not a priority for Steemit but I think for the internet as a whole we really need to consider these things.
What makes you think that people aren't? IPFS and FileCoin, when it exists, are just one of several attempted solutions which will materialize over the next few years.
Also, if we cared about the couple hours of S3 downtime each 3-4 years, we could double our monthly storage costs and save our objects in Azure or Google alongside S3 but the cost/benefit on that is ridiculous.
I mentioned IPFS in the post.
You seem to be very hurt over this. I was talking in general terms and not just about Steemit.
Let's just leave it there though because I don't see the point of continuing this.
I never got this upload image direct to work and have also been having problem submitting post. Strange. I sign off, clear cache and could not sign on IE again. So, I'm leaving my Chrome signed in to do the upvoted but could not Submit post. Then on another network, I could signin on IE and post. Later on, cannot post again. Guess still in beta have to have less expectation.
I have been having similar issues lately with lots of disconnections, generally slow behaviour. Not sure what is up it started before the AWS problems. Further I thought the AWS problems were sorted but there still seem to be issues with some things.
Thanks for your reply. Good to know that it is not that it's not liking me only.