Steem Followers - Status update
There have been a few people contacting me, asking what the happy haps ( or status ) is with Steem Followers.
Right now it's rebuilding the database from scratch, and it will be a few days before it is done. Yes, days.
Why?
Well I decided to use Docker to separate my web server from my database server (They both run on the same physical device).
This ended badly when, while I was on leave and not able to get reliable internet, the droplet on which the site is running, had major issues.
Not a problem I thought, when I could connect, just restart docker!
But, no...
It ended up rebuilding the docker container, leading to me losing the database and all the data in it.
I have a backup, but it is old and missing fields I added recently, so I decided to not use that.
Instead I had the genius idea to let it run from the genesis block of the Steem blockchain and build the data like that. 😩
That was not a great move, but now I am a bit out of options.
I am trying, when I have free time, to find a way to speed it up a bit.
In closing
I can only apologise for the inconvenience caused by my little app being down for so long.
Once I have gotten it caught up, an incremental backup strategy is of the highest importance, so as to prevent issues like this in future.
Hey @shaunmza, I hope everything goes well and the website will be online soon. Do you have any idea when?
I changed tactics and am bringing in January data first, full update at https://steemit.com/steemdev/@shaunmza/steem-followers-january-2018-data-is-rolling-in
Days, probably more than a week.
I thought I could make it go quicker, but am limited to the speed at which I can fetch blocks from the full node.
Am hatching a plan though...
Thanks a lot for taking all the time and effort to get that puppy back online and running again. A very useful app indeed for this platform, and to keep track of things. Pax vobiscum and be well.
all the best @shaunmza, I soooo miss the friends app. anyone know of an alternative in the meantime?
If you are ok with only seeing data for January 2018 for now, it will be back in a few hours.
Full update at https://steemit.com/steemdev/@shaunmza/steem-followers-january-2018-data-is-rolling-in
AWESOME!!!! Big shout out - thanks for all you do - much appreciated. Nice to see who is Following info back again.
All the best gettign it back up and thanks for all the work you put into this its one I check daily or did till the issues
Good luck with the recovery and thanks again
PS will it still be at the same URL or will it be a new one?
Yes, same url. You can access it again.
I changed tactics and am bringing in January data first, full update at https://steemit.com/steemdev/@shaunmza/steem-followers-january-2018-data-is-rolling-in
Thanks I just saw appreciate the hard work
Good to hear! It was helpful to me, glad it's coming back.
I am trying to find a way to bring it back sooner!
I changed tactics and am bringing in January data first, full update at https://steemit.com/steemdev/@shaunmza/steem-followers-january-2018-data-is-rolling-in
Nice, thanks for the update!
Hey @shaunmza, great work on your followers app, sorry you've had some troubles!
According to your post here it looks like you're using MySQL.
You just mentioned:
So I wanted to suggest this MySQL Backup Script I wrote that should help you out.
You can easily deploy a backup instance at DigitalOcean using this script instead as the User Data like I outlined here and have offsite backups running in a couple minutes!
I wanna support the Steemit community, especially those like you building on the blockchain, so let me know if you'd like some help implementing a backup plan, or really however I might help.
Looks good, I resteemed it too.
Let me know if you do something similar for an incremental backup too?
The database will be quite big, so backing the full database up will take a ton of time, especially on a small DO droplet like I use.
What I am thinking of is a full backup once a week, with incrementals every 12 hours or so.
Nice work though :)
Thanks for the resteem!
So, your best bet is to deploy a small droplet (like really, just $5/month) and then attach Block Storage to that instance. The storage is only $0.10/GB per month and you can expand that as your storage grows.
My script literally grabs everything, every table in every database. Like you said, that's probably overkill for your needs but you can tweak it some to fit your scenario. You can easily dump specific tables rather than the full db, which might fit a little better.
For the blockchain data, the full weekly and 12 hour incrementals makes perfect sense. For the incremental, a quick approach is to just use the mysqldump where clause to limit your dump for your incrementals.
You'll need to specify exact table names for a dump like this so you might have a couple separate
mysqdumpcalls.As you move this forward, seriously let me know if you have any questions!
For your app data itself, that I would recommend backing up constantly, at least daily. While it takes a few weeks, the blockchain data can always be reproduced, but the app data is gone.
Any update?
Plan C, D and E failed.
On to plan F
I am trying to batch the imports now, getting data from SteemSQL.
I think this may work though, as I won't be dependent on parsing every single block ever produced.
I changed tactics and am bringing in January data first, full update at https://steemit.com/steemdev/@shaunmza/steem-followers-january-2018-data-is-rolling-in
Is it possible to include Steem Power of followers in the list that is provided?
It definitely is, let me get it up and running again, with backups, then I can look into that.
hii @syaumnza Introduce me new in steemit please support upvote and follow