SteemDB.com - Migration complete and a few thoughts on it's future.

in steemdev •  9 months ago

Well, it took longer than I anticipated, but SteemDB.com is slowly coming back to life.

The data migration I started a couple days ago completed earlier today and I'm bringing the services back online. If you still see the "DB Migration" message still (on every page), your DNS has not updated to the new server. The DNS should update in the coming hours and the site will then become available once more.

If you're very impatient, you can always access the site via it's IP address: http://144.76.176.150

New Server

The new server SteemDB is hosted on is now provided by @privex, and I've had great support from @someguy123 getting this all setup.

SteemDB went from a 96gb RAM server with 8 cores to a 256gb RAM server with 12 cores. I've optimized and upgraded all of the software to the latest versions, configured the filesystem for high IO throughput, and should have plenty of room to grow over the next year or so.

Current Issues and Status

A few notes about this new deployment and issues surrounding the migration:

  • The database sync is currently catching up, as I write this it was processing block 18636474 (2018-01-02). So currently the 9 most recent days of data are missing from some sections. This new server is processing these blocks at a rate of about 3-6 blocks/second. We are currently at a height of 18892572, which means there's about 250k blocks that need to be processed. At a rate of 3 blocks per second (to be conservative), it will catch up to the current block height and finish processing in about 24 hours.
  • The witness pages and other sources that pull data live are unaffected by the sync process, but pages like an accounts transfers/votes will be impacted by the sync.
  • The account history module is disabled for the next 24 hours, so no new snapshots of accounts will be created until the sync is completed.
  • The apps page is missing data and somewhat broken, and I will investigate what's happening here. It's most likely due to the inactive history module, which also creates snapshots of app usage.
  • The homepage is no longer cached, so it's somewhat slow. I'll work on caching this again in the future.
  • Any missed blocks by witnesses during this migration were not recorded unfortunately.

Where SteemDB goes from here

Over the past month I've been doing a lot of thinking about the various projects I maintain in regards to the Steem ecosystem, and one reoccurring thought I have is that I have too many active projects. I am still in the planning phases, but one of my goals in 2018 is to shed or offload some of this workload so I can focus on the projects that actually matter to me, as well as the ecosystem as a whole.

With that being said, SteemDB isn't one of the projects that I want to personally stop maintaining. For months now in casual conversations I've mentioned that I really want to rebuild SteemDB and create a v2 of my block explorer code, and I am hoping to find the time to do so this year. SteemDB was my 2nd Steem related project (created Aug 2016) and I've learned so much since it's creation.

In 2018, I would very much like to take the time to sit down and rebuild SteemDB into something better. Improve the technology, the scalability, it's responsiveness, and usefulness. These foundational tools (wallets, blockexplorers, etc) are crucial to on-boarding new users and developers alike, while providing external transparency into the operations happening on the blockchain.

Luckily, this new server has room to grow, and I can fit a v2 right alongside v1 until it's ready to replace it :)

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Awesome, now lets get some content to the top thats has some artistic nature to it! Right now its all dominated by memes and blockchain news. Please check out my new grand canyon time lapse films if you have a moment. https://steemit.com/video/@cinemaninja/bmvu7-grand-canyon-a-4k-time-lapse-film-ninja-cut Thanks!

·

I couldn't agree more. Once we start having less of a crypto focus and more of a general focus... that's when Steemit will really start to take off.

Good luck taking this app to the next level.

Hi @Jesta,

I heard you during the meeting on Discord in December. I use this comment section as an opportunity to thank you for the work you are carrying out for the community. I discovered Steemit about 3 months ago, am now completely immersed into it and am pretty amazed about what is going on here!

And a lot of this originates from people like you.

So, warm thanks :-)

·

It's a pretty deep rabbit hole :)

Thanks!

·
·

@jesta my Dear friend Your Such Great posts Check My Blogs Did You like Support Me Thank you bro

·

yes its great.
thanks for nice words.
@qami

·

we're pioneers ;)

Best of luck for your data migration work, I hope you did it very well and in the future you will do better work.

That's a monster server!

Out of curiosity is the data stored on SSD drives?

·

Yup, all stored on SSDs, and the entire DB is kept loaded in RAM for even faster access.

Checking it now, very cool!

so interesting!

Never knew this wesite and I know almost all the tools. Thanks man, I'll check it out!

·

Hey all!
I have put together some picks. They are low market cap coins that I believe have great potential for profit. Obviously there is risk and do your own diligence before buying. I am not a financial planner or advisor.
In no particular order:

  1. AirSwap AST (Binance)
  2. Lamden TAU (EtherDelta)
  3. Loopring LPC (Binance)
  4. aelf ELF (Binance)
  5. Luxcoin LUX (Cryptopia)

(Follow me on my new Steemit for daily picks, and have an excellent day!)

·
·

good . thanks for sharing nice information.
@qami

·
·

I bought AirSwap in Binance some weeks ago.Good call

·

Absolutely. All the analytics sites that are being built are a huge help to managing accounts.

·
·

You are right bro, the system needs more tools to gain control of our accounts.
@racoo

·

I'm learning about new tools as I don't know about more #steemit tools.

·
·

Yes, I've the same minings as you, shohana1

·
·
·

wow we both are learning, nice to meet you new friend

·

I also will check this out! Thank you 👍🏽
Upvoted 👍🏽👍🏽👍🏽

·

upvote

Thanks!

How did I not know about this, the website looks amazing man! Glad someone tipped me off to follow you, please keep improving the site, I think it's looking clean and professional already, and so much info, this is an awesome find for me.

Voted for you as witness, really appreciate the work!

·

true, totally agree with you! there needs to be more awareness of the site!

·
·

WHATAVAR YA DU PPL : HODL !

it's good!

Finally I can see how much my reputation SUCKS :P

·

better than mine. I'm not even worth 3 bucks. Way to depreciate one's organs, steemit.

wow, thats some upgrade ,from 8g to 25g, its nice to have steemdb.com back online, missed using it,a very resourceful steemit/steem data collection site

Sounds like things are progressing well. You always do the best work and i can tell you work hard. Thanks for everything you do here for all of us

Well, it took longer than I anticipated, but SteemDB.com is slowly coming back to life.

Glad to know @jesta. Thank you for your contribution! SteemDB is a very helpful service!

I love jesta creations. Thanks for the hard work you do. We need it. :)

·

I agree with you, we need it.

Thanks for the update, @jesta. Which projects are you thinking about offloading? I agree that a richly featured block explorer is vital for people to understand what's going on under the hood. I'm glad that's not the one you are considering dropping.

Thanks for the updated information and looking forward to SteemDB.

Glad to see it coming back as I know so many of us rely on it!

Great though on future.

Are there plans for dealing with the copy/paste reply bots that people are using to game this system? How about for disincentivizing people to post crap comments that contribute nothing? These kinds of things are going to be very bad for this platform in the long run if a stop isn't put to it early on.

·

I agree, they are problems. There's a lot of talk happening on this subject at this point between the witnesses but no concrete answers as of yet.

SteemDB won't solve those problems though :D

·

While I agree with your sentiment, the question then becomes (as in the real world) who is to judge what is a bad comment or in searching for the users gaming the system? How do we find the middle ground where that type of system doesn't censor opinions that are unpopular or that deal with sensitive topics? The second this turns into a Twitter or Facebook with censoring users I believe this platform is heading in the wrong direction.

Thanks @jesta for bringing back SteemDB.com online again...

Great, it would be super cool to have a chart area also!

The only place for steem to go from here is up!!!!!!!!

·

Say no more.

I have no idea about the #steem ecosystem. If it's growing up, that's good sign.

@alexKARKI

I will check this. Thanks for sharing

@jesta did you do all this? Wow, you must be so smart!..

you guys are doing a great job

100% UpGoat.

sounds good, I remain hopeful of the future

@jesta Goodluck hopefully it is going to be fully functional soon!

happy to join in steemit, successful greetings for steemit lovers,
continue to work
hopefully hope and dream of steemit users in this year can be achieved all..!

Hi @jesta. Thank you for your work.

I just started last week and I only had it for a day or two before it went down and I was wondering what was up with that. Thanks for the hard work.

Really nice post, keep up improving the site, looks great :)

The new site looks GREAT. It is actually quite difficult to present so much info in a logic, nice-looking and readable way. This one looks spectacular.

Thank you very much for the hard work. Voted you as a witness, thank you!

I really like steem but i kinda think its a bit slow

I've missed SteemDB, welcome back!

I really want to rebuild SteemDB and create a v2 of my block explorer code

That is such a good news. Steemdb is such a important part of Steem.

one of the things i like about steem, from a software development perspective, is that if you develop the right app, make it free (as you have), if enough people like it you'll make just as much money as if you charged for it elsewhere.

great work. Hopefully i can think of something to provide to the community at some stage in the future.

posts section is not accurate.

·

Yeah, I forgot to mention that in the OP.

One of the things I disabled in a last ditch effort to keep the site running in december was the collection of posts. They make up a huge amount of data within the database.

At this point I'm not going to re-enable it for version 1 of steemdb, but with version 2 I do plan on recording that data once again.

This is quite unclear to me 🤔

To truly reflect the big data value of the work you do, you'd need to move that decimal point above 6, maybe 7 places to the right!

I'm glad to hear that you will continue to support SteemDB. Thank you!

steem being a population on its own i never knew it would have a sample website til my friend @ogochukwu schooled me more on steemit, what ever the problem is, it can be fixed

A few days were in great trouble.I'm glad that the problem is partly healed. Thank you for the information.

Thank you so much for all the hard work you have put into the site. It clearly shows. Right now I've been trying to tell my friends to migrate over here and leave the exploitive nature of so many other places. I feel really cherished on here.

Excellent! I was going to test with connecting apps I was developing only recently and appreciate your efforts, and monster server :D Are there any public projects that I may help out with? Cheers!

Thanks for that information sir. Hope you could follow me someday

This is great news! I regularly use steem.supply and steemd, but less familiar with SteemDB. Would love to hear from others what the most useful data on SteemDB is so that I can better take advantage of it!

It requires a lot of effort, thanks for the had work :)

Wow, I'm quite excited that I belong to this great family of diverse opportunities

happy to join in steemit, successful greetings for steemit lovers,
continue to work
hopefully hope and dream of steemit users in this year can be achieved all

Do you have a discord group??

Great feat! You inspire me greatly with this project! Keep up the good work!

Cracking work you have done.. witness vote your way!!

·

Is this what you mentioned?

Really thannk you for this service!

All I can say is that the future looks bright and secure. I just hope the programmer breaks the code before they steal it. Hahaha!!!

@jesta thanks for your post and a deep look inside the matter. today I tried to access the website on steembd.com but it comes up with site who tried to phish the spam into system. wondering... you still own the Domain? help regarding this problem will be appreciated.
PEAcE

·

steemdb.com, not steembd.

amazing! now I don't have to click on page by page to check the value of steem vs btc.. thanks for this @jesta. helps alot!!

Que buen recuento, tratare de volverlo a ver, para aprender mas .

good content!

Good job on the site! I gives me very interesting stats that help me undertand how Steemit works since im kinda new. Its also good to see who upvoted me! :)

Can someone please explain this like I'm 5?

How did I not know about this, the website looks amazing man! Glad someone tipped me off to follow you, please keep improving the site, I think it's looking clean and professional already, and so much info, this is an awesome find for me.

Voted for you as witness, really appreciate the work!

hi to work well with your willingness to pay for your work

great job @jesta and very nice to see this news
all the best for you
thanks =)

How did I not know about this, the website looks amazing man! Glad someone tipped me off to follow you, please keep improving the site, I think it's looking clean and professional already, and so much info, this is an awesome find for me.

Voted for you as witness, really appreciate the work!

This post has single-handedly dealt with issues of immense importance. It was so helpful especially to new people here'. Thanks @jesta

Congratulations on your project @jesta :) Its one of the first that i heard about here on steemit :)

So many new posts about cool Steemit projects in my feed today. Congratulations on your progress and keep us posted! @jesta

Thank you, will check it out. I'm already using your steemstats monitor and find it very useful. Just voted you for a witness, thanks for all your work to make stemmians lives easier

finally it is back, everytime I went to visit this site, always got the error|

We cannot create an account unless we have steempower already? Thanks :)

great information sir. Hope you could follow me someday

wow! so youre the one who is running this thing. Man you rock!! this thing helps, the realtime data and stats. thanks for this.

all the tools is old add something new in the site.

·

nice..i am hopefull you will upvote my posts also

So thankful for the awesome information, waoooh this is rich. Thank you.

Great info and I will check it out. Voted you as witness.

@jesta what is happening with
https://golosdb.com ?
seems it is falling weeks behind

·

Probably the same issues that SteemDB has been suffering from over the past couple months.

Over the past few weeks I've made tweaks to golosdb, but the RPC nodes available to retrieve data from are also very slow on the golos end of things. It's still running... just slowly.

Yet another reason why I really want to start working on a new version of the explorer (so it better supports multiple chains).

thanks a lot for such information as I look for some answers which I get from here!

very good

Thank you for your contribution! SteemDB is a very helpful service! ;)

you have to be patient

follow me and I follow you @sahari

Excelente iniciativa @jesta
Como verás hace poco me inicié en Steemit y realmente estoy fascinada con todo lo positivo que se maneja en éste mundo de aportes.
Considero que vienes trabajando de una manera genial y te deseo muchos éxitos y más crecimiento...!!!

i am happy to see this post. since i joined i started using SteemDB.com and depended on the site. but i had some problems and as a new user thought i did some mistake and it didn't work for me. late i came to know there are some issues and finally they are getting rectified. good to know this. when we are used to one site conginuing on it is very conveninet for me.

I hear that we can earn steempower when comment? really?

Thanks for the update

Amazing and very interessing post thank you for sharing
resteemed and shared in the steemit

i ever seen this site...after all reading your blog i saw that and i kindly requesting to improve your site, i'll check out of there

Good post! .. I will follow you follow me too and we help each other ... I would appreciate a vote in my last post thanks
https://steemit.com/openmic/@misc.jocs/my-most-appreciated-possession

this is great and useful . thanks!

Hey all, I'm sorry to be commenting on an unrelated article, but I am new to STEEMIT, and I would like to do livestreams a few times a week. How does one start this process ?? Any help is greatly appreciated. Thank you all.

·

I'm not sure I have a great answer for ya on how to accomplish that. I've seen people that write posts when they start live streaming, to direct them to other platforms where the live stream is happening, but I'm not sure how that'd work here. Sorry I'm not of much help in this regard!