Steem Engine Tender: A Sidechain Explorer - 0.0.4 - Performance

in development •  16 days ago 

TL;DR: The site should now be more responsive: https://steem-engine.rocks/

There's been a lot of growth in the Steem Engine sidechain. And I wasn't keeping up with it. I had a slight hack that came back to bite me due to this scalability oversight.

The main feature in this release is the addition of a proper account/symbol index (instead of the hack). Prior to this, account and symbol lookups were done ad-hoc, which was fine when the sidechain was under 400,000 blocks. But now that it's over 1,000,000 blocks, something had to be done.

I actually solved this weeks ago, but needed to rebuild the index in order to get it on steem-engine.rocks. In the meantime, my staging solution died (made useless by a single fan failure, actually) and my ISP throttled me, so I wasn't able to get a new index up. Well, those are my excuses.

Because of these roadblocks, I spent some time tweaking the performance of the transaction processing tasks. Either I could throw more hardware at the public server or make the process more efficient with the existing hardware. Why not both?

One of the design goals for tender is for exchanges to be able to run it independently so they don't have to rely on my public version. If they did this, they'd probably be fine because they'd likely have the hardware required, while I simultaneously applied the real fix.

Other features/fixes added:

  • Bug fixes, gem updates
  • Fixed some UI elements
  • Tweaks to contract interpretation
  • Added a contract action baseclass that all actions now inherent from.
  • Tweaked replay detection
  • Improved checkpoint evaluation
  • Added "turbo mode" to transaction processing
  • Moved redis keyspace reclamation to after commit

Turbo Mode (back-end transaction importing)

For the initial index, I managed to get processing on my staging solution from 30/tps (transactions per second) to 180/tps by deferring commits and adding a limit.

rake tender:trx_ingest[drop_redis_keys,max_transactions,turbo]  # Ingest Steem Engine transactions from Meeseeker

From the project source, run:

$ bundle exec rake tender:trx_ingest[true,1000,true]

Without these options, processing assumes you want all available transactions processed, just like before. But with these options, the above example asserts we only want to process up to 1,000 transactions in memory, then commit them, then drop the keys from redis so they won't be processed on the next pass. This allows the processing to still be interrupted without creating gaps in the index.

As always, you can launch your own tender instance with docker (recommend: run your own local steemd to improve performance):

$ docker run -it --name tender -p 5000:5000 -p 3000:3000 inertia/tender:latest



All changes in v0.0.4 tag: https://github.com/inertia186/tender/commits/v0.0.4 (compare)

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

What is Tender?

I actually don't understand what it is besides it being a sidechain of Steem-Engine.

Steem-engine.com is pretty much useless without steem-engine.rocks, so I'm glad it's back up (and improved) :-)

I came here for the steam engine picture. That much I understand. :-)
Good choice for the picture!

Hi, @inertia!

You just got a 0.98% upvote from SteemPlus!
To get higher upvotes, earn more SteemPlus Points (SPP). On your Steemit wallet, check your SPP balance and click on "How to earn SPP?" to find out all the ways to earn.
If you're not using SteemPlus yet, please check our last posts in here to see the many ways in which SteemPlus can improve your Steem experience on Steemit and Busy.

Heya Inertia,

Know what would be really great? A place where we can see our incoming tokens reward for all of the steem engine supported Tokens.

If it exists, I don't know of it.

Cheers

Thank you for this essential tool!

  ·  15 days ago Reveal Comment