Steem Documentation: Using the Rake Tasks

in utopian-io •  2 months ago

First of all, I'm very pleased that all of my pull requests for devportal have reached the Steemit, Inc. official site, as reported yesterday. My goal was to try to keep my fork and the origin fork as 1-to-1 as possible. There's no reason for me to have unique documentation on my fork.

It looks like it took them only a couple days to fully merge my changes that spanned over a month. Great job, Steemit, Inc! I believe moving forward, this should be much easier now that the relative URL structure has been addressed.

Here, I'd like to go over the various rake tasks that have also been merged to the official repository.

Some of these tasks are shown in the @steemitblog post from yesterday. But here's more detail on what they're all about.


Rake (rake) is a command-line tool that means "Ruby Make." The command implements tasks defined in a Rakefile, usually in a cloned git project.

In the devportal Steem documentation project, the rake tasks are as follows:

$ rake -vT
rake ops_dump[vops,appbase]          # Dump all operation types
rake production:deploy               # Deploy current master to GH Pages
rake production:rollback             # Rollback GH Pages
rake scrape:api_defs                 # Scrape API Definitions
rake scrape:javascript               # Scrape steemjs docs
rake scrape:python                   # Scrape pysteem docs
rake scrape:tutorials                # Scrape all known tutorial repositories
rake scrape:tutorials:js[num,force]  # Scrape JS-Tutorials
rake scrape:tutorials:py[num,force]  # Scrape PY-Tutorials
rake scrape:tutorials:rb[num,force]  # Scrape RB-Tutorials
rake test:curl[apis]                 # Tests the curl examples of api definitions
rake test:proof                      # Want some work to do?  Run this report and get busy

I use many of these tasks to help with documentation. Here's a rundown of the most useful ones ...

Scrape API Definitions

For example, rake scrape:api_defs is especially useful. As it says, it scrapes the API. But what does that even mean?

When it scrapes the API, it's asking the (full) node what methods are available and what those method signatures are. Then, it saves the scraped information to a .yml file so that jekyll can make pages to browse. For example, rc_api is based on scraped data saved to rc_api.yml.

I don't just scrape the methods, though. Once scraped, I add details. Also, sometimes the scrape is missing some of them method signatures, so I add those too.

You might think this is only useful for catching up to methods that already exist, but it's also possible to use it for upcoming methods. All I need is to point the task at a node running on a future release. Often, new methods show up on, so this will scrape from there:

TEST_NODE= bundle exec rake scrape:api_defs

In fact, I could scrape from a testnet just as easily:

TEST_NODE= bundle exec rake scrape:api_defs

It also works just as well to run a tintoy testnet if I'm looking for bleeding edge new methods:

docker run -d -p 8090:8090 inertia/tintoy:latest
# need to wait for it to fully deploy
sleep 300
TEST_NODE=http://localhost:8090 bundle exec rake scrape:api_defs

And here's the result when I ran it just now:

Definitions for: account_by_key_api, methods: 1
Definitions for: account_history_api, methods: 4
    Adding: enum_virtual_ops
    Changed: get_account_history
    Changed: get_ops_in_block
    Changed: get_transaction
    4 methods added/updated in account_history_api
Definitions for: block_api, methods: 2
Definitions for: condenser_api, methods: 84
    Dropped method: get_account_bandwidth (recommend removal from _data/apidefinitions/condenser_api.yml)
Definitions for: database_api, methods: 52
    Changed: find_accounts
    Changed: find_comments
    Changed: find_escrows
    Adding: find_smt_token_emissions
    Adding: find_smt_tokens
    Changed: find_vesting_delegation_expirations
    Changed: find_votes
    Changed: find_witnesses
    Changed: get_active_witnesses
    Changed: get_config
    Changed: get_dynamic_global_properties
    Adding: get_nai_pool
    Changed: get_transaction_hex
    Changed: get_witness_schedule
    Changed: list_accounts
    Changed: list_comments
    Changed: list_escrows
    Changed: list_limit_orders
    Changed: list_owner_histories
    Changed: list_savings_withdrawals
    Changed: list_sbd_conversion_requests
    Adding: list_smt_token_emissions
    Adding: list_smt_tokens
    Changed: list_vesting_delegation_expirations
    Changed: list_vesting_delegations
    Changed: list_votes
    Changed: list_withdraw_vesting_routes
    Changed: list_witness_votes
    Changed: list_witnesses
    29 methods added/updated in database_api
Definitions for: follow_api, methods: 10
Definitions for: jsonrpc, methods: 2
Definitions for: market_history_api, methods: 7
Definitions for: network_broadcast_api, methods: 2
Definitions for: rc_api, methods: 3
    Changed: find_rc_accounts
    Changed: get_resource_params
    Changed: get_resource_pool
    3 methods added/updated in rc_api

This is how I can be ahead of any changes that might roll out. In this example, there seems to be a lot of changes, but most of them are just method signatures with keys in different orders. The most interesting results, to me, are things like Adding: get_nai_pool.

If we focus only on that result, here's what we see newly added to database_api.yml:

  - api_method: database_api.get_nai_pool
    parameter_json: "{}"
    expected_response_json: '{"nai_pool":[]}'

What I usually do at this point is search for the new method in the steemd repo. In this case, that leads to database_api.cpp and pull request #3058. I keep working backwards until I find the reason the method was added. Actually, this one is a little tricky. Looks like I need to investigate issue #2886. But I'll do that later.

Test Curl

The rake test:curl[apis] task is like a smoketest for the curl examples in the API Definitions. The rationale is that if we assume these curl examples are correct, then testing them periodically should show us problems in the API and/or node configuration. Conversely, it could also tell is if our documentation is out of date. Either way, running this rake task will give us a hint on how out of date documentation is.

The tests also accept node options, so we can test against pre-release/testnets and get ahead of any changes that might happen before they go to production.

Ops Dump

As mentioned in the @steemitblog post from yesterday, we have rake ops_dump. When we run it, we get:


These are the broadcast ops that the devportal currently knows about, not including virtual ops. Broadcast ops are typically more important to app developers than virtual ops because they represent actions that applications can perform, whereas virtual ops are actions that the blockchain performs, typically delayed actions based on timestamps and other internal criteria.

Once we know about a new op, we can add it to the broadcast_ops.yml file which Jekyll uses to generate:

Test Proof

The rake test:proof task has Jekyll build the site, then run basically a lint check on the site. This allows us to find problems with the site. I also use Site Sucker to further validate the results.

Other Tasks

The other tasks, like rake production:deploy and rake production:rollback are just used to build the site for use on production environments or switch back to the previous version. We need to do this externally instead of relying on Github Pages because there are various non-standard libraries required to build the site and Github doesn't like running non-standard libraries.

We also have scrape tasks that will pull tutorials in from other repositories.


Previous Posts of this Series


Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Hello @inertia

Thanks for the outstanding work you are providing to the Steem Blockchain.

I have staff picked this contribution as it shows some great amount of work done and dedication towards the project.

Keep contributing.

Sorry for the late review.

Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, click here.

Need help? Chat with us on Discord.



Thank you for your review, @ms10398! Keep up the good work!

Hi, @inertia!

You just got a 1.6% upvote from SteemPlus!
To get higher upvotes, earn more SteemPlus Points (SPP). On your Steemit wallet, check your SPP balance and click on "How to earn SPP?" to find out all the ways to earn.
If you're not using SteemPlus yet, please check our last posts in here to see the many ways in which SteemPlus can improve your Steem experience on Steemit and Busy.

Hi @inertia!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your UA account score is currently 6.529 which ranks you at #153 across all Steem accounts.
Your rank has not changed in the last three days.

In our last Algorithmic Curation Round, consisting of 207 contributions, your post is ranked at #138.

Evaluation of your UA score:
  • You've built up a nice network.
  • The readers appreciate your great work!
  • Try to work on user engagement: the more people that interact with you via the comments, the higher your UA score!

Feel free to join our @steem-ua Discord server

Honestly that is all beyond me, I just upvoted to get you to the $10. :-)

This post has been included in today's SOS Daily News - a digest of all you need to know about the State of Steem.

Congratulations @inertia! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :

You got more than 3750 replies. Your next target is to reach 4000 replies.

Click here to view your Board
If you no longer want to receive notifications, reply to this comment with the word STOP

Do not miss the last post from @steemitboard:

SteemWhales has officially moved to SteemitBoard Ranking
SteemitBoard - Witness Update

Support SteemitBoard's project! Vote for its witness and get one more award!

Hey, @inertia!

Thanks for contributing on Utopian.
Congratulations! Your contribution was Staff Picked to receive a maximum vote for the documentation category on Utopian for being of significant value to the project and the open source community.

We’re already looking forward to your next contribution!

Get higher incentives and support!
Simply set as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).

Want to chat? Join us on Discord

Vote for Utopian Witness!