Experimenting with SDS api for my historical financial transaction script
Last year around US tax time I worked on a script that would spit out my history of steem transactions in a .CSV file that could be read by crypto tax software. Getting your history of transactions (including things like reward claims) isn't too hard from the API, but the things that made things somewhat tricky were:
- Transactions don't report in terms of Steem Power, they report in terms of Vests, which are an accounting thing entirely internal to the blockchain (they're the mechanism that lets everyone earn interest on SP without the witness nodes needing to adjust the balances of every single SP-holding account on the blockchain every tick). But since they're internal, there are no historical records for the value of a vest relative to STEEM.
- We're all continuously earning interest on our SP, but there are never any individual transactions that say "you earned X SP in interest", because it's all done via the Vest mechanism. But since the blockchain documentation talks in terms of interest it seems like it should be accounted for that way rather than via the "exchanges" between STEEM and Vests that they're implemented as.
With the limited amount of Steem Power that I have it's probably mostly in the rounding-error range anyway, but I figured I should try to do things properly if I could. One way to estimate the historical relationship between Vests and SP is to find the fill_vesting_withdraw operations that implement powerdowns. Since they're all taking from the same internal pool, anybody getting a powerdown payout will tell you what the relationship between vests and STEEM was at that point in time. Unfortunately the normal Steem API doesn't provide an easy way to get these (the history API tells you powerdowns that you've done, but not that other people were doing while you were just earning SP). For my script last year I ended up looking block-by-block to find all of them, but since that's extremely time consuming to do via the API I stored the info I got in a local file. I had thoughts to try to "productize" the script to let anyone be able to run it for their own account, but figuring out a way to maintain the DB of those fill_vesting_withdraw operations seemed like it would be a huge headache, and I figured it wouldn't be worth pursuing once the deadline passed anyway.
This year we've got another tax deadline approaching, so I'm looking at my script again. One change from last year is that @steemchiller's SDS API provides a slightly better way to find virtual operations like fill_vesting_withdraw, so I've been doing some proof-of-concept work to see if that addresses the problem I had last year. At first I thought it would, but it seems like the call balks if you ask it for a range of blocks with more than 250 virtual transactions in it, so my proof-of-concept script is asking for ~100 at a time, which ends up being a lot of API calls to cover the entire range of my history since account creation. The SDS calls are pretty zippy, but there are still so many of them with the approach I'm trying that it's probably not sustainable. I'm not sure how I'm going to proceed yet.
I will think about what would be the best way for solving this. Maybe there is an easier way or it could be solved on SDS level by providing an extra API. As it currently appears to me, I would say creating some kind of DB that stores the rates in a specific interval (maybe hourly or even just daily) is the way to go.
BTW I don't think that anyone will follow your earned rewards inside the blockchain, so I would say as long as it's not getting paid out to an exchange, it's basically in an uncontrollable (too complicated to follow) state.
Or, we could probably solve it with math. Similar to the discussion we had yesterday regarding the last irreversible block. The rate at which the vests to steem increase seems to be a constant rate, So, a bit of complex math and identifying the exact pattern here would again help us out, right? Instead of having to store a ton of data that isn't very frequently accessed, we could calculate it on the go.
Of course, all this is still very theoretical and I still need to see if this is something math can solve. Something for the weekend ;)
Yes, for getting approximates that would work (calculating by blockchain time and expected target rate) but not for 100% accurate rates, because there are some actions that can have an influence on the actual rate (for example, what comes to mind is powering up/down, maybe token burning, which affect the number of total tokens).
Yeah, I don't think it matters too much from a practical perspective, but once I started looking at the problem last year I could see how to do at at an algorithmic level and once I see a problem is solvable it's hard for me to be content just leaving it in a mysterious fog state.
That would be cool, and potentially useful for other history-oriented projects.