Dtube Community Support - Change the stream method and replays the blockchain
Repository
https://github.com/evildido/dtube-community-support
Introduction
Dtube is a video plateform based on the steem blockchain that store videos on IPFS. Because most of users doesn't use ipfs. Dtube is a central point that will store content on IPFS node.
Dtube community support is a bot that will "listen" for new dtube content for a specific steem tag and store the video on a local IPFS node. Plus it stores metadata about dtube content like title (steem post) / video size / permlink / author. The main goal is to persist dtube content of your favorite community and or authors on the IPFS network.
DCS will retrieve 480p videos. if not available, it will retrieve the source file.
New Features
Fetch the blockchain block by block
As previously said in my last contribution, I have modified the streamOps function by fetching blocks instead of using streamOperation of steem-js.
In the previous version, streamOps() was implemented in index.js. This function had two roles :
- stream the blockchain
- search dtube content
To enhance visibility, I have splited it in two functions :
- stream the blockchain : catchup() in utils/utils.js
- search for dtube content and pin it on the ipfs node : streamOps() in actions/stream.js
Stream
catchup() uses LightRPC to fetch blocks.
LightRPC is a minimal library for interacting with JSON RPC.
Take a quick look on the code (complete code : utils/utils.js#L89)
function catchup(blockNumber) {
lightrpc.sendAsync('get_ops_in_block', [blockNumber, false]).then(ops => {
if (!ops.length) {
// Blocks does not exists or is empty
// If does not exist -> restart catchup() with the same block number
// If exists -> restart() with blockup number + 1
} else {
// Block exists and not empty
// Save blocknumber in DB (to resume)
saveBlockState(blockNumber);
// Process the block(operations) by calling streamOps()
stream.streamOps(ops);
// retart the catchup() with the next block
return catchup(blockNumber + 1);
}
}).catch(err => {
// try another node and restart the function
});
};
If a block is not empty, catchup() calls streamOps().
Search dtube content and pin it
streamOps() the file has not been modified except for quick fix (will be talked later) but streamOperations() has been removed.
Replay the blockchain
As we previously saw, the catchup() function takes block number in argument. I’ve added the possibility to start the bot by specified the first block to fetch instead of the last one.
The bot is able to replay partially the blockchain.
To do that, we need to specify block number by using '-b' argument.
npm start -- b=21803547
The following script point on start() in index.js.
Resume the bot
In the same logic, we are able to save the last block number so we can restart the bot to the last block.
When starting the bot (npm start), there are three cases :
- No block number specified in argument and no block number saved : The bot will fetch the block number with get_dynamic_global_properties
- No block number specified in argument but a block number has been saved : The bot starts from the last block saved
- Block is specified in argument : The blot starts from this block
Take a quick look
function start() {
if(args.blockNumber!=undefined) {
// A block number passed thought argument
utils.catchup(Number(args.blockNumber));
}
else
{
var state = db.getAsync("block_state");
state.then(result=> {
// Start from the last block number stored in JFS DB
utils.catchup(Number(result.blockNumber));
}).catch(err => {
// Start to the last block
lightrpc.sendAsync('get_dynamic_global_properties', []).then(result => {
utils.catchup(result.head_block_number);
}).catch(err => {
// retry with another node
failover();
return start();
})
});
});
}
}
Before stopped the bot
And after restart, the bot resumes from the last processed block.
Bug fix
Disable logs archiving
Because an issue on Winstan-dail-rotate (issue 125), old zipped log file cannot be deleted during the rotation. I've disabled this options (commit 6d1f5b8 and fea7e32).
fix "check if enought space"
In the last relase, I've added a test to ckeck if the IPFS repository has enough space before pinning a new content. This test consists by comparing content size and repository usage PLUS content size being pinned.
'size_tmp' is used to handle content being pinned. In the release, this variable was not properly updated. Commit 0855211 fix that.
Retrieve created date
Dtube Community Support stores dtube content metadata. "Created" stores the publication date.
By using streamOperations, the publication date given by the system (date()).
Because the bot doesn't use StreamOperations anymore and can replay the blockchain, I've added the commit 21953d2.
The commit consists to fetch the the created date by using steem.api.getContent().
Next step
With the ability to replay the blockchain, the bot t is facing another problem. If we replays too faar, we can try to pin content which has no seed. In that case, the bot continues to run but gets stuck with instances of streamOps.
I have asked here how to handle this point. dht.findprovs(hash) seems to do the job. findprovs finds peer for a specific pinset (hash).
I've tried it on my a ipfs node (go-ipfs).
With an old dtube content (no seed).
With a recent dtube content.
As we can see, findprovs() should do the job. The next step will be to use it for automatic and manual pinning.
Thanks for the contribution!
I really like the way you structured your post and explained everything, was very interesting to read! However, it wasn't that clear to me which parts were refactored code and which parts were new code, so it would be great if that's made a bit clearer for future contributions (it could also just be me being sleepy!).
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Hi Evildido, do you have Discord? I wanted to talk with you regarding your witness efforts and simply getting to know you since we are both witnesses.
Yes no problem. You can contact me on @evildido#8230.
Did that!
Hey @evildido
Thanks for contributing on Utopian.
We're already looking forward to your next contribution!
Contributing on Utopian
Learn how to contribute on our website or by watching this tutorial on Youtube.
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!