#5 - Indexer: cached_post.py and feed_cache.py
What I am learning about Hivemind's design, as I am working on Native Ads
This post gives a technical overview of how these two modules operate within Hivemind. They are found in the
hive/indexer subfolder of the codebase.
This module hosts the
CachedPost class, which contains various methods used to manipulate the
hive_posts_cache table in the database and maintain data consistency between the blockchain and Hivemind. The following are just some of the operations handled by methods in this class (for sake of brevity):
- insert, update and delete posts
- update vote counts
- do child recounts
_sql() method generates the various SQL edit statements that will be used to update the database. The SQL statements are generated according to the "update level" triggered by the individual calls. For example, new post additions (inserts) are handled differently from upvote updates. Extra logic handles recounts that also affect parent posts.
Methods in this module are called from a number of modules, under various circumstances.
CachedPost.vote()is called from
blocks.pywhen vote operations are detected within raw blocks
CachedPost.update_promoted_amount()is called from
payments.pyto record a new "promoted" balance" in the database for particular posts
CachedPost.vote()is also called from
payments.pyto trigger an update in vote stats
- Various methods such as
update()are called from
posts.pywhich handles core post operations and data. For more info, refer to the post I wrote on this module: #4 - Indexer: posts.py
sync.pyalso makes various calls to methods in the
CachedPostclass. They concern the recovery of missing posts and payout backlogs.
This module hosts the
FeedCache class, which has various methods that help maintain the
hive_feed_cache table in the database. This table effectively creates "views" of all accounts' feeds on a rolling basis and helps avail a quick and resource-friendly way to query feeds.
It has 3 simple class methods:
insert(): inserts posts and reblogs as entries associated with an account ID
delete(): removes a post entry from the table and therefore from the feed cache
rebuild(): rebuilds the feed cache when initial sync is complete. This method is called from
hive/indexer/sync.pyscript in the
initial()method handling the initial sync
What have I learned?
When implementing Native Ads, the
hive_posts_cache table will hold JSON data that contains ad properties within every ad post's
json_metadata field. This cache will make data retrieval easy and efficient for an ad's properties.
The feed cache, on the other hand, might be relevant to Native Ads, when one considers how ads can be integrated into feeds. This is an aspect that I am still thinking about and no definitive conclusions have been reached yet. There are also other considerations to make like handling legacy post promotions in the same way, or the effects on user experience flexibility that this move could introduce.
Another way is to allow different user interfaces (front-ends) to determine how and where they place the ads, their frequency and timing. Again, I am still weighing the pros and cons.
Studying these modules also led me to look at
payments.py and how it is designed to track payments to
@null with post URLs in the memos for legacy post promotion. Some modifications will need to be made there to accommodate ad payments made to community accounts or to
@null, but that's a discussion for another post.
Posts in this series
I am currently working on a new feature called Native Ads, that may be added to Hivemind Communities in a future update.
For an overview of the Native Ads feature and how it will work, read this doc.
If you would like to take a look at the code, check out my fork of Hivemind on GitHub. The project is still in pre-alpha and is dependent on the full release of Hivemind Communities, so I haven't started publishing code yet.