Programming Diary #20: Accelerating organic activity

remlaps (74)Verified Developerin Steem Dev • 4 months ago (edited)

Summary

This post introduces the new browser extension that I'm working on, the Steem Conversation Accelerator (SCA). The goal for this tool is to increase the speed and engagement for organic conversations on the Steem blockchain by notifying the browser operator when conversations are taking place.

The post also contains reflections upon the Steem development ecosystem, suggesting that there's a hierarchy of needs for development efforts and there's also a substantial opportunity for developers to make a difference.

Background

It's hard to believe, but it's been 3 months since I posted Programming Diary #19: Follower network strength and preparing for Open Source publication. I've been working on programming whenever time allowed, but didn't have much to say in the diary format.

During the intervening time, however, I did manage to open up two github repos for browser extensions, and I started on a third. Here are the two repos that I opened up:

The Steem Follower Checker; and
The Steem Curation Extension

And here are brief descriptions of the purpose of all three.

Steem Follower Checker

The Steem Follower Checker lets the browser operator check a "follower score' that gives some sort of estimate about the strength of a Steem user's follower network. It's described in more detail here, here, and here. I didn't post about it, but I also released the new version of the algorithm on June 30^th, as expected. The next update is targeted for September 30.

In the June 30 update, I made it harder for an author to get a top-score by averaging the number of followers over the age of the account. This leveled things off a lot, but in the September update, I want to correct for accounts that were influential in the past, but have gone dormant.

The Steem Curation Extension

This is basically unmodified from when I first wrote it a couple years ago, but I think it is useful, so I went ahead and published the repo so that anyone can make use of it. I posted about it here. The extension modifies the appearance of condenser sites like steemit.com and steemitdev.com so that posts with null beneficiaries set and posts which have been promoted by sending SBDs to @null get some visual highlighting.

The Steem Conversation Accelerator (SCA)

I'm now working on a 3rd browser extension, which I intend to publish as open source when it is ready. The intention of this browser extension is to increase the level of organic conversation on the Steem blockchain by delivering browser notifications when a conversation of interest is happening. This will be the main topic for the rest of my diary post.

Eventually, "conversations of interest" will probably include the following:

Posts, comments, and replies by/to a followed account
Comments and replies on a thread where the browser operator participated
Posts, comments and replies in communities that the browser operator subscribes to.

One of the things that I have noticed about X (Twitter) is that many roads on the Internet lead back there. If I read my usual blogs, the authors post links/embeds from Twitter, so when I want more information and click on a link - there I am back on Twitter again. It's like Twitter is a magnet for my attention.

With browser notifications, I'm imagining a similar effect. I can be on the Internet, reading my favorite blogs when a browser alert pops up in a sidebar telling me about some activity here on Steem that interests me. Then, at my leisure, I can click on the notification and find out more about what was going on. As long as I'm active in my browser, there's always a force that can draw my attention back towards the blockchain.

I have two overarching purposes in mind for this:

Accelerate the time between comment and reply when organic conversations are happening on the blockchain.
This returns to a theme from a previous programming diary post that developers should be building bridges between Steem Island and the rest of the Internet.

Overview of the Steem Conversation Accelerator (SCA) extension

While the eventual goal is to provide notifications for multiple sorts of social interactions, step 1 is a simple (or not) attempt to deliver notifications when a followed account makes a post or comment, or when a followed account receives a reply. And for a JavaScript novice, this has already been fairly challenging, even with the help of ChatGPT and Claude Sonnet (for the record, I think Claude Sonnet is presently a more capable JavaScript developer's assistant than ChatGPT). Here's how it's currently structured and partially functional at the moment.

1. The popup window

When the browser extension loads or gets clicked on, it prompts for the information that it needs. This includes:

The polling interval: how often to check for updates
The Steem user account: Whose followers do I want to receive updates from.
The API endpoint
The website to land on when links are clicked.

Here's what it looks like:

Crucially, I'm not mucking around with anyone's keys or passwords. The browser extension is just an information provider, and I'm letting the web servers handle the key security.

The background script

This is a script that sets various alarms and event handlers in order to check for follower updates on the Steem blockchain. It is not connected to any particular browser tab, and it handles things like:

Collecting the follower list
Checking the last update times for each follower using the last_post time from the condenser_api.get_accounts API call.
Delivering browser notifications when activity happens

This script is set-up so that it only runs when the browser is active, and goes to sleep when the browser is idle.

The activity summary page

This opens as a regular browser tab and it uses the condenser_api.get_account_history call to get the details of posts, comments, and replies that happened during the last polling interval.

At present, it looks like this:

Obviously, both of the HTML pages still need plenty of TLC, but for now, I'm just interested in observing that the data is available. 😉

The challenges

Back to the future

I had a rudimentary version of the program working with Manifest V2 and my Brave browser, but then when I tested it in Microsoft Edge, the browser warned me that Manifest V2 will be deprecated, so I decided to update to Manifest V3. That, basically, broke everything, so I've spent the last couple of weekends rebuilding.

As of today, I have it sort-of working, but with plenty of bugs that I still need to iron out.

Race conditions

Javascript race conditions

The background script and the script that displays the activity in HTML format both need to access and update the same data structures. If they do this out of order, it leads to unexpected results. I'm still working out the logic for locks and semaphores so that the results are consistent.

Steem race conditions

The last_post value seems to be maintained in real time by all of the API nodes, but the account_history call falls behind. I've seen it as far behind as ten minutes during my testing. Thus, sometimes when the program notices that an account's last_post has been updated, the corresponding account_history details are not yet available. Right now, I'm just throwing those misses out. The plan is to start tracking last updated and last displayed for each account's activities so that I can catch up when the data becomes available.

Also, the last_post tells me the time of an author's last post or comment, but it doesn't tell me the time of their most recent reply, so the extension is certainly missing some replies.

Performance

I use the @steemcurator01 account for much of my testing, because their followed accounts have plenty of activity. However, updating the activity page for a busy account like that takes an intolerably long period of time.

I'm not far enough along to focus on this yet, but much improvement is definitely needed.

General challenges with JavaScript and asynchronous execution

It seems that with Manifest V3, functions in the background script are executing concurrently in ways that I would not expect and did not observe with Manifest V2.

Also, I spent a great deal of time debugging a problem where the background script was running fine with the Dev Console, but terminating unexpectedly if there was no console running. I was stumped by this, and so were ChatGPT and Google Gemini. Eventually, Claude Sonnet helped me figure out that Manifest V3 had introduced a timeout because the script was taking too long to execute. I worked around that problem, but I still don't totally understand the resolution, so I'm not totally sure that I won't start seeing it again.

Next up

My next focus will be on cleaning up the race conditions, data consistency, and HTML formatting in this program. It's easy to say it in a sentence, but I think that will take a substantial amount of calendar time, since I can only work on it during nights and weekends.

Reflections

On LLMs and coding

Right or wrong, I definitely have a perceived level of quality among the three LLMs that I've been using. It seems to me that Claude Sonnet has been the most useful, ChatGPT was second, and Google Gemini comes in a distant third.

There are some things that they all have in common, though.

The longer you stay in a conversation, the worse they get. At first, I can mostly copy/paste the output that they produce, and it just works. As the conversation continues, however, I have to start making modifications to their suggestions and the LLM loses track of the context. This results in a continuing cycle of increasing modifications and decreasing code relevance from the LLM. Eventually, I wind up just scrapping the conversation and starting over.
As a result of this, I think there's going to be a coevolution of coding quality and prompt engineering skill. It has long been known that it is useful to break programming problems into modular tasks, but it has been easy for the programmer to take shortcuts and skip over that. Now, however, the degrading quality of conversation with LLMs adds another reason why the programmer will want to get really good at modularizing the design.
The LLMs all give you some sort of disclaimer along the lines of "caveat emptor: LLMs make mistakes, so check it yourself, too." Of course, if you take this literally, it eliminates the entire benefit of the LLM. If I have to verify every line of code, than I'm not really saving any time by using the LLM. Rather than verifying the code line by line, I think the key is to do adequate testing. Also, I suspect that the early days of compilers were similar. It predates me, but I'd be willing to bet that users of the first assemblers and compilers went through and validated the machine code. At some point, it became unrealistic to even think about doing that on all but the simplest programs. I suspect that LLMs as developers' assistants will follow a similar path.

On Steem velocity

One of the things that I've noticed while working on this browser extension is that there is a lot of activity going on which I had never before observed. Twitter solves this problem by including replies in the feed. Steem also needs solutions to this. This browser extension might be one solution, but I'd encourage people to look for others, too.

On Steem development hierarchy

Most likely, you're aware of Maslow's Hierarchy of Needs. This is the concept that a person needs to meet their basic needs: physical and safety needs - before they can pursue things at the higher level like belonging, esteem, intellect, and so on...

Similarly, I think the Steem development ecosystem probably also has some sort of hierarchy of needs. In particular, I think that most developers should focus on solving simple problems at the social media layer (and making money by doing it). Only when the basic tiers are saturated would it make sense for freelance developers to move up the hierarchy. At a basic level, the developers should start with one of three goals:

Help social media users have more fun or be more productive
Help social media users and investors earn higher returns
Help developers to generate funding or be more productive

Over the years, I have seen many developers come in with grand plans, then turn around and leave. I can only assume that they frequently left because their product ideas didn't get traction, and I think that probably happened because the lower layers of the hierarchy were not yet in place. I guess what I'm saying is really just the old cliché, "Don't boil the ocean."

What I imagine, instead is a lot of people trying out little "disposable" ideas, then combining and expanding the ideas that succeed. There are plenty of underexplored/underutilized revenue strategies that developers can take advantage of:

Sell your app or fund it with advertising via the Play store, Windows Store, Android Store, etc...
Subscription model: Imagine a browser extension or web site that provides certain information to the user by reading custom_json data on the blockchain. However, the custom_json data only gets stored on the blockchain if someone pays a subscription fee.
Reward models: The application is set up so that the ~~author~~ developer can get a share of beneficiary rewards or generate curation rewards from the people who use it.
Crowd Funding: Developers can seek contributions from Steem community members, in the form of direct transfers or delegations. At one time @utopian-io had a crowdfunding system operating for the Steem blockchain, but I guess it went away after the Witness Wars. I suspect there is a niche available for a team to recreate something similar. Here is a description of their platform, from 2017. And trustworthy project approval in a new generation could be managed using something like the multisig wizard from @rexthetech.
And of course, there's always the Steem Proposal System (SPS), although I would argue that the SPS should be reserved for things at the very top of this hypothetical Steem development hierarchy.

Conclusion

Since I can only spend a small portion of my time on nights and weekends, my development activity continues to be slow and haphazard, but I'm also slowly illuminating some guiding principles. They are:

Build bridges between Steem Island and the rest of the Internet
Decentralization means that things should operate on the end-device and they should be able to tolerate downtime from one or more API endpoint(s)
Work on tasks of manageable sizes
Work on tasks that don't require the user to trust me with their keys or account authorizations

I am far from expert at any of this, but I'm willing to put in the time and effort that it takes to learn what is needed at the time that it's needed. I'd invite others to start simple and do the same.

Also, if you know JavaScript, HTML, and/or CSS; think that the SCA sounds like a useful browser extension; and want to contribute; feel free to let me know. I am open to inviting people with the relevant skills to work on the repo before it's ready to be set to public.