In this post I want to give an #steemdev overview of the architecture and the implementation of #steembay.

The idea for #steembay was initially introduced to me by @pollux.one at the end of 2016 after we joined Steemfest in Amsterdam. Because of a lack of time it took until October 2017 till I started to make myself comfortable with developing bots for steemit using the steemJS API.

The Requirements

In first place the requirements sound very simple. Let's make a bot to support auctions on the steem blockchain by using the given infrastructure of posts and comments. Ok, that's easy ;-)
But as an software developer with decades of experience I know that building a software solution is never that easy as it sounded.

The Detailed Requirements

Diving a little bit deeper into the requirements the following main tasks had figured out.

1. Identify a post as an auction
2. Give the sellers and bidders some advice about how to work
3. Allow the seller to start the auction and set a starting price
4. Place Bids by steemians
5. Indentify and notify the highest bidder
6. End the auction automatically if the cashout time of the auction post has reached

Soon we recognized that there are some more features we have to support to give the seller and the bidders a better control about their actions.

7. The seller needs the possibility to end an auction before the running time of the auction has finished
8. The seller needs the possibility to cancel an running auction
9. The bidder must have the chance to revoke a bid for example if he made an typo
10. Support other languages than English (Multi-Language)

The Architecture of #steembay

Now it was time to think about how to implement #steembay.
Some rules that I have imposed on myself were
- to keep the technical dependencies as low as possible (e.g. using special databases, etc.)
- to separate the concerns to improve maintainability and testability
- to start as simple as possible but to take care of the expandability and scalability

And this it the outcome of my considerations:

Queue(s)

Queues are used to exchange data between the different bots. Every entry in a queue consists of a single file in a specified folder of the file system. Using the file system is a simple and reliable approach and avoids special dependencies.

Auction Repository

The auction repository stores a complete auction post with it's replies using the file system (one file per auction). It is important to know that this storage is just used as a cache to speed up processing. All information stored in this caching files can be restored from the blockchain at any time.

Blockchain Listener Bot

The Blockchain-Listener-Bot listens the blockchain for new steembay posts and comments (steem.api.streamOperations). The posts and comments that are important for steembay can be uniquely identified by checking if Tag1 is equal to steembay. Every detected item is written to the queue for further processing by the Auction-Processor-Bot.

Auction Processor Bot

The Auction-Processor-Bot is the heart of steembay. Every time when it receives a new item from the queue it loads the latest auction data from the blockchain, determines the current state of the auction and triggers the creation of confirmation comments on demand.

The inner logic of analyzing and processing an auction is extracted as an particular component. It gets an JSON object as input (the auction post) and returns an object that includes all further actions that have to be done (e.g. Create auction opened comment, etc.). This facilitates the decoupled testing of this component with Unit Tests to reach a higher level of correctness and quality. So everytime something was changed in this component (Bugfix, new feature) the correct functioning can be easily tested in a wink.

Create Comments Bot

The Create-Comment-Bot is responsible for the creation of the confirmation comments. It decides which steembay-account has to be used to create the comment regarding to the priority of the comment and the restriction that a singel account can create only one comment every 20 seconds.

Create Daily Report Bot

The Create-Daily-Report-Bot selects all matching auctions and creates the daily report post of steembay.

Monitor Auctions bot

The Monitor-Auction-Bot monitors the auctions to determine which ones have exceeded the cashout time. This is done by evaluating all the auctions of the auction repository.

The Challenges of steembay in Detail

Challenge 1: Load a post including all comments

The steemJS API doesn't provide a method to load a post with all it's replies (comments). So you have to do this by a recursive call of 'steem.api.getContentRepliesAsync()'. On a post that consists of many replies this could be very time consuming because you have to make a request for every child reply recursively. If you may know a better solution for loading a post with all of it's comments, please let me know :-)

Challenge 2: Confirming analyzed comments

Every analyzed and processed comment that is important for #steembay gets confirmed with a reply of the steembay bot. This has several advantages.

1. First is that the user gets feedback that it's action was successfully processed.
2. The original values of the analyzed user comment are stored within the confirmation. If the user changes it's comment afterwards, for example to change the value of the bid, this has no effect to the running auction.
3. It's easy for the auction processor to identify already processed comments. Therefore the bot uses the JSON metadata of the confirmation comment:
   3.1. Tag2 to store the type of the recognized user action ('auction-opened', 'bid-accepted', 'bid-rejected', etc.)
   3.2. Tag3 to store details to Tag2 (e.g. 'rejected-amount-tolow', 'rejected-cashouttime-exceeded', etc.)
   3-3. 'data' object to store the values of the action (e.g. 'author', 'amount', etc.)

Challenge 3: All auction data have to be stored on the blockchain, avoid external databases

One important requirement was that all auction data have to be stored exclusively on the blockchain. But we decided to use the file system as a simple storage to cache the current processing state of each auction. The reasons for this decision are the following:

• To have a fast (offline) access to all running and finished auctions (see Daily-Report-Bot)
• To facilitate the contemporary detection of finished auctions (see Monitor-Auction-Bot)

Challenge 4: Multi-Language Support

Because the community of non-English speaking users on steemit grows, we decided in December 2017 to add a multi-language support to #steembay. And again, the decision to store user action informations inside Tag2, Tag3 and the data-object turned out to be very beneficial. It enables us to put any text to the comments we like. Always when a new confirmation comment has to be created, we can use the translated text of our internal localization table matching the users language. For further processing the text of the comment is completely irrelevant for the bots because that exclusively work with the metadata of the comments.
Many thanks for the great support of several steemians on translating the texts!

Questions?

Now I'm at the end of my tour through the technical inside of steembay. I hope this information was valuable and interesting for you. Feel free to write some comments or ask some questions about the implementation of steembay.

And the last word is for my loyal followers. Maybe you now understand why I made so few posts in November and December 2017 ;-D

@schererf