In my previous blog, Project S - Day 3, I talked about making our primitive back-end send Push Notifications to our App.
Now that we got our basics sorted out, I'm going to talk about some design concerns we need to think about before we can continue.

Speed

---

Although I find myself sometimes tempted to do it anyway, one of the golden rules of programming is to never prematurely optimize your code. It is good practice though to think ahead about how your code is going to scale up.

Actually, I ran into a performance issue already. The code I wrote printed a few debugging messages to the screen, amoung which was the current block number. After some hours, I noticed my processing was lagging behind on the blockchain. About two hours to be precise. That's not what we want! After quite some time trying to figure out why this was, I figured that maybe retrieving the current block number (a function provided by the steem-python library) during streaming of posts, might trigger a delay when resuming the streaming. I removed the block number from the debug code and that seems to have done the trick! Now, there might be other issues, for example:

We will need to figure out a way to detect when we are lagging to much and catch up again. I believe this is going to be the most difficult part of the whole server component.

Persistence

---

We would need some kind of persistence of user data. There's a lot of options for that, ranging from super fast search databases like Elasticsearch and Solr to full-fledged relational databases like PostgreSQL and MySQL. For this app, however, I choose to use to use Redis. Redis is not only a database, but also a cache and so called message broker. Above all, it's super fast because it runs in-memory (this also has quite some disadvantages, but most of these can be mitigated or are not applicable to our application).

SPAM

---

When watching my small Python back-end parse the posts on the blockchain, I noticed there was posted some spam on a regular basis. We need to think of some way to filter this out, but I really want to stay far far away from any form of sensorship. I will need some more time to think this through. Downvotes might come in too late, because the post has already been processed. I'm thinking some user-configurable filter might be the way to go.

Interconnecting components

---

We could put all components of our back-end into one and be done with it, but this doesn't scale well if the number of users starts to grow (hopefully). To separate some logic and prevent creating a performance issue, we will create different components that are connected through Redis. We can have one component process the blockchain as it comes in, and another component provide our app with data. A third will take care of sending the push notifications to our app. We can make some kind of queue in Redis that components can fill with tasks, and other components can pick up when they are ready to execute that task. This way we can add more workers that process tasks as we scale up.

---

NEXT UP: Persisting information in a datastore

(I'll be going away with the family for the weekend, so it might be that the next blog will be postponed until Monday)

---

If you support my efforts, please vote, resteem, follow and/or leave your thoughts in the comments.

Any help is greatly appreciated! Thanks for reading and STEEM on!