Running Condenser, Jussi and a new service locally + adding feature flags to Condenser

in steemdev •  8 months ago

Flag - Howard Hodgkin

Intro

We're part of the team working on Steemit.com, and wanted to share some of our recent work getting the site ready for what will be a very busy 2018.

Benjamin Chodoroff (github | steemit)
Iain Maitland (github | steemit)

The work discussed here also covers contributions from other steem employees and the amazing community.

Abstract

The frontend for steemit.com is Condenser, a universal React app. This codebase gets updated all the time, with bugfixes and new features. Sometimes, these features are so complex that the team wants to only enable them for a small subset of users to see how they work and to allow users to give us valuable feedback into how the change works or does not work for them.

To do this, we needed to set up a "feature flag" system: for logged-in users, while the site is being rendered, the client asks the server which special features should be turned on, and the client will either enable or disable those features for that particular user.

On the server side, we added some Conveyor API endpoints to let users ask for their feature flags, and let developers or management define new feature flags as well as the probability that a flag is enabled for a user.

On the client side, we created a system where a new feature can be contained in a component and decide whether or not that component gets rendered based on the user's feature flag state, which is inserted in our Redux store when a user starts their session.

Getting all of this written & doing integration tests is a bit outside of our usual workflow, working mostly with Condenser -- we aren't often rolling out new backend API endpoints at the same time the corresponding frontend functionality is being defined. In cases like this, developers need to be able to run a local instance of the backend service as well as Condenser. Sitting in between Condenser and the new backend service is Jussi, our JSON-RPC call router, which can be configured to route most calls to the normal prod services (like steemd) and only route certain calls to our local Conveyor dev server.

Developers working with Condenser will occasionally run into circumstances when they need to run local services with Jussi, so we documented the steps here:

Goals

We're going to get Conveyor working on top of Jussi. This will involve having Condenser, Jussi and Conveyor all running simultaneously on our dev machine.

Jussi is a proxy for many JSON-RPC services related to the Steem blockchain.

Conveyor is a JSON-RPC micro service for Condenser configuration.

Condenser is the Steemit.com react app, with toggle-able components dictated by Conveyor.

Set up Jussi

Clone Jussi and apply this diff - this preps Jussi for local conveyor development:

git apply jussi_local_conveyor_config.diff

There is one environment specific edit that's not covered by this diff.
In PROD_UPSTREAM_CONFIG.json, for the config:

["conveyor","http://192.168.11.3:8090"]

You need to specify an IP that is accessible to the Jussi docker image, i.e. your wireless card's or docker vlan's address.

To discover this, in the terminal run ifconfig, in the case of Docker's Vlan, the IP is identified at dockerO under inet, something like 172.17.0.1.

If you run conveyor with make devserver it will default to port 8090, therefore in this example case we edit PROD_UPSTREAM_CONFIG.json:

["conveyor","http://172.17.0.1:8090"]

Build Jussi

Build Jussi's Docker Image -- this step takes a while!

docker build -t="$USER/jussi:$(git rev-parse --abbrev-ref HEAD)" .

Run Jussi.

docker run -itp 9000:8080 "$USER/jussi:$(git rev-parse --abbrev-ref HEAD)"

Condenser usually claims port 8080, so we ask Docker to make the Jussi container port 8080 to 9000 on the host.

Set up Condenser

Apply this diff to Condenser:

git apply condenser_local_conveyor_config.diff

This diff will tell Condenser to talk to our local Jussi instead of api.steemit.com.

Run condenser

SDC_DATABASE_URL="mysql://root:hunter2@127.0.0.1/steemit_dev" NODE_ENV=development node ./node_modules/babel-cli/bin/babel-node.js  ./webpack/dev-server.js

Your database connection string will probably be a bit different ;)

Set up Conveyor

Conveyor uses the node-config module; per the load order we can create a local.toml config like so:

cat>config/local.toml<<EOF
admin_role = 'foobarman'
rpc_node = 'http://localhost:9000/'
EOF

where foobarman is some existing steem username that you have the private posting key for, and the rpc_node is the api for the steem blockchain we will be authenticating that user against (our local Jussi).

Run Conveyor

make devserver

Set up some feature flags.

All being well, Jussi, Conveyor, and Condenser are all up and running locally. It's time to test out the feature-flags functionality we are interested in.

Download the rpcit command line tool for making rpc requests.

rpcit is a command line tool used to make API requests to a JSON-RPC server like Jussi.

yarn global add rpcit

To make privileged calls, like the ones needed to interact with Conveyor's feature flags endpoints, you must supply a private key & username to be used in signing the requests. You can find your private keys at https://steemit.com/@accountname/permissions - note the 'private' posting key needs to be toggled on to be displayed.

Add a feature flag

RPCIT_KEY=wif RPCIT_ACCOUNT=foobarman rpcit -a http://localhost:9000 -s conveyor.set_feature_flag_probability testflag :0.5

This creates a feature flag called 'testflag' with a 50% probablity that it will resolve to true for a given user.

Test it out

Navigate to the locally running instance of Condenser in your browser. The pertinent call to conveyor happens in the user saga. So to test we need to be a logged in user. As a logged in user, refreshing the page should trigger a call from Condenser to get currently enabled feature flags from Conveyor (via Jussi). We can watch the Conveyor logs in the terminal to determine if this is happening.

    rpc_req: {
      "id": 554497485567665,
      "method": "conveyor.get_feature_flags"
    }

Wrap a Condenser feature in a flag.

Now we can wrap front-end code in the <ConnectedFlag /> component to have it conditionally display depending on what the given flag resolves to. In the event that the flag resolves to false or if it is not found due to some error, a fallback component is rendered.

Demo usage:

// Conditionally render any number of wrapped Children
<ConnectedFlag
    flag="testflag"
    Fallback={<LoadingIndicator/>}
>
    <h1> Hello World </h1>
</ConnectedFlag>

// Explicitly Render a singular component.
<ConnectedFlag
    flag="testflag"
    FlagComponent={<Icon name="user" />}
    Fallback={<LoadingIndicator/>}
/>

// If flag is false or not present, render a fallback
<ConnectedFlag
    flag="testflag"
    FlagComponent={<Icon name="user" />}
    Fallback={<LoadingIndicator/>}
/>

This can be tested by applying this diff:

git apply condenser_demo.diff

and navigating to http://localhost:8080/faq.html where 'Hello World' will appear whenever the flag named testflag evaluates to true (this will be the case for 50% of users based on our RPC command issued above) and a loading indicator will appear when it does not.

Conclusion

Thanks for reading! If you have any questions leave a comment & we'll do our best to respond :)

Benjamin Chodoroff (github | steemit) & Iain Maitland (github | steemit)

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Those news and info about the technologies being used are in sync with what I want to learn (ReactJs) and that I will get involved in Steem development (to learn at the beginning) and implement my own ideas running around my head since months. I'll use this post as guide (Actually not that I think about it would be nice to have a feature in steeemit to bookmark articles if there isn't one already that I don't know of).

·

very sweet you're learing react! what are your ideas for development?

if you're new to javascript and react, following the steps in this post will definitely expose you to a lot of disparate programming concepts -- feel free to ask questions if they come up.

·
·

I'm not new to javascript but to React. My first idea is a small feature to publish to steem the results of my work outs in a given format from an Android app (With a web version to start with).

Nice
Thank you...

Thanks for that piece.

This is really great information @maitland, and more importantly it's great to get some transparency behind the team working on SteemIt.com as well as the rest of the Steem platform.

Currently one of the biggest drawbacks to Steem (in my opinion) is a lack of a clear roadmap / timeline / transparency about the plans for the future and the work being done.

Are you and Ben SteemIt, Inc employees, or are you contractors, or how does that work? I would really appreciate any information you can share on this subject.

Also I think I can speak for the entire Steem community when I say THANK YOU for all of the work you are doing and for making a post like this with great information for other developers to create their own sites and services on top of the Steem blockchain!

·

the team has been busy on a bunch of different projects; for a peek into what might be coming up you can browse https://github.com/steemit/ and take a look at each project's issue queue -- you might find some exciting stuff ;)

the team will be writing more posts like this one to document our work & let everyone know what's going into the codebase's development. thanks for reading!

I wish you best of luck for your efforts

Upvoted ☝ Have a great day!

Excellent review with realistic goals

This is very helpful and so much informative .i appreciate your work.Carry on your activity.
thanks a lot

Hi, with these steps I can setup a different interface for steemit.com?

·

yes -- follow the instructions at https://github.com/steemit/condenser and you can run your own interface to the steem blockchain!

·
·

Thanks man

Congratulations @maitland, this post is the second most rewarded post (based on pending payouts) in the last 12 hours written by a Newbie account holder (accounts that hold between 0.01 and 0.1 Mega Vests). The total number of posts by newbie account holders during this period was 4220 and the total pending payments to posts in this category was $5453.19. To see the full list of highest paid posts across all accounts categories, click here.

If you do not wish to receive these messages in future, please reply stop to this comment.

Congratulations @maitland! You have completed some achievement on Steemit and have been rewarded with new badge(s) :

Award for the number of upvotes

Click on any badge to view your own Board of Honor on SteemitBoard.
For more information about SteemitBoard, click here

If you no longer want to receive notifications, reply to this comment with the word STOP

By upvoting this notification, you can help all Steemit users. Learn how here!

Excellent information

Some excellent and helpful information