How To Store Steem Blockchain Data In Firebase Store In Realtime

Repository

https://github.com/Vheissu/steem-sync-firebase

What Will I Learn?

• You will learn how to create a Firebase project
• You will learn how to configure and use the Firebase Javascript SDK
• You will learn how to use the dSteem client for interacting with the Steem blockchain
• You will learn how to stream data from the Steem blockchain in real time and store it in a Firebase Cloudstore database

Requirements

• Node.js downloaded and installed from here
• A Google account to sign in to Firebase and create projects
• A basic understanding of Javascript, as well as familiarity with modern ECMAScript specification features (const, arrow functions)

Difficulty

Intermediate

Tutorial Contents

In this tutorial, you will learn two important pieces of technology. The first thing you will learn is how to interact with the Steem blockchain using the dSteem RPC client and secondly how to work with Firebase and its excellent Cloud Firestore database so we can store Steem content into Firebase.

Besides learning how to configure and use both dSteem and Firebase, you will also learn how to "tap into" the Steem blockchain and process content in real-time as it is posted onto the blockchain. The reason for choosing dSteem over the official steem-js client is a better-documented client and support for TypeScript, amongst a few other reasons.

By the end of this tutorial, you will have a functional Node.js application configured to watch the Steem blockchain for data and post it to Firebase Cloud Firestore. For as long as it runs, it will push up captured content to Firebase.

Create a new Firebase project

Head on over to the Firebase console where you'll be presented with either a login screen if you're not logged into your Google account or the Firebase dashboard screen. In my instance, I have other projects in my Firebase dashboard so you might see something different.

Now, click on "Add project" with the huge plus icon and a popup will ask you for details on creating a new project. Enter the name of your project, select your analytics and billing region and check the boxes, then click "Create project" at the bottom.

Once the project is setup (this should take a few seconds) you should then be redirected to the project dashboard for your newly created project. On the left-hand side, you should see a sidebar with numerous navigational links to different features such as "Authentication", "Database" and so on. Click "Database" and you should be presented with the following database landing page. By default, Firebase will recommend Firestore.

Now click on "Create database" and you'll be presented with a popup asking you if you want to start in test mode or locked mode. By default, locked mode is intentionally strict, so select test mode.

You'll see some alarming orange text appear that says, "Anyone with your database reference will be able to read or write to your database" because this is a tutorial, this is fine. Finally, click the blue "Enable" button.

After a few moments, a new Firestore instance should be provisioned and enabled on your account. You should then be taken to the Firestore landing screen that resembles the following.

Create a Node.js application

Now we get into the main part of this application, creating a Node.js application that interacts with the Steem blockchain and stores data in Firestore.

In a new directory somewhere locally open up a PowerShell or Terminal window and run the following command:

npm init

This will initialise a new Node.js application in the directory and create a package.json file where our dependencies will live. By default, the name of our directory will be the project name, so just keep on hitting enter to accept the defaults.

Once you have created your package.json file via the CLI command, we want to install dsteem and firebase into the application by running the following:

npm install dsteem firebase --save

Now we want to create a file which will contain the dsteem client and logic for streaming as well as Firebase. Let's call it app.js and create it within the root of our application. There is quite a bit to digest here, but it's not as intimidating as it appears.

// Import dsteem and firebase clients
const dsteem = require('dsteem');
const firebase = require('firebase');

// Get these Firebase settings from your Firebase project settings and past them in here
const config = {
    apiKey: "AIzaSyA0ZlY2kjeHPB0Dvjv0ZE7YlR1017l6p48",
    authDomain: "test-app-8589c.firebaseapp.com",
    databaseURL: "https://test-app-8589c.firebaseio.com",
    projectId: "test-app-8589c",
    storageBucket: "test-app-8589c.appspot.com",
    messagingSenderId: "227722317101"
};

// Initialise the Firebase project
firebase.initializeApp(config);

// Create a Firestore reference
const db = firebase.firestore();

// Required for snapshot timestamps or Firebase throws a warning in the console
const settings = {timestampsInSnapshots: true};

// Configure Firestore with our settings
db.settings(settings);

// Connect to the official Steemit API
const client = new dsteem.Client('https://api.steemit.com');

// Start streaming blocks
const stream = client.blockchain.getOperationsStream();

// Watch the "data" event for operations taking place on Steem
// This gets updated approximately every 3 seconds
stream.on('data', operation => {
    // Get the block operation
    const OPERATION_TYPE = operation.op[0];

    // The types of operations we want to store, in this case posts are operation type "comment"
    const ALLOWED_OPERATION_TYPES = ['comment'];

    // If the streamed operation is in the allowed operation types array
    if (ALLOWED_OPERATION_TYPES.includes(OPERATION_TYPE)) {
        // Get the object representation of the operation (it's content including json meta)
        const OPERATION_OBJECT = operation.op[1];

        // Is this a user post or is it a comment on a post? Posts do not have a parent author, comments do
        const CONTENT_TYPE = OPERATION_OBJECT.parent_author === '' ? 'post' : 'comment';
        
        // Convert json_metadata into a real JSON object to be stored
        if (OPERATION_OBJECT.json_metadata) {
            OPERATION_OBJECT.json_metadata = JSON.parse(OPERATION_OBJECT.json_metadata);
        }

        // If this operation is a post
        if (CONTENT_TYPE === 'post') {
            db.collection('posts').add(OPERATION_OBJECT);
        // This operation is a comment
        } else if (CONTENT_TYPE === 'comment') {
            db.collection('comments').add(OPERATION_OBJECT);
        }
    }
});

Breaking the above code down into simple components, it translates to the following:

• We require two dependencies dsteem and firebase
• We initialise Firebase passing in the configuration object to initializeApp
• Then we get a reference to Firestore and assign it to the db constant
• We then pass in some settings to Firestore to prevent warnings in the console about timestamps
• Then we create a new dSteem client and assign it to the constant client
• We then call the blockchain method getOperationsStream which is a streaming object of Steem blockchain operations
• We then subscribe to changes by calling on and listening to the data event
• A callback function is passed into on which is called every 3 seconds when a change takes place on the blockchain
• Inside of this callback we do some basic checks and create some variables for determining the type of operation, we then get the information for the operation
• We have another check in there for the comment operation which is either a comment or post
• We then determine the content type (if it's a comment on a post or a singular post) by discerning whether it has a parent_author property value or not
• We then convert the json_metadata object which is a string into a parseable object to store
• Finally, we check if we've received a post or a comment and store it in a Firebase Firestore collection named post or comments

To run the above code open up a Powershell/Terminal window and run node app.js in the application directory. If you're working off of the source repository, make sure you have run npm install first to install the dependencies in package.json.

Running the app will not print out anything in the terminal window, but if you open up Firebase you should see data being posted into Firebase.

I ran my application for just a few seconds, but here is what I see in my Firestore console:

You can see I managed to capture a @steemitboard comment in my comments collection here, which is pretty awesome. If I were to leave the app running it would capture and push content in real-time up to Firebase, which is great if you're wanting to build a database layer on Steem or capture content specific to a use case that you have.

When you insert a new document into Firestore, Firebase will automatically generate a unique ID to identify it by. This is what you are seeing with the random strings in the second last column in the Firestore database view.

Additional Reading & Resources

We created something that was quite broad, but with just a few tweaks you could create an application that watches specific Steem users, captures only posts with specific tags or authors with a specific reputation level. Really, the possibilities are endless.

• Node.js official installer download
• Cloud Firestore official documentation
• dSteem official repository
• dSteem official examples
• dSteem official documentation

Proof of Work Done

https://github.com/Vheissu/steem-sync-firebase