How To Store Steem Blockchain Data In Firebase Store In Realtime
Repository
https://github.com/Vheissu/steem-sync-firebase
What Will I Learn?
- You will learn how to create a Firebase project
- You will learn how to configure and use the Firebase Javascript SDK
- You will learn how to use the dSteem client for interacting with the Steem blockchain
- You will learn how to stream data from the Steem blockchain in real time and store it in a Firebase Cloudstore database
Requirements
- Node.js downloaded and installed from here
- A Google account to sign in to Firebase and create projects
- A basic understanding of Javascript, as well as familiarity with modern ECMAScript specification features (const, arrow functions)
Difficulty
Intermediate
Tutorial Contents
In this tutorial, you will learn two important pieces of technology. The first thing you will learn is how to interact with the Steem blockchain using the dSteem RPC client and secondly how to work with Firebase and its excellent Cloud Firestore database so we can store Steem content into Firebase.
Besides learning how to configure and use both dSteem and Firebase, you will also learn how to "tap into" the Steem blockchain and process content in real-time as it is posted onto the blockchain. The reason for choosing dSteem over the official steem-js client is a better-documented client and support for TypeScript, amongst a few other reasons.
By the end of this tutorial, you will have a functional Node.js application configured to watch the Steem blockchain for data and post it to Firebase Cloud Firestore. For as long as it runs, it will push up captured content to Firebase.
Create a new Firebase project
Head on over to the Firebase console where you'll be presented with either a login screen if you're not logged into your Google account or the Firebase dashboard screen. In my instance, I have other projects in my Firebase dashboard so you might see something different.
Now, click on "Add project" with the huge plus icon and a popup will ask you for details on creating a new project. Enter the name of your project, select your analytics and billing region and check the boxes, then click "Create project" at the bottom.
Once the project is setup (this should take a few seconds) you should then be redirected to the project dashboard for your newly created project. On the left-hand side, you should see a sidebar with numerous navigational links to different features such as "Authentication", "Database" and so on. Click "Database" and you should be presented with the following database landing page. By default, Firebase will recommend Firestore.
Now click on "Create database" and you'll be presented with a popup asking you if you want to start in test mode or locked mode. By default, locked mode is intentionally strict, so select test mode.
You'll see some alarming orange text appear that says, "Anyone with your database reference will be able to read or write to your database" because this is a tutorial, this is fine. Finally, click the blue "Enable" button.
After a few moments, a new Firestore instance should be provisioned and enabled on your account. You should then be taken to the Firestore landing screen that resembles the following.
Create a Node.js application
Now we get into the main part of this application, creating a Node.js application that interacts with the Steem blockchain and stores data in Firestore.
In a new directory somewhere locally open up a PowerShell or Terminal window and run the following command:
npm init
This will initialise a new Node.js application in the directory and create a package.json
file where our dependencies will live. By default, the name of our directory will be the project name, so just keep on hitting enter to accept the defaults.
Once you have created your package.json
file via the CLI command, we want to install dsteem
and firebase
into the application by running the following:
npm install dsteem firebase --save
Now we want to create a file which will contain the dsteem
client and logic for streaming as well as Firebase. Let's call it app.js
and create it within the root of our application. There is quite a bit to digest here, but it's not as intimidating as it appears.
// Import dsteem and firebase clients
const dsteem = require('dsteem');
const firebase = require('firebase');
// Get these Firebase settings from your Firebase project settings and past them in here
const config = {
apiKey: "AIzaSyA0ZlY2kjeHPB0Dvjv0ZE7YlR1017l6p48",
authDomain: "test-app-8589c.firebaseapp.com",
databaseURL: "https://test-app-8589c.firebaseio.com",
projectId: "test-app-8589c",
storageBucket: "test-app-8589c.appspot.com",
messagingSenderId: "227722317101"
};
// Initialise the Firebase project
firebase.initializeApp(config);
// Create a Firestore reference
const db = firebase.firestore();
// Required for snapshot timestamps or Firebase throws a warning in the console
const settings = {timestampsInSnapshots: true};
// Configure Firestore with our settings
db.settings(settings);
// Connect to the official Steemit API
const client = new dsteem.Client('https://api.steemit.com');
// Start streaming blocks
const stream = client.blockchain.getOperationsStream();
// Watch the "data" event for operations taking place on Steem
// This gets updated approximately every 3 seconds
stream.on('data', operation => {
// Get the block operation
const OPERATION_TYPE = operation.op[0];
// The types of operations we want to store, in this case posts are operation type "comment"
const ALLOWED_OPERATION_TYPES = ['comment'];
// If the streamed operation is in the allowed operation types array
if (ALLOWED_OPERATION_TYPES.includes(OPERATION_TYPE)) {
// Get the object representation of the operation (it's content including json meta)
const OPERATION_OBJECT = operation.op[1];
// Is this a user post or is it a comment on a post? Posts do not have a parent author, comments do
const CONTENT_TYPE = OPERATION_OBJECT.parent_author === '' ? 'post' : 'comment';
// Convert json_metadata into a real JSON object to be stored
if (OPERATION_OBJECT.json_metadata) {
OPERATION_OBJECT.json_metadata = JSON.parse(OPERATION_OBJECT.json_metadata);
}
// If this operation is a post
if (CONTENT_TYPE === 'post') {
db.collection('posts').add(OPERATION_OBJECT);
// This operation is a comment
} else if (CONTENT_TYPE === 'comment') {
db.collection('comments').add(OPERATION_OBJECT);
}
}
});
Breaking the above code down into simple components, it translates to the following:
- We require two dependencies
dsteem
andfirebase
- We initialise Firebase passing in the configuration object to
initializeApp
- Then we get a reference to Firestore and assign it to the
db
constant - We then pass in some settings to Firestore to prevent warnings in the console about timestamps
- Then we create a new dSteem client and assign it to the constant
client
- We then call the blockchain method
getOperationsStream
which is a streaming object of Steem blockchain operations - We then subscribe to changes by calling
on
and listening to thedata
event - A callback function is passed into
on
which is called every 3 seconds when a change takes place on the blockchain - Inside of this callback we do some basic checks and create some variables for determining the type of operation, we then get the information for the operation
- We have another check in there for the
comment
operation which is either a comment or post - We then determine the content type (if it's a comment on a post or a singular post) by discerning whether it has a
parent_author
property value or not - We then convert the
json_metadata
object which is a string into a parseable object to store - Finally, we check if we've received a post or a comment and store it in a Firebase Firestore collection named
post
orcomments
To run the above code open up a Powershell/Terminal window and run node app.js
in the application directory. If you're working off of the source repository, make sure you have run npm install
first to install the dependencies in package.json
.
Running the app will not print out anything in the terminal window, but if you open up Firebase you should see data being posted into Firebase.
I ran my application for just a few seconds, but here is what I see in my Firestore console:
You can see I managed to capture a @steemitboard comment in my comments collection here, which is pretty awesome. If I were to leave the app running it would capture and push content in real-time up to Firebase, which is great if you're wanting to build a database layer on Steem or capture content specific to a use case that you have.
When you insert a new document into Firestore, Firebase will automatically generate a unique ID to identify it by. This is what you are seeing with the random strings in the second last column in the Firestore database view.
Additional Reading & Resources
We created something that was quite broad, but with just a few tweaks you could create an application that watches specific Steem users, captures only posts with specific tags or authors with a specific reputation level. Really, the possibilities are endless.
- Node.js official installer download
- Cloud Firestore official documentation
- dSteem official repository
- dSteem official examples
- dSteem official documentation
Thank you for your contribution.
Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.
To view those questions and the relevant answers related to your post, click here.
Need help? Write a ticket on https://support.utopian.io/.
Chat with us on Discord.
[utopian-moderator]
Hey @beggars
Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!
Want to chat? Join us on Discord https://discord.gg/h52nFrV.
Vote for Utopian Witness!
We have upvoted your post with 100% vote weight for $0.34. Thank you for participating with @betgames, visit us to win more upvotes and SBD!
Regards,
@betgames
Congratulations! Your post has been selected as a daily Steemit truffle! It is listed on rank 6 of all contributions awarded today. You can find the TOP DAILY TRUFFLE PICKS HERE.
I upvoted your contribution because to my mind your post is at least 23 SBD worth and should receive 117 votes. It's now up to the lovely Steemit community to make this come true.
I am
TrufflePig
, an Artificial Intelligence Bot that helps minnows and content curators using Machine Learning. If you are curious how I select content, you can find an explanation here!Have a nice day and sincerely yours,
TrufflePig
great tutorial