steemblog -- browsing your steem posts faster

in utopian-io •  12 days ago 

steemblog -- mirror your steem posts to blog hosts such as GitHub pages, Netlify, etc., and help you organize and search your posts faster.

  • This is my first submission for @utopian-io project
  • 本文用于首次尝试向 utopian-io 提交开源项目

Repository

steemblog

blog.png
Image Source: Pixabay

Introduction

steemblog is a blog service and tool built by myself that synchronize your steem blogs into GitHub pages or other static site hosts, e.g. https://steemblog.github.io/@utopian-io/,
https://steemblog.github.io/@robertyan/

The blog service has recently served around 20 users mainly in Steem Chinese community who have provided very positive feedback to the steemblog project.

Features

  1. Convert all the posts of one or multiple steem accounts with Hexo into static pages, and publish them onto GitHub page or Netlify;
  2. Provide smooth user experience to browse, classify and organize one's blogs with the blog themes;
  3. Make the posts updated daily with Travis CI or cron jobs to keep the blogs synchronized from Steem.

1. Example

By using steemblog, we could easily mirror @utopian-io's posts into https://steemblog.github.io/@utopian-io/, where you could look at @utopian-io's posts in the history, and search through all the posts, with better experience.

Comparing to Steemit, this looks to be a "real blog" and is cleaner and more responsive.

(1) The home page of the blog

image.png

(2) Tag cloud:

tag-cloud

(3) Archives:

archives

(4) Recent posts:

recent-posts

(5) Search:

search

(6) Post: https://steemblog.github.io/@utopian-io/what-is-utopian-state-of-the-art-forever-stored-in-the-blockchain-107-days-since-the-beginning/

image.png

screenshots from: https://steemblog.github.io/@utopian-io/

2. How to Use

To deploy the project for yourself, you could fork https://github.com/steemblog/blog to your account, and setup the blog with travis-ci, by updating the environment viariables in travis project with your accounts.

  • GIT_USERNAME: your GitHub account
  • GIT_EMAIL: your GitHub account
  • GITHUB_PAT: your GitHub token
  • BLOG_REPO: the repo to deploy your GitHub pages
  • STEEM_ACCOUNTS: the steem accounts to synchronize the posts, separate by comma. e.g. utopian-io,robertyan

Or you can also contact me (@robertyan) to help setup one synchronized blog for your steem account at https://steemblog.github.io, e.g. https://steemblog.github.io/@robertyan/

Technology Stack

This is a Python + JavaScript project.

  1. steemblog extended and enhanced the steem blog mirroring tool project by myself
  2. The interaction with Steem blockchain is built with beem project.
  3. The blog generation is built with hexo blog framework.
  4. The blog service by default used an customized hexo theme: steemblog/hexo-theme-icarus. The theme has good user experience, but the build speed is terrible. We did intensive enhancements to hexo-theme-icarus to increase the build speed by 20~100 times.
  5. invoke for task management.

1. Build the Blog

To build the blog, we need to construct the front-matter for markdown, by retrieving data from steem, and then use Hexo to generate the static pages.

blog/message.py: the front matter template

MESSAGES["blog"] = """
---
title: '{title}'
permlink: {permlink}
catalog: true
toc_nav_num: true
toc: true
position: {position}
date: {date}
categories:
- {category}
tags:
{tags}
thumbnail: {thumbnail}
sidebar:
    right:
        sticky: true
widgets:
    -
        type: toc
        position: right
---

{body}
"""

blog/builder.py: fetch metadata from steem

    def _write_content(self, post):
        folder = self._get_content_folder()
        c = SteemComment(comment=post)

        # retrieve necessary data from steem
        title = post.title.replace("'", "''")
        permlink = post["permlink"]
        body = c.get_compatible_markdown()
        position = self._get_position(body)
        date_str = post.json()["created"]
        date = date_str.replace('T', ' ')
        tags = "\n".join(["- {}".format(tag) for tag in c.get_tags()])
        category = c.get_tags()[0]
        thumbnail = c.get_pic_url() or ''
        url = c.get_url()

        # build content with template
        template = get_message("blog", footer=True)
        content = template.format(title=title, permlink=permlink,
                                  position=position, date=date,
                                  tags=tags, category=category,
                                  thumbnail=thumbnail, body=body, url=url)


blog/command.py: build static pages with hexo

@task(help={
      'debug': 'enable the debug mode',
      })
def build(ctx, debug=False):
    """ build the static pages from steem posts """

    configure()
    os.system("hexo clean")
    build_cmd = "hexo generate"
    if debug:
        build_cmd += " --debug"
    os.system(build_cmd)

2. Make Build Faster

However, the builds for some accounts with 1000+ posts and 200+ tags are extremely to build (take > 1 hour). In order to serve more accounts, we have to increase the build speed.

We used below strategies to accelerate the build process by 20~100x.

  1. build a customized theme: steemblog/hexo-theme-icarus
  2. use cache in rendering intensively
  3. implement incremental build in hexo
  4. put widgets into standalone html pages and load with ajax
  5. use more efficient layout (timeline)
  6. reduce total page numbers

themes/icarus/layout/layout.ejs: Use fragment cache as much as possible:

    <%- partial('common/footer', {}, {cache: true}) %>
    <%- partial('common/scripts', {}, {cache: true}) %>

    <% if (has_config('search.type')) { %>
    <%- partial('search/' + get_config('search.type'), {}, {cache: true}) %>
    <% } %>

themes/icarus/layout/layout.ejs: Generate widget page with hexo generators:

/**
 * Widget page generator
 */
module.exports = function (hexo) {
  hexo.extend.generator.register('widget', function (locals) {
    const widgets = hexo.extend.helper.get('get_config').bind(this)('widgets');
    const component_widgets = widgets.filter((w) => (w.component))

    return component_widgets.map(function(widget){
      return {
        path: `widgets/${widget.type}.html`,
        layout: 'component/pjax_widget_src',
        data: {
          widget: widget,
          __widget: true
        }
      };
    });
  });
}

themes/icarus/layout/component/pjax_widget_ref.ejs Use reference to page instead build the page itself, which is critical to make incremental build possible.

<div class="card widget">
  <div class="card-content">
    <div id="widget-<%= widget.type %>" data-pjax="<%= `${get_config("root")}widgets/${widget.type}` %>.html" style="position: relative; width: 100%; display: block;"></div>
  </div>
</div>

themes/icarus/includes/generators/category.js: only the affected category pages will be built in hexo generators

function needs_update(category) {
    if (config.incremental) {
        // in incremental mode, update the affected category pages only
        const updated_categories = list_updated_categories();
        if (updated_categories &amp;&amp; updated_categories.length > 0 &amp;&amp;
            updated_categories.indexOf(category['name']) != -1) {
            return true;
        }
        return false;
    }
    return true;
}

return locals.categories.reduce(function(result, category){
    if (! needs_update(category)) {
        return result;
    }

    const posts = category.posts.sort('-date');
    const data = pagination(category.path, posts, {
        perPage: perPage,
        layout: ['category', 'archive', 'index'],
        format: paginationDir + '/%d/',
        data: {
            category: category.name,
            parents: findParent(category)
        }
    });

    return result.concat(data);
}, []);

We cannot show all the details of the implementation here, but how to build the blog from steem data, and how to make the build faster are the key efforts here to build a robust and efficient blog service.

Roadmap

While the steemblog service is easy to use for some users, it requires more enhancements.

  1. Switch languages and themes easily;
  2. Improve search performance; add search functionality by adding Google / Baidu search;
  3. Support write back to Steem blockchain when neeeded.
  4. Connect with steem-engine / SCOT, and help more Hexo users to post on steem
  5. Promote the project to help more people who wants a better blog user experience.

How to Contribute

For anyone who wants to contribute to this project, feel free to fork https://github.com/steemblog/blog or
https://github.com/steemblog/hexo-theme-icarus , and submit Pull Requests.

Or please directly contact me (@robertyan) on Steem if you have any questions to discuss.

Github Account

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Thank you very much for your contribution. This is an excellent tool that I have been looking for for a while. I remember @dapeng has developed something similar, but he is a big fan of R.

Could you add me to the travis before I forget, so that I have my blog, if I understand it correctly?

Here are my a few cents of comments:

  1. you might provide a fall-over mechanism if current Node is offline.
  2. maybe we don't need the active key for just reading data.
  3. there are a few classes in collector.py. It is better to only define a class in one file
  4. the classes at blog/steem are very general-purpose, you might consider add this to beem as well :)
  5. Unit tests are missing
  6. For SEO purposes, you might want to add the canonical links.

Your contribution has been evaluated according to Utopian policies and guidelines, as well as a predefined set of questions pertaining to the category.

To view those questions and the relevant answers related to your post, click here.


Need help? Chat with us on Discord.

[utopian-moderator]

·

thanks @justty (行长),

yes. @dapeng developed one R package or command for building blog with hugo I think, e.g. https://steemit.netlify.com/ . Comparing to @dapeng 's version, this one might provide a bit more advantage in user experience and more optimization in build efficiency to make the service scalable and open to everyone.

I have added you into steemblog. Your home page is: https://steemblog.github.io/@justyy/

Your kind comments are very practical and helpful. Thanks a lot. Some feedback here:

  1. I have setup a backup with Netlify actually, but GitHub pages fit the scenarios better. We may open another subdomain on github.io as a backup site.
  2. we actually didn't use any steem keys to fetch the data from steem. (the blockchain data are open :)) The file settings.py was copied from a previous project, but we didn't do code cleanup. We may need the posting keys later if we want to enable steemblog to write back into steem blockchain, and we can remove the active keys.
  3. Umm... I also thought about that earlier. I can do some refactor, or even generalize them and merge them into beem project.
  4. sure. I can refactor that module and send to holger for review whether it's necessary to include part of them into beem.
  5. sure. We will add unit tests.
  6. cool. I'll put that into the future plan.

The above suggestions are logged as issues in the steemblog/blog project in order to track and schedule.

·

Thank you for your review, @justyy! Keep up the good work!

Thanks being awesome NBC holder! Your 36.68 NBC earned you 6% team-cn upvotes!

你那里天气如何?想要参加活动但是不知道从何开始?关注寻宝团@cn-activity每日整理社区活动!如果不想再收到我的留言,请回复“取消”。

先来赚赚审查 XD

·

哈哈,谢谢村长来赚

Hi @robertyan!

Your post was upvoted by @steem-ua, new Steem dApp, using UserAuthority for algorithmic post curation!
Your post is eligible for our upvote, thanks to our collaboration with @utopian-io!
Feel free to join our @steem-ua Discord server

这个萝卜好粗

Posted using Partiko Android

·

机机也很粗啊。。。

Hey, @robertyan!

Thanks for contributing on Utopian.
We’re already looking forward to your next contribution!

Get higher incentives and support Utopian.io!
Simply set @utopian.pay as a 5% (or higher) payout beneficiary on your contribution post (via SteemPlus or Steeditor).

Want to chat? Join us on Discord https://discord.gg/h52nFrV.

Vote for Utopian Witness!

萝卜进军utopian了!加油!

Posted using Partiko iOS

·

谢谢小冰,主要想了解下utopian-io社区的情况,感觉目前并不是很活跃的样子。。。