Introducing utopian_data - Store public Utopian data on your local Mongodb

utopian_data is a CLI application which fetches public data of Utopian into your Mongodb instance.

Supported collections:

• moderators (api/moderators)
• sponsors (api/sponsors)
• posts ( api/posts)

Why

I have been working on a separate statistics application (private) for a while to analyze Utopian data better. However, using a REST API for statistics is not practical.

You need a fast system where you can make complex queries. Syncing Utopian posts' data into your system (you can set a crontab and sync it in every 5 minutes) is much better.

Example Mongodb aggreegation Queries

---

Total Contribution Count

db.posts.find({}).count()

Top 3 moderators on blog category (moderation count)

db.posts.aggregate(
  [
    {"$match": {"json_metadata.type": "blog"}},
    {"$group": {_id: "$moderator", count: {$sum : 1}}},
    {"$sort": {"count": -1}},
    {"$limit": 3},
  ]
)

Top 3 Contributors

db.posts.aggregate(
  [
    {"$group": {_id: "$author", count: {$sum : 1}}},
    {"$sort": {"count": -1}},
    {"$limit": 3},
  ]
)

And more... You can pretty much filter/group everything you want once
you learn how Mongodb aggregation queries work.

Installation and Usage

You need to install and configure golang first. After that:

$ go get github.com/emre/utopian_data
$ cd $GOPATH
$ go install github.com/emre/utopian_data
$ cd `$GOPATH`/bin
$  utopian_data --mongodb_uri=localhost

Output

---

Let me know if you encounter any issues.

---

Posted on Utopian.io - Rewarding Open Source Contributors

---