Steemit.com Developer Bounty - Advanced Search - $300 SBD + STEEM/SBD Author Payout From This Post

in #bounty8 years ago (edited)

In order to encourage more community involvement with Steemit.com development, I have worked with the Steemit development team and @pharesim (one of the top 19 witnesses) to organize a bounty for adding an "Advanced Search" feature to Steemit.com. This bounty is to develop the back-end search engine (using ElasticSearch) that will support the addition of this feature to the website.

The deliverable for this bounty is a daemon service, which will continuously monitor the blockchain and keep an ElasticSearch database up-to-date.

Technical Requirements:

  1. Import all blocks from steemd (except the most recent N blocks) into ElasticSearch using nodejs.
  2. The block import should be a single command that will start from scratch or catch up if restarted.
  3. Any third party libraries will need to be approved in advance by the official Steemit development team.
  4. The application should match the coding style of Steemit.com as much as possible.
  5. The service must be compatible with the Steemit.com environment, and be able to be installed on the Steemit.com servers.
  6. The updates to the ElasticSearch database should be done in a reasonably efficient manner.
  7. It should be resilient and easy to deploy.

The ElasticSearch database must support the following queries:

  • Full-text search queries of post text
  • Query by author
  • Query by tag
  • Query by date/time range that the post was created

These are not 'required' but should be supported if possible:

  • Query by number of upvotes (range)
  • Query by number of comments (range)
  • Query by the number of followers an author has (range)
  • Query by the reputation of the author (range)
  • Query by pending payout amount (range)
  • Custom sort orders (newest vs. oldest, relevance, highest payout vs. lowest payout, etc.)
  • The ability to query the ElasticSearch database for all the data elements needed to render a "preview post" - Title, thumbnail, preview text, author name, author reputation, category, post time, etc.
  • If payouts are to be indexed, that part of the data does not necessarily need to be kept up-to-date in real-time. The active 24 hour posts could be re-indexed every 15-30 minutes, and the 30 day posts could be re-indexed every 1-2 hours.

Bounty Requirements:

  • In order to receive the bounty, the developer must implement the ElasticSearch database described above, which meets all of the requirements listed in this post.
  • The application must be accepted by the official Steemit, Inc. development team and pass all internal tests and code reviews.
  • The bounty must be claimed by January 31, 2017. If the application is not completed by this time, the bounty will be voided. An extension may be provided if reasonable progress can be demonstrated, although this must be discussed with me and approved by me ahead of time if it is needed.

Technicalities:

  • The bounty payment will be $300 SBD + the STEEM/SBD portion of the author payout from this post.
  • After the payment period for this post closes, I will be exchanging any STEEM from the payout to SBD, using the internal exchange. I will make note of the final bounty amount in the comments after the exchange is complete.
  • If any other users in the community add to the bounty, I cannot guarantee that part of the payment unless they provide me with the SBD to hold and distribute as part of the official bounty reward.
  • It is the developer's responsibility to ensure that the code meets all of the Steemit dev team's requirements, and make whatever changes are necessary in order for the application to be accepted and published on the live site. I have no control over what the Steemit development team will accept.
  • It is the developer's responsibility to ensure that all the requirements in this post are met.
  • If you are unsure about anything, please ask.

Final Suggestions:

  • It is highly recommended that if you are planning to pursue this bounty, you let me know in advance.
  • If you have any questions, you can reply to this post or contact me in Steemit.chat.
  • You should seek reviews from the Steemit dev team frequently throughout the process.

If anyone would like to add to the bounty, please indicate so in the comments below. I can hold and distribute any bounty funds as part of the official award, or provide you with a notification if/when the bounty is satisfied. Any funds sent to me would be returned if the bounty is not met.


A huge thank you to @pharesim for contributing the $300 SBD to make this bounty possible! Also thank you to the Steemit developers who took the time to help me put all this together so that we can make this bounty a success!


Remember to vote for witnesses here:
https://steemit.com/~witnesses

Sort:  

Now even the basic search doesn't work.

I no longer use Steemit/Steem. I am on Hive and peakd.com.

Might talk to @kaptainkrayola - I think he did most of this already at steemshovel.com or could do so pretty easily with what he already has. I know as funds dried up he likely stopped working on the project.

I'd happily share any code, elasticSearch Queries, whatever. I'm not all that interested in working on the Steemit.com codebase though after having gone down that road. I'll sign into steemit.chat though and we can talk more.

Ok, cool! I'll reply once I get home tonight.

I will try to track him down on Steemit chat. Thanks!

I have a slack that is used by him and some other friends. I added a link to this post in a #steemit channel on our slack too... so hopefully he'll see it.

Cool, thanks :)

could be like an extensions to @arcange 's steemsql ? :) He already has the ingestion in place...

Using the JDBC plugin for logstash -> https://www.elastic.co/guide/en/logstash/5.0/plugins-inputs-jdbc.html#_usage to hookup his mysql storage of the blockchain?

Hey @roelandp - thanks for the suggestion. I'll review it with the Steemit devs and see what they have to say.

Thrilled to see this bounty posted. I've just sent you 100 STEEM to add to the reward in whatever way makes the most sense.

Awesome, thanks @mada!! Do you mind if I convert it to SBD on the exchange? The bounty will likely be open for a while, so that will be the best way to protect against inflation + price fluctuation.

Sure thing - I hereby officially defer to your judgement on how best to use this bounty reward contribution: )

Done :) I have converted it to 11.422 SBD, which will be added to the official bounty. Thanks!!

The Total Bounty Amount is: $329.532

300.00 (@pharesim)
+11.422 (@mada)
+18.110 (author reward)

I love this idea. Which we had a bounty platform to manage all of this

Yeah, me too! I tried pushing for it a few weeks ago, but it doesn't seem to be going anywhere.
https://steemit.com/bounties/@timcliff/proposal-for-a-bounty-management-forum

I suggest that using ElasticSearch is overkill for the moment considering the fact that the blockchain is only ~4GB. Even at 10x the size, that easily fits in system RAM on a modest computer.

I'm working on a project right now that takes the blockchain contents and jams it into a standard vanilla SQL database; this will easily provide full-text search for the foreseeable future (until the blockchain gets too big to fit into memory). A distributed search cluster is overkill for the moment.

Hi @sneak - Thanks for the feedback! Based on what you are saying, it sounds like you do not need any assistance from the community on this item. Is that correct?

I think that's probably true, yeah. We'll be open-sourcing all of our services.

This post has been linked to from another place on Steem.

Learn more about and upvote to support linkback bot v0.5. Flag this comment if you don't want the bot to continue posting linkbacks for your posts.

Built by @ontofractal

This post has been ranked within the top 25 most undervalued posts in the first half of Nov 22. We estimate that this post is undervalued by $8.64 as compared to a scenario in which every voter had an equal say.

See the full rankings and details in The Daily Tribune: Nov 22 - Part I. You can also read about some of our methodology, data analysis and technical details in our initial post.

If you are the author and would prefer not to receive these comments, simply reply "Stop" to this comment.

I hope we will see a successful fundraising here with upvotes, too.
This would be an awesome feature without depending a 3rd party (Google).

Coin Marketplace

STEEM 0.18
TRX 0.13
JST 0.028
BTC 57688.87
ETH 3100.54
USDT 1.00
SBD 2.37