Steem Sincerity - Update and Community Involvement

in steemdev •  7 months ago

Sincerity Project Update

Over the last week I've been integrating some of the data from the SteemPlus crowdsourcing process to expand the amount of training data I can use to teach the machine learning algorithm how to distinguish spammers and bots from human content creators.

I'm fairly new to machine learning, and when somebody (sorry I can't remember who) mentioned random forest classifiers a couple of weeks ago, I decided to investigate them more. Following that, I have also now changed part of the classifier from using a nearest neighbours algorithm to a random forest classifier.

In terms of how well the software predicts the training data (cross-validation), these factors have resulted in a slight improvement in accuracy. The 'false positives' have also been reduced, meaning that fewer non-spammers should be wrongly labelled as spammers in SteemPlus and other services using the Sincerity spam API. Obviously people have different ideas of what a spammer is though, so this tool can at best reflect some kind of community average.

I am also now collecting extra data which will be ready to incorporate at the end of the month, which seem likely to further improve accuracy.

By that time, I would also like to have collected even more crowdsourced data for adding to the training set. I am very happy with the software, but the training data could still be better.

The trouble with allowing people to anonymously report spammers is that some people seem to use it as a way to try and remove content that they just don't like or understand. For example, many non-english language accounts were labelled incorrectly as spammers, as were some popular youtubers who have recently joined the platform, perhaps with divisive content.

I am thinking about better ways to collect crowdsourced data, feel free to let me know if you can help, or have any ideas. If you are in a position to delegate some SP for me to upvote comments with useful training data, that could be very useful.

Current Training Data

Whilst I can't reveal full details about the classification algorithm for a couple of reasons, the training data I used is shown here (and will be kept up to date when retraining happens). I don't have time to check all these accounts myself, but if you do, and find any inaccuracies, that is very helpful information for improving the classifier. One incorrectly classified account here could affect the software classification of many accounts!

Pending changes (changes I plan before the next software training)

Thanks to the validation by @fraenk, I've updated the changes list.

To be removed from all lists:

arcange
dailytop10open
jehovahwitness
new-york
altobot
austrobot
dailypick

To relabel as bots:

coin.info
followforupvotes
tts

New APIs Methods Coming Soon

Project Sincerity isn't just about spam and bot classification though. More of the large amount of data being collected will soon be available in the form of new APIs which relate to characteristics of voting, commenting, etc.

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

I am going through some of the "human"-classified accounts to check whether those contain some potentially misleading data-sets. I'll collect the results in comments below:

DISCLAIMER: the following interpretations are only MY subjective opinion, nothing else.
P.S.: this starts looking a bit spammy in and of itself, sorry, did'n expect this to be so many so instantly...
P.P.S.: I also started looking at the "spammer" classified training data, lot's of humans and bots in there (imho)... the training data seems to me like it could do with a much more thourough vetting process!

·

andybets - (&steemreports) has the right classification as human, but to be honest, you should probably remove your own account from the training data and see how your own AI ranks yourself, just to get a first-hand feel for it...

I would remove this account from the training-set to avoid any subjective in-house-biasing

·

Thanks for this. It's very helpful, I'll review your suggested changes.

·

I agree with all the changes you suggested, and have adjusted the data sets. :)

·
·

Awesome, I am glad I could help out!

As mentioned above, I believe there's also quite some "false positives" under the training-spammers, too... I'll go through some more of those when I find the time.

·
·
·

That'd be great. I have just sent you 2 SBD as a small thanks.

·

dailytop10open - might be operated by a human, but the pattern looks more like a bot-classification to me, very repetitive content and the comments are primarily "functional"

I would reclassify as bot or just remove it from the training set to avoid ambiguity

·

jehovahwitness - ok, i might be biased, but looking at their comments close up reveals the same set of a dozen or so "inspirational" comments being repeated over and over, I think it's questionable if this is actually a human and it may even be seen as spam by some.

i would remove this from the training data due to it's ambiguity

·

new-york - I think this is without a doubt spam, and probably bot-spam! The same identical "promotion" comment for a "resteem-service" is being posted over and over and over

I would reclassify this as spam

·

austrobot - self-proclaimed trailing bot that posts manual content (?)

I would remove this from the training data due to ambiguity

·

altobot - a self-proclaimed bot posting "manual" reports, probably it should be seen as more bot than human?!

I would remove this from the training data due to it's high ambiguity

·

dailypick - curation service, might be manual, could be automated, repetitive comments look very bot-like

I would either reclassify this as a bot or remove from the training data to avoid ambiguity.

·

coin.info - definitely a bot, has no original content leaves comments notifying of crypto rates of coins mentioned in the original posts

I would reclassify this as a bot

·

followforupvotes - self-proclaimed voting bot random voting it's followers and leaving repetitive comments and posts. No question this is a bot

I would reclassify this as a bot

More of the large amount of data being collected will soon be available in the form of new APIs which relate to characteristics of voting, commenting, etc.

You have me more than curious! Keep up the great work, your tools are already essential to "screening" suspicious activities.

I commend your progress on making this community more transparent!

This is really a great project. A good classification is so hard to achieve, even for humans.

Thank you so much from removing me from the lists.

I'm stoked to see its progress!

So, Skynet next? :)
@tipu upvote this post with 0.5 sbd

·

Hey @cardboard, I love smartness's @tipu! ;)
I just joined it: thank you for this innovative service!

Is there an irony that your account shows 0.7% spammer and 0.2% bot? ;)

·

It's gone down to 0.6% spammer now because I've been trying not to spam! ;)

It's awesome to see this make progress and improve in accuracy ...

It still leaves me with some worries when I check the current Top-Spammers according to the sincerity API:

While some of these accounts are in fact leaving very repetitive comments that may well be seen as spam... they are certainly lacking the volume to be in the ranks of "top-spammers".

At least that's my subjective interpretation of how to define spam... quantity does play a major role here.

Taking into account that there are accounts like @a-0-0 leaving 27k comments in the same timeframe, I think something should be done on that aspect.

OR, if the API purely want's to classify, it maybe just shouldn't publish a "ranking"?!

·

It's a fair point. I will also add a list of accounts sorted by the most comments made.

I guess that because accounts like this which already have a negative rep, probably aren't interfering with most people's experiences anymore, so aren't being reported as spammers by the community. This software is increasingly using a community average of spammer as its classification definition.

·
·

Great!

This software is increasingly using a community average of spammer as its classification definition.

I think that's exactly what we need and I am stoked to see this increasing in "accuracy" of reflecting that.

The classification score for those "top-ranked" spammers does not feel inaccurate to me to be honest, but calling those the top-spammers is taking the result a bit out of context imho.