Development Update on my Steem Monsters Bot

in steemmonsters •  6 months ago  (edited)

Ten days ago I released the first version of my Steem Monsters bot into the wild. I've been tweaking and experimenting a lot since then. The current version pushed into CHAMPION III and stayed at the edge of it pretty consistently.

The highest rating it has reached was 3856. At season end it stood at 3711, to which it recovered from 3630. Overall the bot was trending between 3600 and 3800 after the last change.

Initial Rise

The first version of the bot just tried to beat the last five teams of the enemy in the current ruleset. As the current API only allows to get the last 50 battles easily, there are rarely suitable matches. Therefore I had to group similar rulesets. Which helped with some, but rulesets like "Keep Your Distance" are too unique to be grouped together.

This approach, let's call it the "naive bot" was sufficient to reach Gold III and sometimes Gold II. Then I put some money on the table and upgraded some of my splinters to (almost) max level. Including Fire, Earth, Death, Dragon and Normal Splinter and some strong units from the other Splinters (e.g. Angel of Life). The new team, along with a Breadth-First Tree Initialization of depth 3 allowed the bot to rise to Gold II and stay there consistently. But for some reason, it couldn't progress beyond that point. Win streaks were followed by long losing streaks and prevented progress. The "naive initialization bot" needed an improvement.

One Step Up

As I've explained in my last post, at the core of my bot I use a Monte Carlo Tree Search (MCTS). This algorithm allows to search large state spaces and finding local or global optima within them without looking at every single combination. In the naive initialization bot, the algorithm was only used for my team. In the next version, I applied the MCTS to both my team and the enemy team. The MCTS tries to beat my last 5 teams with the enemy monster collection and does the same for the last 5 enemy teams with my collection. This created two teams, one from the enemy collection, and another from my collection. Afterwards, I set the algorithm to find a team which can defeat both. The Versus Bot was born. This version left the gold league behind and set its aim at the Diamond League. It crushed the 2400 rank within minutes and quickly thereafter followed 2500, 2600 and 2800. But that's where the Versus Bot got stuck. Barely getting its feet wet in the Diamond League. Never advancing further than 2900, sometimes falling back all the way to Gold I. Although it quickly recovered back into Diamond III a new approach was needed to advance further.

Learn from your mistakes

The salvation for the versus bot was a weak "learning" approach. I wouldn't really call it learning, but it sort of is a very naive and weak learning approach. The bot stores both its own past team selections as well as the enemy teams. Not player specific, but in general. They get ranked per ruleset and splinter with their win/loss count. At the start of each Monte Carlo Tree Search, the bot does the usual Breath First Tree Initialization and afterwards does a couple of "Depth-First" initializations with the last 5 winning teams of each splinter for the ruleset. In total 90 leads for promising teams.

This new approach was doing good at first but seemed to taper off the longer the bot ran. The mistake was pretty obvious, I introduced a limit of 50 teams, but new teams were added to the end. This caused the bot to only remember teams with multiple wins (which rarely happens) and old teams which were added at the beginning. One quick rewrite later new teams were added at the start of those lists. When sorting this kept the strong teams still in front but propagated the new teams at the front of teams with equal win/loss counts. Making them "fresher" in memory. Old teams are forgotten and pushed off the list. With this change, the bot showed a much stronger play and generated weak teams much more rarely.

With this approach, the bot left Diamond III and II behind and fought its path through Diamond I, reaching the Champion League. That's where it got into trouble again. It pushed up to a rating of 3856 but stumbled back to the Edge of the Diamond League and stayed there. Walking in and out of Champion III. That was just a couple of hours ago. Now I need to think up new improvements.

Future Work

With the current state of the bot, I suspect I need more drastic measures to advance. For now, I'll experiment with some sort of team decay. Making the bot forget teams after a sufficient number of new teams were added. This could lead to the bot improving its generated teams step by step. My thought process behind that is as follows: If a team stays long at the top it influences each team generation. If too many successful teams stay there, the bot becomes predictable and it notices way too late if teams are losing consistently. If the bot starts to forget them, let's say after 10 new teams were added those new teams are already influenced by this old team, and should, in theory, be better adapted to the current opponent situation.

Aside from that an easy quick fix would be to store teams player specific. If the bot encounters an opponent again this gives the bot a good idea which teams it will encounter.

Now onto more drastic measures. One further way of improving would be guiding the MCTS with a classification approach. For example a Random Forest or Neural Network which decides to reward, penalize or prune parts of the game tree. This would guide the MCTS to more efficient teams more quickly. The main issue with this approach is, that I'd need to lower the iterations of the MCTS because the new approach would take longer. On one hand, this can lead to improvements, but on the other, as seen in Googles Alpha Go it can also wrongly guide the search and overlook better solutions. Surely, this can lead to an overall improvement but that is something I need to evaluate before I speculate further. I can also go down a purer Reinforcement Learning approach or combine both approaches. I'll look into them over the coming days.

Final Words

I'd like to thank you for staying/reading until this point. As my bot has been massively more successful I than I initially expected, I'd like to share some of its League rewards below. The first comment gets Daria Dragon Scale and a Black Dragon. All the other cards get distributed randomly to the next commenters in sequence. Just comment below and include your Steem Monsters username so that I can send them to you. I'll wait with the random distribution until at least ten people have commented. Or until tomorrow evening. Then I'll distribute the cards to the people who have commented until then.

First Commenter Prize
Daria Dragonscale lvl 1Black Dragon lvl 1
Random Distribution
Naga Fire Wizard lvl 1Imp Bowman lvl 1Gold Wood Nymph lvl 1Javelin Thrower lvl 2Vampire lvl 3
Daria Dragonscale lvl 1Skeletal Warrior lvl 2Highland Archer lvl 3Rusty Android lvl 3Goblin Mech lvl 3

Authors get paid when people like you upvote their post.
If you enjoyed what you read here, create your account today and start earning FREE STEEM!
Sort Order:  

Woah, nice! Just don’t make it too intelligent! 😉

skynet online

Thanks :) Don't worry, once I implement the self learning part fully it's only a matter of time until it figures out how to tap into the worlds financial resources to destroy every steem monsters player on the planet :D
Are you taking part in the giveaway as well ?

Congratulations @bauloewe! You have completed the following achievement on the Steem blockchain and have been rewarded with new badge(s) :

You received more than 2000 upvotes. Your next target is to reach 3000 upvotes.

You can view your badges on your Steem Board and compare to others on the Steem Ranking
If you no longer want to receive notifications, reply to this comment with the word STOP

Vote for @Steemitboard as a witness to get one more award and increased upvotes!

I am always amazed by such things. I mean, bots who are intelligence. IA is an intriguing domain, and I find it very important to explore, better and deepen their understanding. They can do tasks of crucial importance like in the medical domain, or task of crucial entertainment, like the one you're doing, but either case, the importance is great!

I certainly agree with you on that, the importance of artificial intelligence is not to be underestimated. I don't think the majority has actually grasped what the advances in machine learning and reinforcement learning over the last couple of years actually mean for humankind as a whole. It's not just a bot that is capable of beating a human at go or chess, or solving a rubrics cube the most efficient way. Those are toy problems, these advances work for all kinds of planning and classification tasks. Improving the efficiency of power plants, search and rescue robots, medical screening etc.

I've lost with this bot few times :-)

My playername is @itharagaian
I am amazed by what you did, even if I still consider bot as unfair feature, sorry for that

Wow seems I was first commenter, cool ;)

It seems like you're getting the Black Dragon and Daria :D

But you are certainly right, Bots, if they are good give an advantage to the users. The bot keeps doing its thing while you can work on something else. The curse/blessing of automation.

On the other hand, it's also a lot of work to make a good bot. I think I've worked since June 11. on the bot pretty much every free hour.

I received both card, Thank you VERY much

Great work! Bot building can be challenging and frustrating at times but is very much rewarded when you see the progress your hard work is achieving! Thanks for sharing your results. Interesting as always!

Thanks :) That's certainly true, software development, in general, is just frustrating a.f :P .If you got a self learning component in it that just increases tenfold, because the machine isn't just doing wrong what you've told it, but also the algorithm itself can produce puzzling results:)

Are you taking part in the giveaway as well?

very interesting and loved how you tweeked it. congrats on making it all the way to champions league.

This is @welshstacker acting on behalf of the @ssg-community account. We could always use more cards on that account.

Good luck for next seaon, I hope you make it higher.

Thanks, there's still a lot of work to be done :)

Right now it's playing in Gold I. It's something which I've observed many times, it seems like in the beginning all of the good players fight their way up and the weaker players get pushed further down until the elite has scurried away to champion level and further :D

The later in the season the easier it is to advance :)

Regarding the cards, I assume you're taking part as @ssg-community and not @welshstacker?

good morning @bauloewe.

Yeah @ssg-community needs the cards more than I do. Although I did get matched against you yesterday (welshstacker) and you kicked my ass.

Alright, giveaway is drawing to a close, I'll distribute the cards to:


@lebastion will receive the first commenter price on his account itharagaian

I couldn't find a corresponding Steem Monsters Account for @djennyfloro, so I'm not going to include djennyfloro's account.

For the next giveaway I'm going to look for a transparent giveaway platform, but for this one I'll keep it at a small random script :)

The results will follow below within the next couple minutes

Ah, my bad. I forgot to tell you that I commented because I found your post fascinating, and wasn't going to participate since I don't have a steem monster account.

Nice giveaway through :)
Have fun!

The random number generator output :

@doloknight Imp Bowman, Javelin Spear Thrower, Highland Archer, Skeletal Warrior
@xtrimmer Vampire, Rusty Android
@ssg-community: Naga Fire Wizard, Gold Wood Nymph, Daria Dragonscale, Goblin Mech

All cards are distributed. Hope everyone received their cards :)

Congratulations, your post was discovered and featured by @OCD in its daily compilation #378!

If you give @ocd a follow – you can find other Gems!

With this nomination you will be able to use @ocdb - a non-profit distribution bot for whitelisted Steemians. We usually add everyone nominated at the end of the week. :) Check our website for the whitelist, queue and delegation info. Join our Discord channel for more information.

We also have a witness. You can vote for @ocd-witness with SteemConnect or on Steemit Witnesses to help support other undervalued authors!

Beautifully described the development thought process. Great read!

Thanks for taking the time to read it. And glad you liked it.

Posted using Partiko Android