The Amanuensis Is a Video Game | The Amanuensis: Automated Songwriting and Recording Blog #4

Repository

The Amanuensis is an automated songwriting and recording system aimed at ridding the process of anything left-brained, so one need never leave a creative, spontaneous and improvisational state of mind, from the inception of the song until its final master. The program will construct a cohesive song structure, using the best of what you give it, looping around you and growing in real-time as you play. All you have to do is jam and fully written songs will flow out behind you wherever you go.

If you want to try it out, please get a hold of me! Playtesters wanted! You only need to be a musician who likes to jam, no programming knowledge required.

This is an open-source project, if you'd like to get involved: https://github.com/to-the-sun/amanuensis

Introduction

The purpose of this blog is to promote the project by giving people a first-hand view of it in action, go over features and discuss the current state of development.

Today I will discuss

• the video game aspirations of The Amanuensis
• how it compares to established rhythm games
• the potential for collaboration with MIREX 2018
• the intent to develop a game in The Amanuensis the likes of which the world has never seen
• the need for Python programmers, graphics designers, etc. to get involved and help make it a reality!

Post body

The Amanuensis is a video game

I don't know if you're like me, but I used to play a lot of Guitar Hero. And it was said all the time back then (of people other than me… mostly), "now if you just put that much time into playing an actual guitar, think of the music you could write." And it was so true. Even the premise of Guitar Hero is ludicrous: a band blows up and becomes world-famous, by playing nothing but cover songs? I don't think so. Well, enter The Amanuensis.

At its heart, The Amanuensis is a game. It's a far more open-ended version of Guitar Hero where, instead of playing other peoples' songs that are set in stone from the onset, the music is your own and it evolves with you as you play. The end result is a song written and recorded by you, something valuable to show for your time spent. But really it was no work at all, because all you were doing was playing a game.

In the same way that Guitar Hero attracted musicians and nonmusicians alike, The Amanuensis is ideally meant to be fun and easy to jump into. Not only will it take care of the dirty work of arranging the song and engineering a recording, but it helps guide the novice player in the right direction by analyzing the fundamentals of what makes something musical, rewarding them by capturing only the bits of audio that begin to form a distinct rhythm, rather than just noise.

Where does the project currently stand?

The more visual user interface elements that can be implemented, the closer The Amanuensis will come in closing the loop and becoming a true video game. There's something strangely satisfying about having a little score pop up every time you hit a button; the only thing along these lines currently occurring in The Amanuensis is the circular "recording" icon turning on when you're in rhythm and off when you're not, but there are plenty of other analytics that could potentially be displayed.

In addition to real-time scoring, a game like Guitar Hero also shows you a visual representation of upcoming moments that will score highly, so you can plan for them and take aim with your actions.

When you're following along to an already-written song, that's essential, whereas a musician who's comfortable with improvisation needs only the music they're hearing to gauge when to act. But the exact same schedule of upcoming moments already exists in The Amanuensis, due to the rhythmic analysis it's conducting. The first major upgrade towards a game UI such as this, a stepping stone for casual players, was recently implemented (courtesy of the ever-skilled @ajmaln) in the form of the Rhythm graph that can be seen in this clip:

The spikes in the graph are projected moments where many established beats coincide. The peaks move from further out in time towards the left of the graph, where 0 represents the present moment, and are essentially the "targets moving down the fretboard" (to put it in Guitar Hero terms). When the user plays a note, it will be recorded if it corresponds with one of these.

To officially become a video game

At the moment it's still choppy, because it only updates as necessary. What it really needs is to update at a steady interval, like say, 30 frames per second, so it will move smoothly towards the player. Therefore, I see putting a proper game UI on the main Python script, with a proper game engine, as quickly becoming a pertinent task for the project moving forward. I am assuming so far that the pygame package would be the best tool for this. So if anyone out there has experience with it and would like to help out (or has a better idea) please get a hold of me!

My idea is not to copy Guitar Hero directly, with specific targets or symbols of any kind moving down a conveyor belt. There's really more of a gray area in terms of what constitutes a "good" or "bad" action when using The Amanuensis, and more information that can be conveyed to the player as well. So what I envision is a sort of gradient representation of the Rhythm graph. Here's a quick mockup of what's in my head:

Imagine the shaded area flowing downwards like a waterfall until it hits the line at the bottom, which could light up or activate in some way when the user plays a note. Roughly speaking, if that occurred in a lighter area of the shading the note could be considered a "hit" or in a darker area a "miss", but this representation also could accommodate a more sophisticated sliding-scale scoring system.

The Amanuensis is also already capable of simultaneous multiplayer. Not just two-player, but actually up to 16 players can contribute at once, one for each track (or potential instrument) in the recording. If a second track becomes active the UI could double up to represent this. There is another quality rhythm game out there called Amplitude which utilizes tracks for each instrument as well:

The Amanuensis also currently allows its player(s) to switch between tracks at will, just as quickly and easily as seen in the video. The main difference right now is that the rhythm being calculated by The Amanuensis is universal, that is, the music being played in each track is all combined and then each are judged based upon this same data. But there is certainly room for more individualized analysis, and it is one goal of the project.

And with that, I would like to make a quick aside…

MIREX 2018

The Music Information Retrieval Exchange (MIREX) is a series of competitions put on by the University of Illinois at Urbana-Champaign focused on various aspects of musical analytics. One in particular, Patterns for Prediction, is all about judging the likelihood of future musical events, given something that's already been played. This is exactly what The Amanuensis aims to do.

I have been in contact with Tom Collins, PhD, one of the "captains" of this competition, about the potential for collaboration with any of its participants. The requirements for participation in Patterns for Prediction involve more than just rhythmic analysis; like The Amanuensis, it's MIDI-based, but pitch information must also be taken into account. Therefore, any of the algorithms submitted will be more sophisticated than those used in The Amanuensis and yet still highly compatible with it.

The deadline for submissions is August 25th, so after that point in time The Amanuensis may get a significant boost in its intelligence. We will have to see. In the meantime, the project can be prepared with a UI suitable for such an augmented consciousness, whenever it is that that enlightenment inevitably occurs.

Back to video games

With pitch information being taken into account in its predictions, another opportunity for game-like scoring arises. As in Amplitude or Guitar Hero, the tracks could be subdivided by individual note. Whereas those games only have three or four different notes you can hit, the tracks in The Amanuensis could subdivide dynamically based on the number of notes the player is actually using. The gradient scheme could still be used, with the shading aligned along each subtrack lighter or dimmer based on what pitch is expected to be heard at that moment.

But by no means would it ever be necessary to do exactly as the program predicts, hitting targets or risking failure like in the aforementioned rhythm games. There is no way to fail with The Amanuensis. However, to take the cues it gives you (and thereby emulate your past playing) is to increase the patterning in your developing song, which is essentially what makes something musical rather than just noise. Again, a stepping stone and learning tool for the novice musician.

But if there's no way to fail, what kind of game is that?

In Guitar Hero you'll be presented with insane solos which will force you to improve your skill if you ever want to complete the game, but if you're writing your own music what's to stop you from just hitting the same note over and over in a steady beat like an open-car-door chime? It would be the surest way to be extremely rhythmic and score highly with every action.

The simple answer is: nothing. If that's the sort of music you want to create, then so be it, but on the other hand even the most skilled musician can find it all too easy to slip into familiar tracts of playing (the same old scales, time signatures, etc.) and become uninspired. And at the same time, the nonmusician might need something to entice them into the realm of writing music in the first place.

So what's to reconcile these two things, to give the player the option of being challenged in their creativity without specifying certain rules or structure for the song?

Something totally arbitrary

I have in mind an entirely disparate game slapped on top of the one I've so far described. What if the line of "0 ms into the future" in the mockups above was actually a physical surface on which a common and beloved platform game could take place? And what if you played this platform game with the same controller/instrument you were using to make music? Ideally, you could roam around and explore any sort of virtual environment at the same time you were generating the soundtrack you heard as you went along.

Now here's where things really get interesting…

It's already entirely possible to run The Amanuensis in the background while you play your favorite computer game. I once set up a Smash Bros 64 rom so that every time you kicked with Captain Falcon you would hear a kick drum and to use Link's spin move was to unleash a crash cymbal on your opponent. I found it hilarious and awesome to "dance battle" this way, but it was extremely challenging to keep a cohesive beat and do well at the same time. If you just fought like you normally would, what you heard would be chaos and The Amanuensis would never catch a steady beat to begin your song with.

What if you played an actual rhythm game like Dance Dance Revolution or Guitar Hero? The results are definitely more interesting and The Amanuensis will get to work trying to assemble a song for you, but here again, you're really just following a script you were given on-screen and what it captures will only wind up being a series of excerpts from it.

The crux would be in closing the loop and ensuring that every event that occurred in this integrated game was taking cues from and being informed by the rhythmic analysis as much as possible, so the actions taken by the player in order to survive in this new world did not result in purely random instrumentation on the one end of the spectrum, nor was rigidly directed on the other, but merely arbitrary and still musical.

The overall notion here would be to create a game that was simple and fun enough that anybody could sit down with it and get sucked in. All the while a soundtrack builds around them without them even realizing it's their creation (monitoring could actually be turned off, so all that was heard was what had been recorded, not the instruments initially). At first it might be more ambient and less obviously connected to the moves the player is making, but as they reached the later levels and the difficulty intensified it would become increasingly necessary for them to be precise and articulate in order to succeed, meaning that more and more of their playing would contribute to the soundtrack and it would grow more frenetic right along with the on-screen action.

Really this wouldn't have to be a platform game at all. It could potentially take any form: a sidescroller of some other kind, a top-down dungeon crawler like Crypt of the Necrodancer, a puzzle game or with enough manpower and time invested, maybe even some sort of first-person shooter. All the options are still on the table. The only necessity would be in establishing some ground rules to ensure the game did not just randomize the users' playing, but corral it arbitrarily.

A quick definition of what I mean when I say arbitrary

An arbitrary choice selects from a set of possibilities that all meet certain criteria, and involves purposeful decision-making. A random choice is not the result of purposeful decision-making, and chooses from the set of all existing possibilities.

Some ground rules

In keeping with the goals of The Amanuensis, the game should never force the player to do anything, but only reward them when they act in a way conducive to musicality. So for example, a weapon could be continually charging and discharging with the rhythm of the song so that it does more damage at certain moments than others, an enemy might let down its shield to vent gases at rhythmic intervals or a powerup might appear on beat and disappear otherwise.

A more advanced enemy might lock onto the player and start copying their next few movements; the only way to release their hold would be to also continue to repeat that little riff, thereby adding to the complexity of the patterns in the growing song. This would be the one central guideline that all gameplay would need to fit within and advocate:

to increase the number and complexity of patterns for The Amanuensis to find

So for example

Once a song has begun recording The Amanuensis has already established the length of a quarter-note beat according to your playing; with this in mind, it wouldn't be hard to create a grid across the background for the game to function upon. Obstacles, bonuses, enemies and everything else could slide into place aligned with this grid, meaning these items could interact with ripples in the gradient, and the beats they represent, further out in projected time than just the present moment.

Another consideration is symmetry. What I mean by that is, if the levels always extend to the right, then the control that moves the character in that direction will wind up being used far more often than the others and therefore so will the note it produces. In a platform game, the fact of gravity could have this same sort of influence. It may be that a top-down game of some kind, where the action could proceed in any direction, would be the most fundamentally unrestrictive for our purposes. At all times the game should remain unbiased and generic if it can, so as to leave as many creative options open for the musician as possible.

Maybe even more fundamental to take into account would be the movements of your character itself. A player might hit the button to begin their movement at a rhythmic moment, but wind up interacting with some other object at an arhythmic one simply because of the transit time between locations. Considering this, it might be best if your character was not humanoid, but some sort of rocket-propelled robot or something else that realistically would flit around suddenly and quickly, arriving at every destination near-instantaneously.

The graphics and design aesthetic

It's these sort of functional considerations that will inform the aesthetic and premise of this game and I would love to discuss the possibilities with anyone who's interested in seeing these notions come to life. So far I've been leaning towards a solar apocalypse of some kind as the setting. The gradient raining down in the background reminds me of a kind of stellar radiation and many of the enemies, objects, etc. will be sputtering on and off in various ways as it passes through them.

However, I am not in any way set on this trajectory; really anything could work. What I would love is to find a person or group of people who are interested enough in this concept that they would like to take over the design aspects of this part of The Amanuensis completely, while I focus more on the audio end of things.

So at this point I will make a call out to anyone who

• knows some Python and would like to help with coding
• is good with graphics and would like to contribute something in that regard
• or simply likes video games and would be interested in discussing any of the above

Just reply to this post or find me on discord @to_the_sun!

Until next time, farewell, and may your vessel reach the singularity intact

---

https://tothesun.bandcamp.com/
https://soundcloud.com/to_the_sun
https://sellfy.com/to_the_sun

References

• Guitar Hero image taken from wikihow.com.
• Amplitude video found on Higher Plain Games's YouTube channel
• Definition of "arbitrary" taken from quora.com

All products mentioned are property of their respective owners.

Series Backlinks

Blog #1 - Introducing The Amanuensis: Automated Songwriting and Recording
Blog #2 - Birth of a Song and the Battle for Its Evolution
Blog #3 - Getting Set Up