OPENGL TIMELINE WITH NEW MOUSE CONTROLS: A new feature for the open-source project The Amanuensis: Automated Songwriting and Recording

logo by @camiloferrua

Repository

https://github.com/to-the-sun/amanuensis

The Amanuensis is an automated songwriting and recording system aimed at ridding the process of anything left-brained, so one need never leave a creative, spontaneous and improvisational state of mind, from the inception of the song until its final master. The program will construct a cohesive song structure, using the best of what you give it, looping around you and growing in real-time as you play. All you have to do is jam and fully written songs will flow out behind you wherever you go.

If you're interested in trying it out, please get a hold of me! Playtesters wanted!

New Features

• What feature(s) did you add?

Previously the timeline view of the user's growing song was displayed on the screen using scripting to dynamically delete and create panels every time a change required them to be redrawn. This assemblage of rectangles fit in the background behind each track, but it was about as inelegant a way as possible to achieve the purpose. With longer songs/many spans noticeable slow-down and lag was beginning to occur with each redrawing of the spans.

Everything to do with the timeline GUI has now been redone, using OpenGL in js, which is more sophisticated and really the way it should have been done all along. This implementation is light as a feather and far cleaner.

All of the hotkey functions pertaining to spans have been re-imagined as well, utilizing quick and intuitive mouse actions rather than clumsy key movements. Spans can be trimmed by holding SHIFT and dragging a span's boundaries inward. Spans can be deleted by ALT-clicking them, or ALT-dragging to delete multiple spans. Spans can be split in two by CTRL-clicking them. The loop boundaries are no longer adjusted using an rslider above the timeline, but simply by dragging anywhere within it. The comp track can be selected simply by clicking it, rather than having to arrow to it.

• How did you implement it/them?

If you're not familiar, Max is a visual language and textual representations like those shown for each commit on Github aren't particularly comprehensible to humans. You won't find any of the commenting there either. Therefore, I will present the work that was completed using images instead. Read the comments in those to get an idea of how the code works. I'll keep my description here about the process of writing that code.

To be very clear, I paid a freelancer to create this OpenGL timeline. I am not claiming ownership of that code or the work it required. What I am submitting for your approval is all of the code and work required to hook it up to the existing system and make it into something fully functional, which was no small task in itself.

Therefore, below I will show as many of the major portions of code as possible that I alone was responisble for. It's safe to say that any of the project's existing files were modified only by me. The new files are a mix.

These are the primary commits involved:

• Commit #1

The timeline is displayed in a jit.pwindow and drawn using OpenGL in Javascript (through js). The patcher that houses it all is the new timeline.maxpat, where I fed the required data to and from the Javascript code. The subpatchers in the following screenshot were not created by me, therefore I will not show their contents; I simply utilized them in making the patcher as a whole.

timeline.maxpat, housing most all of the new timeline functionality as an abstraction loaded into a bpatcher in GUI.maxpat

One of the major hurdles was in deciding how exactly to integrate this new jit.pwindow into the existing GUI. The former interface displayed a lot of information simultaneously by overlaying the instrument/sound options for each track, with the panels denoting spans appearing behind. It turns out that using OpenGL in this new manner causes all the pixels rendered to appear on top of everything, whether or not the actual pwindow itself is on top.

Therefore it was decided that the track options would need to be separated from the timeline. Now the user can switch between the two views by hitting the SPACE key (formerly god mode), which causes the timeline to become invisible, revealing the options underneath.

While comping, there also were UI elements which I initially thought could just be overlaid, but again, this proved not to be true. Here I had to delve into the Javascript myself and add a few elements to the actual OpenGL: a flashing rectangle in the comping track, as well as vertical lines denoting the edges of the looping spanse. Although the results don't look quite as nice as I had things previously, this really is the "right" way to do it and as lightweight as possible.

// create a [jit.gl.gridshape] object to act as a loop_start
var loop_start = new JitterObject("jit.gl.gridshape","ListenWindow");
loop_start.shape = "plane";
loop_start.dim = [2,2];
loop_start.poly_mode = [1,1];
loop_start.color = beige_border;
loop_start.blend_enable = 1;
loop_start.position = [0,0];
loop_start.scale = [0,1];
loop_start.layer = 8; // highest value means object closest to viewer.
loop_start.enable = 0; // not rendered (unless comping(1) called)

// create a [jit.gl.gridshape] object to act as a loop_end
var loop_end = new JitterObject("jit.gl.gridshape","ListenWindow");
loop_end.shape = "plane";
loop_end.dim = [2,2];
loop_end.poly_mode = [1,1];
loop_end.color = beige_border;
loop_end.blend_enable = 1;
loop_end.position = [0,0];
loop_end.scale = [0,1];
loop_end.layer = 9; // highest value means object closest to viewer.
loop_end.enable = 0; // not rendered (unless comping(1) called)

//create a rectangle to denote the comp area
var audition = new JitterObject("jit.gl.gridshape","ListenWindow");
audition.shape = "plane";
audition.blend_enable = 1;
audition.dim = [2,2];
audition.layer = 7;
audition.color = beige_fill; 
audition.enable = 0;

instantiates the added GUI elements

function comping(i, pass) {
    loop_start.enable = i;  //only visible when comping
    loop_end.enable = i
    audition.enable = i && pass != -1;  //only visible when comping and a pass is auditioning
}

function loop(start, end) { //position the loop bars and flashing rectangle based on the current loop
    loop_start.position = [-aspectratio + (2 * aspectratio * (start - timeline_offset) / total_length), 0];
    loop_end.position = [-aspectratio + (2 * aspectratio * (end - timeline_offset) / total_length), 0];
    comp_length = (loop_end.position[0] - loop_start.position[0]) / 2;
    audition.scale = [comp_length, span_height];
    comp_x = -aspectratio+(2*aspectratio*(start - timeline_offset)/total_length) + comp_length; //
    comp_y = (2*(tracks-(clicked_track-0.5))/tracks-1);
    audition.position = [comp_x, comp_y];
}

new functions to manage the added GUI elements

function bang() {
    flash += .05;
    audition.color = [0.937255, 0.952941, 0.72549, flash % 1.0];

the js receives a bang at a set framerate; this code causes the audition rectangle to flash

There is no practical way to render text in OpenGL so it was decided to move the labels that display information about the comp and pass numbers above the timeline itself. Not quite as intuitive, but it works just fine, especially since the rslider previously occupying this area has been removed (setting the limits of the loop is now handled simply by dragging the mouse). A new subpatcher was created to supplant the rslider, delivering the values from the Javascript via r drag.

this screenshot shows the new subpatchers rslider_replacement, comp_labels and comp_position and their locations in the parent patcher, GUI.maxpat

Aside from any number of minor alterations littered throughout the system, the last major bit of work done by me pertains to the new functionality of splitting span with a CTRL+click. The split of the span itself in handled in the Javascript, but it was necessary for me to also connect up my end of things to the process. This entailed modifying the cues in the second half of the split span to belong to the newly created span.

p alter_second_half_cues was added to GUI.maxpat, receiving messages from the Javascript through r split

• Commit #2

Cleanup and commenting

GitHub Account

https://github.com/to-the-sun