Analysing musical multimedia (Cook, 1998)

Nicholas Cook’s book Analysing Musical Multimedia from 1998 moves from musical analysis (the relationship between notes) to multimedia-analysis (the relationship between music and images). He goes in against media-agnostic theories from pioneers Eisenstein and Eisler with a metaphor model and three concepts; conformance, complementation and contest. Sergei Eisenstein (1898-1948) and Hanns Eisler (1898-1962) (who wrote his book with Theodor Adorno) had a discussion about parallel and counterpoint in the relationship between music and images.

“Only a contrapuntal use of sound in relation to the visual montage piece will afford a new potentiality of montage development and perfection. The first experimental work with sound music must be directed along the line of its distinct nonsynchronization with the visual images. And only such an attack will give the necessary palpability which will later lead to the creation of an orchestral counterpoint of visual and aural images.” (Sergei Eisenstein, Grigori Alexandrov and Vsevolod Pudovkin 1949 [1928], p. 84)

Example: Battleship Potemkin (1925)

‘The most striking and immediate impression [of correspondence with the audio] will be gained, of course, from a congruence of the movement of the music with the movement of the visual contour — with the graphic composition of the frame.’ (Eisenstein 1957 [1942], p. 173)

Eisenstein would call this the audiovisual inner movement

Diagram of Alexander Nevsky (1938) from The Film Sense (1947)

“There is a favorite Hollywood gibe: ‘Birdie sings, music sings.’ Music must follow visual incidents and illustrate them either by directly imitating them or by using clichés that are associated with the mood and content of the picture … Music cut to fit the stereotype ‘nature’ is reduced to the character of a cheap moodproducing gadget, and the associative patterns are so familiar that there is really no illustration of anything, but only the elicitation of the automatic response: ‘Aha, nature!’ (Adorno & Eisler 1947, p. 12-3)

“It is true that there must be some meaningful relation between the picture and the music. If silences, blank moments, tense seconds, are filled out with indifferent or naively heterogenous music, the result is a complete nuisance … The actual inventive task of the composer is to compose music that ‘fits’ precisely into the given picture; intrinsic unrelatedness here is the cardinal sin.” (Adorno & Eisler 1947, p. 68-9)

As an answer on this ongoing conflict Cook introduces the metaphor model, where music and images are related like two terms in a metaphor. These two terms must have some enabling similarity in them like; love is war or love is a pot of noodles and not love is a gecko. Also then there need to be a transfer of attributes that will make a formula like;
War is (x, y, z)
---------------------------------
Therefore, love is (x, y , z)

He then refers to Marshall & Cohens diagram

“If the respective attributes of [Music] and [Image] intersect (as indicated by the shaded segment [x]), then some or all of the remaining attributes [a], then some or all of the remaining attributes of the one become available as attributes of the other” (Cook 1998, p. 69).

But between what elements are there possible audio-visual relations? On the musical level it can be between musical style and genre, lyrical meaning, emotional character, structure, musical parameters like pitch/contour, rhythm, tempo, timbre, dynamic and (leit)motives. The pixels can focus on visual style and genre, structure, mise-en-scène, cinematography and montage.

At the end the (ax) can be conforming; so the music and images make each other stronger because of the enabling similarities and transfer of attributes that have the same meaning in them.
Complementation would mean that the music adds something extra to the images because of the use of enabling similarities. The similarities are not on the foreground though but in the form of coming together and forming a new meaning. While contest means that the meaning of the images and music are both different, but when put together because of the transfer of attributes is possible to work to a meaning. A note is that it is the most unused form of audio-visual editing. Most of the times complementation is used.

Examples of videos where I used this theory to edit 3 videos with different music but the same images: https://steemit.com/music/@inmusicalterms/3-different-soundtracks-on-the-same-images-by-me-supporting-cook-s-theory

Yours Sincerely @inMusicalTerms