Making Music Videos with Game Engine Cinematics – Some Aesthetic Considerations

Game engines, being basically engines for the simulation of anything that exists in time and space, and which is subject to physical laws like cause and effect — which is, like, everything! — can be used to make music videos through their integrated cinematic tools.

Doing so, however, is somewhat contrarian from a media aesthetics perspective. For decades, the aesthetics of music video have been dominated by montage-oriented approaches, and the logic of association and juxtaposition we also find in related art forms such as poetry and collage.

The stylistic opposite of montage is continuity shooting and editing, whereby the filmmakers try to recreate a coherent flow and sense of time and space across all the little slices of footage they’ve edited together. Audiences are not supposed to notice all those brief slices, but instead apprehend a whole uninterrupted world of action.

The music videos that do make use of continuity-related approaches, such as from practices of narrative and documentary — telling a story (with usually dialogue-less acting) and capturing the real (such as recording a live performance) — are the exceptions that prove the rule, as they are vastly outnumbered by the montage-based videos.

Continuity and documentary style also exist within montage oriented videos, as moments of narrative or reality that are to be subsumed in an overall montage style that treats these moments of unfragmented spatiotemporal representation as an element to be juxtaposed like any other in its grammar of constant contrast.

Game engine cinematics are generally going to give you the equivalent of the long shot, which is the purest form of continuity. You can of course sequence together pieces of these virtual renderings of seamless time and space in a way that evokes traditional continuity editing, such as following Hitchcock’s maxim to ‘cut out the boring bits.’ The smoothness of movement through a virtual space is effortless — no Steadicams or gimbals required to get that fluid shot gliding through an environment.

The automatic continuity style of this medium seems at odds with much of what we expect from music videos, so in the works discussed below I have sought ways of introducing that customary variety of imagery in montages within the simulated continuity-producing game world.

Syzygy, for example, uses several methods for inducing what might be called ‘montage reminiscence’ within its overall long shot movement around a pop art solarpunk fantasy island. These include:

– Constantly changing art visuals in the form of moving through a virtual art gallery of sorts. The video appropriates the artworks of appropriation artists who appropriate elements of popular culture in their visuals. As many of these works are collages, montage is formally represented by its 2D counterpart — photomontage.

– Achieving exactly one edit by moving the camera straight into one object and having it emerge out of another located some distance from the opening scene.

– Using video media as textures so that the grainy low resolution aesthetics of Super 8 analog film are brought into the pristine 3D rendering of the virtual environment.

– Introducing some ‘Easter Egg’ moments, such as changing the clothes of characters when they are viewed from different perspectives, or suddenly populating a level of the structure when the camera moves to it, which had previously been shown to be empty.

These are montage ‘games’ that one can play within game engine cinematics that default generally to a pure continuity style. Another game I play with this medium is what I call ‘virtual photography,’ where I treat the interactive environment of a game engine as a source of still 2D imagery, preventing the viewer from also being a user. Those scenes set up for producing 2D images can be repurposed later for moving image works.

A minimalist meditation with analog sync drift on the relationship between music and internet memes with subtle overtones of dwelling on our own mortality.

Rasasvada is part of another project I call the Phenomenology of Fun, which is not the same as Liz Sills Phenomenology of the Funny but perhaps it is funny, too. Its name is also related to my interest in ‘applying’ (whatever that may actually mean) non-Western aesthetics in my work. In a globalized world, ‘the Western’ is there for anyone to appropriate as a social default, so openness to the non-Western by Westerners is probably artistically responsible.

While consisting of a single shot, a montage sensibility is imparted through:

– Regularly changing the objects that appear on the grid when the giant pink cube starts bouncing.

– Using a video texture that is visually busy and redolent of low resolution analog experimental animation.

– Switching video textures between animation and live video.

– Leaving all video loop start and end points undisguised, to impart a touch of the 60s French New Wave and Cinéma Vérité ‘reveal the apparatus’ sensibility.

– Along those lines, I often like to leave computational errors in my game engine based works. This is the same kind of leaning that architects of concrete structures have when they show all the holes and markings from the wood and metal forms the wet concrete was poured into. The horizontal lines in the main grid pattern of the ground plane are in fact errors — one is supposed to adjust lighting and shadow parameters to get rid of them, but I liked the look of the error so left it in.

This low key approach to post-apocalyptic cinepoetry takes a tour through an abandoned structure in a depleted land, where imagination seems to be the only hope.

Down Temp Oh uses some of the same techniques already discusses above. Additionally,

– It switches between 2D still and 3D moving image content, opening with the former.

– This switch, however, is not seamless, and the change in the general atmospheric conditions (the smog level) is a visual cue to the edit point.

– We emerge fully out of the game environment into live action video at the end. While the bulk of the video is game engine based, it is bracketed at the start with a 2D image (derived from the same environment) and at the end with a mix of live action video and other forms of animation, which relatives its medium. This is similar to music video styles that montage not just footage content but media types, such as integrating Academy 35mm, black and white 16mm, analog and digital video resolutions from various technical eras, Polaroids, animation, etc.

– And in fact, the final part of the music video, after we emerge from the game engine world, is more or less a montage, just a very slow moving one.

– A rather strange thing happens at the end, introducing what could be protagonist in the very last moments, converting what is usually a center of focus (the main character) into essentially a (perhaps puzzling) visual fragment.

– It places some odd items in some odd places, like a mirror-gazing cat in the trusses, or a psychedelic cube also up there, or a glowing goat we stare at for a long time. Many of these are objects that have no real world analog, whereas the objects in the world of Syzygy (mostly!) made general representational sense.

– It includes a wider range of glitchy and abstract VJ loops peppered spatially throughout.

– There is some visual beat synchronization at ‘the big reveal’ when the sound system shows up out of nowhere, potentially giving causal or narrative justification for the music that is playing.

– And, compared to the other videos above, the virtual camera is a bit jumpier.

One thing these videos often have in common, beyond sharing the same or similar techniques across them, is that they sometimes share content. This is intentional intertextuality, where the videos make occasional reference to each other.

As to what sense these content interlinkages make, that is part of the Easter Egg hunt game being played with this game engine media. The viewer will just have to watch as many of the videos as possible, as many times as possible, in order to find all the Easter Eggs, so as to be able to say that those Easter Eggs have all been found ; )

To give away one of these free eggs, one of these intertextual elements is simply the creative strategy of employing more media relativization, such as infusing game engine based cinematics with earlier approaches to interactive and generative visuals such as using video synthesis elements. Video synthesis is the equivalent to the modular analog processing of a video signal instead of and audio signal. Only, the video synthesis methods used here are digital emulations of these analog processes.

Music and sound effects often provide the motivation for making cuts in a montage sequence, and music reactive particle systems produce a general ‘movement on the beat’ that evokes that sensibility. Montages are all about rhythm, and making game engine content beat and audio spectrum responsive will disrupt continuity since spatial-temporal seamlessness is not supposed to be punctuated by the soundtrack. Music that totally dominates the visuals is usually extradiegetic in film — not part of the unfolding narrative itself, but instead playing directly on the audience’s emotions — and that also works against continuity’s usual purpose.