Aesthetics of Audiovisual Signal Processing

Signal Processing

Today’s computational media tools present an infinity of options for the creator, and infinity can be hard to grasp when making aesthetic decisions! Below I present a typology relating signal processing effects to aesthetic experiences. This section excerpts a much longer piece and has been edited considerably for lecture-approachability.

Conditioning

Much effects work involves relatively minor adjustments that do not call attention to themselves as effects at all, but due to a media saturated environment, the media would be ‘off’ (unpolished, unprofessional, lackluster, too casual, consumer-esque, amateurish etc.) without this layer of processing.

Media exist in an environment with other media, and not applying basic ‘conditioning’ imparts a substandard or at least raw and unfinished quality. In photography, conditioning might mean simply adjusting the dynamic range — the black and white levels on the histogram so that the image ‘snaps’ into the right range of exposure, giving the image a little extra ‘punch’ or ‘pop.’ These aesthetic qualities are definable relative to either objective criteria such as what black and white levels ‘should be’ expressed as a dynamic range or to contextual criteria such as what all the other published images are doing that is part of the look and feel of imagery produced at the right level of craft, or to more subjective criteria such as personal taste.

In audio, conditioning may involve adding compression to a voice, removing low frequency ‘mud’ from the bottom end of a mix and ‘brightening‘’ the high end with equalization. In video, it may entail color correction based on rendering flesh tones, cropping the image around a selection of skin with the software oscilloscope in view . This instrument depicts a line for the industry standard ‘correct’ spectrum of flesh tone, which is actually determined by the color of blood under the skin, and hence considered to be ‘race neutral’ but not without some social and cultural contestation.

A good metaphor for conditioning is tuning a piano. Piano tuning is anaesthetic — supportive of many different aesthetics — and in fact when played out of tune is an aesthetic in its own right, symbolic of social or technical chaos etc. Tuning a piano in well, equal or mean tone temperament are conditions of playing the piano, which can be an aesthetic choice in some instances, e.g. if trying to compose music contemporaneous to Pythagorean sensibilities.

Interpretive Affect

Effects can also be engaged within an interpretive framework of embodying or incorporating specific intentions, presuppositions, meanings, directions given and explicit goals into the texture of the media experience. This concept indicates the dual positioning of the effects worker in knowing what affect is being aimed at while also knowing why that affect is being sought and how to achieve it through their technical and industry knowledge.

The effects worker distinguishes between ‘dry’ and ‘wet’ media — unprocessed and processed media — and has typically received instructions from a creative director or producer ranging from general to highly specific goals to achieve in the effects work. Additionally, the effects artist will generally have their own taste and a knowledge of effects work, and so there is a field of practice and knowledge that is called upon.

An example of this interpretive affect mode is the common practice in documentary production to give the male narrator the Voice of God treatment — this is the actual term used by producers, to describe the sense of authoritative omniscience in the voiceover. On the interpretive side, the effects operator understands the instruction, knows what audio effects can give a heightened sense of authority, such as hard dynamic compression, removing “ums” and strategic boosting of low frequency resonances.

A parallel to this in cinematography would be shooting a character from a low angle to increase their aura of power. The audience receives the affect — an increased sense of danger or criminality or significance in this character — but for the camera operator, the affect is also an intention, a goal aimed at, and likely an instruction from a director or even written directly in the script.

While effects in the Conditioning mode are typically not noticeable as effects (unless compared side by side, e.g. wet versus dry), effects produced in an Interpretive Affect mode can run the gamut from minimal — as in color grading, where perfectly well-produced media is subtly altered for purposes of achieving an overall look and feel — to maximal, where a flashy special effects sequence is constructed. What distinguishes Interpretive Affect from Conditioning is that the latter has a much more restricted scope that is more about ‘tweaking the percepts to make it look or sound not bad,’ rather than aiming to produce intended affects in the media reception (though conditioning, e.g. making basic image adjustments, can sometimes cross the threshold into producing intended affects).

Below, the image on the left can be understood as signal processing. The image on the right is better understood as image processing, because to composite a complex visual effects sequence requires working with assemblages of images as images, rather than manipulating their base properties at the pixel level.

Image Source 1 and Source 2.

Open

It has been said of filmmaking that the three most important elements are the script, the script, and the script. This adds to the interpretive dimension of the prior discussed mode. But some scripts are more open than others — music video, for instance, is full of beats that may justify cutting at any one of them. And of course there is also media production without scripts. The Open mode of signal processing is less dominated by interpretive requirements and is closer to a general play with the plasticity of perception and media.

Signal processing is mediated through what Deleuze and Guattari call functives, logico-mathematical operations on a plane of reference, not necessarily electronic, but increasingly so.

Deleuze and Guattari make a distinction between, on the one hand, ‘concepts’, which are the proper domain of philosophy and, on the other, ‘functives’ which are the domain of science and all other empirical matters. (source)

Signal processing requires variables with ranges — some particular attribute is manipulable within a delimited set of values, typically from a low to high value of some kind. Because the image to be processed has been parsed out mathematically down to the electron charge of a pixel, or the range of hearing from silence to bursting ear drums, the number of variables at hand to be varied, each scaled between minimal and maximal ranges, quickly adds up to levels of complexity approaching near-infinity.

In Openness towards signal processing, there is a kind of oscillation (cycling back and forth) between what one might call attitudes of exploratory discovery and affective resonance. In exploratory discovery, the operator is making use of the functives of the processors, which may involve a mixture of one’s present and potential knowledge-in-the-making, with regards to outcomes of techniques performed. With unfamiliar software there may be some ‘noodling around’ that can produce startling results. Much learning in signal processing often progresses from this general noodling to the discovery of the effect produced — functives afford this kind of learning through the production of effects and, in the audience, their resultant affects.

One of the stark differences between coding in a programming language and working with ‘off the shelf’ signal processors is that with programming languages, one cannot really noodle — noodling produces nothing but syntax errors and precisely nothing happens, whereas signal processing software presented via a user interfaces allow an instantaneous learning of effects produced through causes that one need not necessarily know much about. Knowledge can be developed through subsequent re-tracing of actions, so that noodling progresses to knowledge derived from a specific case or type of action.

Affective resonance is an aspect of the Open mode wherein a creator produces an affect that is not simply pure variability or alteration in the image, but also has affective significance. This occurs when an affective value is produced beyond mere change, i.e. it may be a ‘cool’ or ‘poetic’ or ‘ahhh’ moment that resonates across other regions of subjectivity in some way.

Openness is typically the result of a free oscillation between affective resonance and exploration — settling on the affect that one is going for, and perhaps co-discovering the affect one hopes to achieve through open exploratory discovery of the possibilities of the software. As the photography student who produced the image below states in her blog: “I played with shutter speeds and apertures to capture different types of movements to see what sort of image they created.”

https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fplayer.vimeo.com%2Fvideo%2F167880482%3Fapp_id%3D122963&dntp=1&url=https%3A%2F%2Fvimeo.com%2F167880482&image=https%3A%2F%2Fi.vimeocdn.com%2Fvideo%2F765111552_1280.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=vimeoColor is Dangerous!https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fa56RPZ_cbdc%3Ffeature%3Doembed&display_name=YouTube&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3Da56RPZ_cbdc&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2Fa56RPZ_cbdc%2Fhqdefault.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=youtube

Experimental

Elsewhere I have discussed the polysemy of the term experimental and suggested that a more concrete use of the term should avoid a general reference to just ‘anything unusual’ and be more explicitly connected to specific interventions in systems of mediation.

What does “experimental” mean today? The term has become something of an umbrella category — much like the “unclassifiable” category employed by the iTunes software– for a range of practices which might span from a desire to be “avant-garde” to the paradoxical intention of thwarting egoistic intentionality to creative processes highly dependent on the rules of specific techniques for generating the work (“the tool is the message” as it has been described). In this paper I will argue for a reconceptualization of what being “experimental” might mean. I will start with drawing a distinction between two kinds of play — what I will call parametric and originary play. This distinction rests upon the following premise– there is one kind of play we are engaged in when we are playing by the rules of the game, and another kind of play that has to do with inventing the game to be played in the first place. (source)

As a mode of signal processing, Experimentation means giving some additional autonomy to the technical apparatus. This autonomy can range from reserving an ‘editorial’ function to the operator — the right of accepting or rejecting what the technology has produced — to completely erasing this editorial role and playing instead the role of a systems designer, leaving in place entirely effects production to the system designed.

The Open mode described above still occurs within the defined parameters of the signal processing environment — one is working with the given capabilities of the system with which one is working, e.g. using Photoshop as the environment in which to manipulate a digital image. There are many tools in Photoshop, and it is a deep program, but all the processes are found in the menu somewhere and pre-defined at Adobe HQ.

A prepared piano, such as that of John Cage or his student and practitioner Margaret Leng Tan, is considered ‘experimental’ because the parameters of the piano — its tuning, timbre, what the keys are supposed to do — have been modified, and this modification of the core parameters — because of the screws, bolts, paper clips, twigs and other ‘foreign objects’ inserted between the strings — makes the performative situation experimental.

A degree of autonomy is given to the instrument, as it is not exactly predictable what sounds will be produced when various objects of differing sizes, materials and shapes are inserted at different locations and angles between the strings, despite an established tradition of prepared keyboards extending back through Cage and his Concerto for Prepared Piano and Chamber Orchestra (1950–51) to Erik Satie’s Piège de Méduse (1913 or 1914).

The notion of Experimental here differs somewhat from common or historical characterizations. While the idea of a relative autonomy to the instrumental processes is shared with early formulations of experimental artistic practices, the terminology was typically related to the ‘lab-like’ technical objects and spaces in which formal experimentation occurred.

Because of all the ‘gadgety’ and ‘scientific’ apparatuses lying about, the terminology of experimentation propagated. This discursive atmosphere suggested that artistic innovations with technology might enjoy the same status of scientific experiments.

We can trace the scientific or lab-based notion of artistic experimentation in the writings of Cage and McLaren. For Cage, the experimental position is achieved largely through regarding sound as a physical phenomenon, in much the same way that science or experimentation is addressed toward some empirically constituted object.

Now, on the other hand, times have changed; music has changed; and I no longer object to the word ”experimenial.” I use it in fact to describe all the music that especially interests me and to which I am devoted, whether someone else wrote it or I myself did. What has happened is that I have become a listener and the music has become something to hear. Many people, of course, have given up saying ”experimental” about this new music. Instead, they either move to a halfway point and say ”controversial” or depart to a greater distance and question whether this ”music” is music at all.

For in this new music nothing takes place but sounds: those that are notated and those that are not. (source)

The autonomy granted to the technics is succinctly stated by Cage as: “Those involved with the composition of experimental music find ways and means to remove themselves from the activities of the sounds they make” (source).

Norman McLaren’s writing exhibits a complementary character to experimentation, which is the high degree of technical manipulation and control afforded by new technologies of mediation. His ethos does not share Cage’s Zen project of ego effacement, but indeed regards technical apparatus as an expanded field for compositional control:

To this end, twenty-four degrees of dynamic level were used (representing a decibel scale) and opposite each note in the score the number representing the desired dynamic level of that note was written.

For instance, 0,1, and 2 represent three differing degrees of ppp [pianississimo]; 9, 10, and 1 1, three shades of mp [mezzo piano]; 12, 13, and 14, three degrees of mf [mezzo forte];21,22, and23,three degrees of fff [fortississimo]; 24 represents a ffff.

Subdivisions of these twenty-four degrees were constantly being used (particularly in crescendos and diminuendos), but were seldom written into the score. In local or rapid crescendos and diminuendos only the starting and finishing dynamic marks were written and the type of crescendos and diminuendos (such as “arithmetical”or “geometric”) were indicated by a small sketch.

The volume was controlled sometimes by manipulating the shutter or diaphragmof the camera and so affecting the exposure(variable density control) but more often by covering up the one- inch-wide drawing until only a half or fourth or other fraction

of its width was visible (variable area control). Whichever method

was used, the calibration was in decibels, giving the composer

complete control of dynamics.

The sound of a note, however, is affected not only by volume but by its attack, sustention, and decay, or tone contour….(source)

The newness of technology involves the need to create new rules to new games. Over time these rules and games congeal into styles or forms which have defined parameters that one plays within, at which point they are no longer to be considered experimental, which should refer instead to inventing the rules of the game to be played.https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2Fvideoseries%3Flist%3DPLiAAvpUW05vgh5jGx5Y-X3mMtGM9gesh4%26start%3D239&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3DZrKgyY5aDvA&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2FZrKgyY5aDvA%2Fhqdefault.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=youtube

Clearly, these operational modes of signal processing have some overlap: there is no hard edge between them— at the boundary of boundaries is zero, as the physicists say. Diagrammatically we can imagine these modes and their overlapping border cases as follows:

The four aesthetic modes of signal processing.

Early Signal Processed Video

When we think of analog synthesizers, we usually think of synths for sonic or musical production, but the same technology supported video synthesis as well, back in the day. Here’s a great example of one of these electronic beasts, in the form of a ‘5 Minute Romp’ through the aesthetic potentiality of an image processor.

It will be clear that what Sandin is doing in this video plays out at the conditioning level — mainly just showing the methods available for altering the pixels of moving images. At the same time, each moment of analog ‘noodling’ is suggestive of aesthetic possibilities that are merely nascent at the level of knob twiddling and cable patching, and it remains for additional creative elaborations to realize these potentials into full works of media art.https://cdn.embedly.com/widgets/media.html?src=https%3A%2F%2Fwww.youtube.com%2Fembed%2F8qh6jRzjmcY%3Ffeature%3Doembed&url=https%3A%2F%2Fwww.youtube.com%2Fwatch%3Fv%3D8qh6jRzjmcY&image=https%3A%2F%2Fi.ytimg.com%2Fvi%2F8qh6jRzjmcY%2Fhqdefault.jpg&key=a19fcc184b9711e1b4764040d3dc5c07&type=text%2Fhtml&schema=youtubeA 5 Minute Romp through the Image Processor.

Today, analog development of video synthesis continues with hardware modules such as the Chromagnon, which is based on the EuroRack hardware framework so that it can interface easily with analog synthesis modules. More commonly, though, computational media and design folks work in digital emulations of modular analog video synthesis technologies, such as with with Lumen app (Mac only) or Max/MSP’s Vsynth package.

The Chromagnon hardware video synthesizer
Max’s Vsynth package