Video Art & Databases – The Role of Clip Libraries in Creative Making

The Key Metaphors

Databases in video art can be used practically or creatively. It’s the latter use that is most of interest here, but it’s worth touching on the former use first just to get it out of the way and establish some general context.

One can think of a ‘database’ in different ways. It can be literally a database, written in SQL or Excel, stored on a database server and integrated into an app, or it can be more of a metaphorical concept and rely on earlier-established metaphors like the library, depending on the situation.

In the screenshot above from Resolume (the Arena/Avenue product line), at the top of the UI you can see many thumbnails of video clips that can be called upon in an interactive video scenario. Should we refer to those thumbnail rows as a library or database of video clips?

In Resolume, clips are actually organized into ‘Decks’ which is also an appropriate term. A new metaphor can be added, then, based on the idea of a card deck or slide deck — which stores a collection of cards or slides — and so deck can work as an organizing metaphor just as well as database or library.

What they all have in common is the database principle of easy access or in general computing terms Random Access. This is clear from the Resolume manual, in the section on the terminology:

Excerpt from the Resolume manual

With linear video, a clip in the middle of a movie is not so easy to access! You have to fast forward (then rewind, the fast forward, then rewind) in the streaming window to find it. Or, with a YouTube video, you can click around the timeline bar in search for a specific moment. But, in general it’s not easy to access because with linear video everything is baked into the same long sequence and you have to go hunting and clicking for some specific moment in the footage.

Researching the creative use of databases in video art and design can be tricky, because there are literal libraries (which also have online databases) such as the Video Data Bank (VDB), which curates, distributes and stores an archive of video art — this and other kinds of video databases, most of which are quite technical, clog up the search results when researching this topic.

VDB is one kind of actual, physical, institutionalized video database — literally it has rooms of shelves, and on those shelves are videos just like books on the shelves of a library. Some portion of those videos have been digitized and are available online for viewing in the typical web-based media database paradigm.

Another paradigm VDB adheres to is the idea of only giving you an excerpt of the video for free viewing, since they are a distributor and need to earn their revenue on rentals. A typical excerpt of their video database might be Ann Oren’s The World is Mine.

VDB can also be understood as an archive, which is a related metaphor, close to the concept of the library.

library is a collection or group of collections of books, and/or other print or non-print materials, organized and maintained for use (reading, consultation, study, research, etc.). The term also refers to the room, building, or facility that houses such a collection. Libraries collect and provide access to published materials in order to disseminate information, promote scholarship, and provide entertainment.

An archive is a collection of records created or received by a person, family, or organization, and preserved because of their continuing value. The term also refers to the place or repository that collects these records. Archives collect and provide access to unpublished materials in order to preserve institutional and cultural memory, and to ensure government accountability. (source)

A video production company, record label, film studio or tv network will have massive archives of material from all their past creative efforts. For example, a single documentary on The History Channel might involve a few hundred source tapes, which will be stored somewhere and managed by the video archive staff.

This media can be analog or digital, e.g. you can have an archive of 16mm or 35mm film rolls stored in cans, digital or analog beta tape stored in plastic cases, or digital media files stored entirely as 0s and 1s on servers somewhere. Here I am just introducing a set of related conceptual metaphors — database, library, deck and archive — to capture some of the fluidity of technologies, terms, practices and contexts that can be utilized in creative practices of nonlinear video.

It’s the relationship to nonlinear video, of course, that the various database metaphors (I will primarily use this term, ‘database,’ if only because it’s in this article’s title : ) is of most relevance. With linear video, such as a tv show on Netflix or a film in the theaters, the database function is usually going to come into play only for the editors — who decide which specific frames to include out of all the various shots and takes in the original footage — and the librarians and archivists who keep track of the source media, making sure it is well preserved and labeled etc.

In other words, with linear video, the media database isn’t really part of the audience’s experience. It’s not generally considered to be part of the work, even though everyone knows that most material in the video or film has ended up ‘on the cutting room floor’ and that there was lots of source footage shot and so on.

With interactive and generative video, however, the database really is part of the experience. It’s visible in some way, not necessarily visible as in the Resolume UI in the banner image above where you see each separate clip all at once, but the experience provided to the audience clearly suggests that there is a large collection of video clips in computer storage being regularly accessed to create the experience.

A good example of interactive video that relies on its database or deck of clips would be a VJ performance that uses software such as Resolume:

interactive video

A good example of a generative video, and one that is fairly well-known in new media academic circles, is Lev Manovich’s Mission to Earth, an example of what he called software-based cinema or ‘Soft Cinema.’ Generative database-driven video will play back differently each time because, well, if it didn’t do so, it would just be linear video instead!

generative video

Interactivity and Generativity

Sometimes these terms aren’t very well understood — interactive and generative — and they are sometimes even used interchangeably. The difference between them is worth its own separate and full length treatment, but I will just mention a few of the features that distinguish them.

First, there are objective and subjective dimensions to each concept, interactive and generative. This differentiation between subjective-psychological responses and objective-system features in the technology itself is common in new media.

A parallel difference is with the terms presence and immersion, which are also both often used interchangeably. Theorists sometimes have different definitions of these terms for sure. I personally find this difference to be the most useful:

Presence describes the psychological feeling of ‘being there’ so that the media-world becomes almost as rich in experience as the real world usually does and absorbs attention and activity to the same or a similar extent.

Immersion describes the specific system features that allow for users to separate themselves from ‘the real world, e.g. putting on an HMD and headphones to replace the sources of their sensory inputs.

This is a common way to distinguish these terms, at least in the realm of new media theory. Marketing people, journalists and bloggers, however, often use these terms without any kind of fine distinctions such as this and even make up their own meanings out of their own gut feelings or something they may have heard at a party, who knows.

Objectively, interactive media has affordances — spatial-physical features that allow for certain limited types of actions to be performed. Subjectively, interactive media gives users a sense of agency, feedback and control because choice is available to users to change the output of the media system. This combination of choice and affordances allow for psychological processes like goal-seeking, challenge and skill development to become part of the media experience in which users also become co-authors of the content.

Objectively, generative media requires minimal user input — it may require nothing more than a virtual button press, like activating the iTunes Visualizer, and then the media ‘does its own thing’ out of its own internal procedural and algorithmic processes. Subjectively, generative media is neither annoyingly random noise and chaos, nor repetitively monotonous, but finds a certain ‘sweet spot’ in our aesthetic expectations. It provides a certain mix of novelty and familiarity so that we stay more engaged with it, compared to random noise and redundant repetition.

The difference between interactive and generative can also be understood as a ratio of inputs to outputs. With interactive media, there are a lot of user inputs relative to the possible outputs, while with generative media there is minimum user input relative to the output.

Actually flying a plane is very interactive because there are dozens of buttons, dials, readouts, and controls to engage with. Pushing the autopilot if very generative because the algorithms do all the work, but they need at least a button press to be activated or deactivated.

Interactive!
Generative!

Here are some shorter generative works, because maybe you didn’t watch all 47 minutes of Mission to Earth above and are looking for something with a bit more audiovisual contemporary (and less academic researchy) feel.

Aesthetics and Poetics of the Video Database

So what makes for a good video database in a creative work? And how do you make them? Is there something that can be identified as a database approach, or is the clip collection mainly a practical matter of digital creatives keeping their material organized?

In case these terms seem synonymous or unclear to you — Aesthetics and Poetics — the former term refers to how media addresses perception and experience, whereas the latter term refers to principles of making. Thus, they are not mutually exclusive terms but they have clear areas where they will be interrelated, because often you have a principle for making something in a certain way in order to produce a certain perceptual affect, or vice versa — you want an effect, and then seek the means of its realization.

A good example of this Poetics vs Aesthetics difference is with my university’s main campus ‘iconic’ feature of The Quadrangle.

Aesthetically — meaning what is presented to perception and experience — the Quadrangle design is all about 1960s utopian Brutalist concrete forms and all that is supposed to evoke.

At the level of its poetics, however, is something that you would never actually see but is part of its raison d’etre. This design is supposed to symbolize and practically encourage the ideals of interdisciplinarity, because all the professors in their offices literally can look out of their windows at the office windows of profs in other departments.

We have seen one kind of video database above in the Resolume UI. Here’s another one, from the Max programming environment. The database ‘moment’ in the patch is the Read message. That’s where we will have to hunt for some particular video to load in an interactive manner.

There are many ways you can create a simple database in Max. For example, you can load a collection of video clips into a pop-up umenu object, and then trigger their playback based on their index position in the menu list. Playback of a video file via index position can be either interactive or generative, e.g. the index position can be done manually, or according to some program logic.

creating a list of video files in a pop-up

So, this quickly covers two general approaches to interactive and generative video database creation. Obviously the exact approach will depend on the toolset being used and out of that toolset, what tools-within-the-toolset are available for tailoring to your needs.

Besides the interactive/generative difference, another general set of categorical difference that comes into play in database organization, at the highest level of abstraction that is possible (since it has a thousands-year-old pedigree) is the Form vs. Content difference. Form pertains to perceptual and material aspects of media (how it’s presented), and Content relates to its meaning and interpretation (what it’s about).

Form and Content

Art is a combination of form and content.
Form is the physical manifestation of the artwork. It answers these types of questions:

what is it made out of (the medium)?
what techniques are used?
how were the design elements and principles used?
what is the style (abstract, impressionistic, etc)?

Content is the essence of the artwork. It answers these types of questions:

what is the subject/theme?
what is the context?
what is the meaning/intention?
why was this artwork created? (source)

In the visual arts, there is also a triadic distinction between Representational, Abstract and Non-Objective (or non-representational) imagery which can also be used as an organizational principle. This kind of distinction in many ways is a reiteration of the Form and Content difference, but where there is a third term, Abstract, which entails a hybridization of the two.

Painting, sculpture, and other artforms can be divided into the categories of representational (sometimes also called figurative art although it doesn’t always contain figures), abstract and nonrepresentational art. Representational art describes artworks — particularly paintings and sculptures–that are clearly derived from real object sources, and therefore are by definition representing something with strong visual references to the real world. Most, but not all, abstract art is based on imagery from the real world. The most “extreme” form of abstract art is not connected to the visible world and is known as nonrepresentational.

Representational art or figurative art represents objects or events in the real world, usually looking easily recognizable. For example, a painting of a cat looks very much like a cat– it’s quite obvious what the artist is depicting.

RomanticismImpressionism, and Expressionism contributed to the emergence of abstract art in the nineteenth century as artists became less interested in depicting things exactly like they really exist. Abstract art exists on a continuum, from somewhat representational work, to work that is so far removed from its actual real-world appearance that it is almost impossible to easily discern what is being represented. Abstract art is always connected to something visual from the real world.

Work that does not depict anything from the real world (figures, landscapes, animals, etc.) is called nonrepresentational. Nonrepresentational art may simply depict shapes, colors, lines, etc., but may also express things that are not visible– emotions or feelings for example.

Abstraction indicates a departure from reality in depiction of imagery in art. Abstraction exists along a continuum; abstract art can formally refer to compositions that are derived (or abstracted) from a figurative or other natural source. (Some people also use this term to refer to nonrepresentational (non-objective) art that has no derivation from figures or objects. However, in this class we do not the term abstract like this.) Picasso is a well-known artist who used abstraction in many of his paintings and sculptures: figures are often simplified, distorted, exaggerated, or geometric. (source)

So those are some very high level or ‘top down’ ways that you might organize a database. Lower level or ‘bottom up’ ways to organize a database would develop its categories directly out of its source material. For example, if you are shooting an interactive video based on a script, you might develop database categories and labels such as:

Cafe/Night
Cafe/Day
Bedroom/Morning
Train Station/Afternoon
Park/Evening
Fight scene
Rescue scene

Or, you might use levels of complexity as an organizational tool:

Simple
Intermediate
Complex

Or color:

Greyscale
Monochromatic
Red
Blue etc.

Themes and motifs might be appropriate:

Sad
Excited
Party
Death
Aloof
Victorious
Beaches
Cities
Planets
Halloween/Horror
Tech etc.

You can look at online collections of video loop-based media to get some ideas as to how video database logics might be organized.

New Frontiers for Databases

The latest in database development for moving images has to do with Machine Learning (ML) methods, which are too advanced for the introductory scope of this essay. ML is a branch of AI in which algorithms are ‘trained’ on a corpus of examples, which in this case would be examples of other media.

We’ll have to add ‘corpus’ to our list of guiding metaphors, so now we have: database, library, archive, deck and corpus. Corpus means ‘collection of texts’ and a ‘text’ can be anything, even video files, especially since in the digital realm everything consists anyway of files comprised of 0s and 1s.

With automated ML techniques, a corpus (i.e. database etc.) of video clips can be used to produce automated computational behaviors. A good example of ML applied in the field of media imagery is the website thispersondoesnotexist.com which, on every browser refresh, will generate a new and totally artificial but photo-realistic human face. The algorithm has been trained on a database of thousands of faces, which has taught it what a digital photo-face is supposed to look like.

ML and AI techniques generally displace the database away from the perception and experience of the viewers, similar to how linear video does not showcase all the clips in the nonlinear video editor or in the archive vault. This means that interactive and generative video can be thoroughly produced by a database logic, but the mechanics may remain entirely hidden to viewers.

This produces new possibilities for real-time computational video. Imagine an airport installation where computer vision technology scans passersby and analyzes them for information related to any number of predetermined variables, such as: ethnicity, likely home country, socio-economic class, education level, employment type, gender, fashion and cultural sensibility etc. and then produces ‘personalized’ video clips that relate to each person as they walk past the video installation in the airport.

This kind of example will perhaps make the video database works below seem too primitive!

Database as a Lens

This final section provides examples of video works that involve some kind of database or library, archive, deck etc., as an experiential and design logic factor. When viewing such works, whether you come across them IRL or through the medium of online videos, you can think about the selection, organizational and presentation aspects of the clip collection as a specific design feature to understand and interpret.

Database-based video works invite you to not just look at each individual image in its solo frame, but also consider the groupings and inter-relationships across the frames, whether those unfold spatially — e.g. screen next to screen — temporally — video after video — or along both axes.

Comments are closed.
Stay up to date

Subscribe to our newsletter to stay up to date with all the latest goings on at Optophonia.