Basic Concepts in Video Synthesis – Fundamentals for Creative Exploration

The Core Tech

The general idea of electronic oscillators and their waveform primitives is widely understood in the realms of audio and music, whether in the context of synthesizers or just producing test tones for calibrating equipment. These waveform shapes will be very recognizable to anyone with at least some beginner’s familiarity with the audio signal space.

Video synthesis also uses oscillators, but instead of being presented through speakers for auditioning, it is sent to a visual monitor for viewing. These monitors were originally analog (just like the music synths of the era when this technology was invented) and so an oscillator’s electronic signal output would be processed by the zigzag presentation format of a CRT TV (source).

A CRT TV is a television device that uses cathode ray tubes. These tubes contain one or more electron guns and fluorescent screens that are used to view images.

Below is a video demonstration of analog hardware producing image patterns based on electronic oscillator signals. The key concepts illustrated in the video are:

  • Line rate: the speed of the zigzagging scan of the video monitor, which will be mathematically keyed to AC current rate (60 Hz in North America).
  • Oscillator frequency speeds: how many cycles per second the waveform produces its alternating peaks and valleys in the signal. Video signals are at much higher rates compared to audio signals. Most sound is presented for listening in the 20 Hz to 20 kHz range. Video signal oscillators may go up to 3 mHz and beyond. Megahertz, or MHz, is much faster than kilohertz — a thousand times more to be precise, or millions of cycles per second.
  • Sync: with video, there is both horizontal and vertical sync (described a bit more technically below) which work together to keep the image centered in the screen. When watching a tv show, you likely do want each frame perfectly centered in the screen where it’s supposed to be. With video synthesis, though, sync is just another creative parameter to play with. Horizontal sync resets oscillator phase on every frame, while vertical sync does so with every horizontal line. Phase, you may recall from your knowledge of acoustics or physics, indicates where we are at in the progression of the waveform at any given moment in time.
  • Combining & Modulating Frequencies: just as with audio, where complex sounds are built up out of many overtones and layered sounds, with video synthesis you can create very complex patterns out of simpler signal elements. This can happen by summing multiple signals together, or by modulating signals with other oscillator signals or control signals such as LFOs (low frequency oscillators). LFOs play a similar role in video synthesis as they do with audio synthesis — they produce regular (repeating, rhythmic) changes at a very low frequency such that the changes can actually be perceived. E.g. a 2 Hz LFO will produce a sonic or visual rhythm twice per second, which is audible or visible as a repeating change of some kind. Usually the frequencies involved with sounds and images are so fast, we cannot perceive each individual cycle and are not supposed to perceive them individually, but rather altogether as a complex composite that simply presents a sound or an image.
  • Multi-Oscillators: You will often have setups in hardware and software synths with three oscillators. In audio, this is done to enrich or ‘fatten’ the sound so that it can have a lot of spectral components and be very ‘thick’ and so on. With video, three oscillators are useful because they can be assigned to R G B (red green blue) channels of color information, similar to how in a Photoshop image, each channel of color information by itself is just greyscale information, and they only create a color image when combined.
Photoshop — each color R G B as greyscale information individually. (image source)
  • Higher and Lower Signal = Brighter and Darker Visual Areas.

Here’s a handy summary of short definitions related to image presentation on electronic monitors:

Line rate: It indicates the actual speed with which the bits are sent onto the wire

Frame rate: frequency at which an imaging device produces unique images(frames)

Progressive video: is the method by which all lines (whole frame) are captured at the same instant, representing a single moment in time

The screen aspect ratio is width divided by height, and the same for the pixel aspect ratio.

Horizontal blanking interval refers to a part of the process of displaying images on a computer monitor or television screen via raster scanning. CRT screens display images by moving beams of electrons very quickly across the screen. Once the beam of the monitor has reached the edge of the screen, the beam is switched off, and the deflection circuit voltages (or currents) are returned to the values they had for the other edge of the screen; this would have the effect of retracing the screen in the opposite direction, so the beam is turned off during this time.

Vertical blanking interval is an interval of time between the last line of a given frame and the beginning of the next frame, during which the incoming data stream is not displayed on a CRT screen. It is the time interval allowed for the analog TV electron gun beam to move from the bottom of the current frame to the top of the next one as it scans images. This requires the last 45 lines of each 525-line frame. (source)

Frequencies

Here is a single red oscillator (‘osc’ for short)with a signal of ~5 Hz. Note that it is very easy to count its pulse 5 times per second, as the signal is moving between high (bright) and low (dark) values.

There are many complex interactions that occur in the inter-relatedness between frequency rates and line rates. For example, here is the same osc at 1.639 Mhz.

Now let’s look at it at 1.662 MHz, which seems like a tiny difference of only .023 MHz from the above clip, but at the frequency rate in interaction with the line rate, you get this instead:

In between these two extremes of osc frequency, here’s an example at 121.8 Hz, which seems somewhat mellow and relaxing in comparison:

Let’s get even closer to twice the electrical current rate of North American circuits (60 Hz), and compare the clip above to the one below, with an osc frequency of 120.1 Hz. You’ll see that the signal has slowed down to a crawl, and would practically be in sync (in stasis) if not for that .1 Hz difference.

Because of the 60 Hz screen refresh rate — which is not coincidentally the same as AC electrical current rate — at frequencies below 0.125 Hz you will generally just see pulsing colors, whereas above 10 Hz is where the scrolling line patterns will start to appear. As you increase the frequencies, the lines will become diagonal in orientation and ultimately at the very high ends resemble pure visual noise.

Sync

Here is a ~1 kHz osc frequency with horizontal sync applied. You’ll see that ‘sync is your friend’ when you want to impart some stability into the signal, so that it’s not constantly shifting across the screen in some basic pattern that results from the interaction of monitor line rate and osc frequency rates.

And just for good comparison, here is that same signal with vertical sync applied.

Often you will want to be either ‘close to sync’ or ‘not far from sync’ in the sense that you would like a pattern with some motion but perhaps not too much motion. As you can see, pure sync, either horizontal or vertical, will often freeze your image. I say ‘often’ because depending on the complexity of your patch, other kinds of modulation may also impart movement beyond whatever tendency towards stillness sync can provide.

Wave Shapes

We can impart the waveshape of the osc waveform into the visual pattern. Recall some basic electrical waveform shapes:

Here is what a vertically synced sine wave looks like when synthesized as video, where we see a gradual gradient happening between the brightest and darkest image areas:

Sine pattern

Here is what a ramp or sawtooth type pattern looks like, where there is a sudden jump to the highest value and a gradient to the lowest and darkest value:

Ramp / Sawtooth pattern

Perhaps somewhere in between these two extremes, you get the triangle pattern which has a little less sense of transition between brightest and darkest areas when compared to the sine wave pattern:

Triangle pattern

And finally, here is the square wave pattern, which is basically an either/or — full on, full off, or highest/lowest — high contrast pattern:

Square pattern (note the either/or lack of gradient starkness of the contrast)

Modulation

Depending on what tool you are using, you will likely have many options available for modulation. Typically you will be able to modulate parameters such as:

  • Oscillator frequency
  • Oscillator sync
  • Oscillator phase
  • Oscillator hue
  • The Amount of modulation (for sync, phase, frequency, hue etc.)
  • Oscillator waveform shape

Additionally, you will have sources for modulation signals. Those will often include:

  • Any oscillator, which can modulate other oscillators
  • LFOs (low frequency oscillators), along with LFO parameters such as their strength (amplitude) and frequency rate

You will generally have different kinds of sources available for video synthesis, such as:

  • The oscillators! Already discussed at length.
  • External signal from a live webcam
  • Internal video file from the hard drive
  • Displacement maps, e.g. alpha channel information or other greyscale information that can be used to displace pixels.

What we do with all of these modulations, of course, is make these rather simplistic patterns complex, and thereby convert them into visually interesting patterns.

Patterns

Of course, the real magic of video synthesis is when you create complex patterns out of these elementary signal behaviors.

Here’s a simple pattern where things are a bit more complex than in the oscillator demo videos above. There are three videos below, each from the exact same patch. The videos show that over time, a pattern can evolve significantly, and in fact much of what’s interesting about video synthesis is creating patches where over a long period of time there can be quite a lot of generative evolution in the resultant imagery.

Effects

As you can do with any video signal, effects have a role to play with video synthesis. The number and possibilities of effects are infinite, or at least too infinite for this intro essay. Effect parameters can also of course be modulated by any modulation source, which increases the available complexity to draw on for your design.

Below is the same pattern as above, but with some rotation, scaling, kaleidoscopic mirroring and time delay trails applied.

Interactivity

What we have been looking at so far is primarily generative behavior of synthesized video. In other words, we set up some initial values and states, and then its signal logic auto-imparts the exact real-time rendering of its pixel play.

We can of course introduce all manner of interactivity into our video synthesis patch, using either virtual or external controllers and sensors such as MIDI hardware, audio input, proximity sensors, webcams and anything else that generates some kind of data output in response to some input, which affords interactive control.

This next video is the same patch as above, but simply adds an interactive element to its resulting output pattern.

Tools

This discussion has been kept at a very high level suitable for introductory coverage. The exact look and technical possibilities will depend on which tool you are using. Even though video synthesis originated in the analog hardware realm, its processes and aesthetics are emulated often in the software domain. Here are some tools you may want to look at to explore video synthesis further:

Ylem: “(in the Big Bang theory) the primordial matter of the universe, originally conceived as composed of neutrons at high temperature and density.” Image synthesis techniques based on nonlinear signal flows (aka ‘chaos’) evoke fundamental energies and particles of our universe.
Stay up to date

Subscribe to our newsletter to stay up to date with all things Optophonia.

The Optophonia Festival of Electronic Music, Performance Visuals and Audiovisual Culture