Making and Installing GLSL Composition Modes for VDMX

Download the completed tutorial project including sample GLSL shaders and VDMX setup.​

(note: this tutorial is for VDMX5 only; VDMX6 has updated to use Metal shaders for composition modes)

To get the best performance out of using the Hap video codec within VDMX we also recently added another new feature making it possible to use GLSL shaders to perform composition between layers. While the standard 'OpenGL- Add' and 'OpenGL- Over' modes are the most efficient options when it comes to rendering, when more complex composition modes such as 'Difference' or 'Multiply' are needed shaders are the best alternative when playing back movie files, particularly when you're not using CoreImage or Quartz Composer based FX on the layer.​

One of the other benefits of using GLSL shaders for composition is that it is very easy to write your own and add them to VDMX as a way to further customize your output to your own visual style.

In this two part tutorial we'll look first at the basics of adding new 3rd party shaders to the VDMX assets folder on your Mac, and then move on to the intermediate level step of creating new custom blend modes.​

For more GLSL intergration into your VJ setup there is a great tutorial by Jim Warrier on converting shaders into Quartz Composer patches to use with VDMX.

Two layer VDMX project using GLSL shader based 'Threshold' composition mode.

Basics: Installing and using new GLSL Composition Modes

First download the example set of GLSL shaders created for this tutorial.

From the 'Help' menu within VDMX, you can quickly access the special directory where the GLSL shaders are installed. Copy the downloaded .fs files ('Threshold' and 'ThresholdRGB') into the 'glslCompModes' sub-folder.

​Launch VDMX and choose the new option from the Layer Composition Mode pop-up menu.

​Choosing the 'Show Assets Folder in Finder' menu option.

Viewing the contents of the glslCompModes assets folder in the Finder.

Installed ​GLSL shaders listed in the Composition Mode menu.

Intermediate: Creating your own GLSL Composition Modes for VDMX

To create a new GLSL composition mode, first create a new plain text file in TextEdit, BBEdit, or any other plain text editor. Save the file with the path extension ".fs" to the assets folder as indicated in the earlier part of this tutorial.​

When creating a composition shader for VDMX, only three functions are needed, one for each potential render situation between the layers: top only, bottom only, top and bottom mix.

In this example we'll be making a basic 'threshold' style composition mode in which pixels in the top layer whose brightness is above the 'topAlpha' input variable show through to the background. To begin first create a "Threshold.fs" file and place it in the assets folder.​

Then in your text editor, enter the GLSL code below, or replace with your own.

Editing 'Threshold.fs' in BBEdit.

For reference a complete list of available functions and more information on the GLSL specification can be found on the OpenGL.org website here: http://www.opengl.org/documentation/glsl/​​

/*

    Discussion: 'Threshold.fs'

    //  Fill in this function to composite the top layer over the bottom layer
    vec4 CompositeBottomAndTop(vec4 bottom, float bottomAlpha, vec4 top, float topAlpha)
    
    //  Fill in this function for conditions when top layer is invisible  / ignored (opacity = 0)
    vec4 CompositeBottom(vec4 bottom, float bottomAlpha)

    //  Fill in this function for conditions when bottom layer is invisible / ignored (opacity = 0) 
    vec4 CompositeTop(vec4 top, float topAlpha)
    
    //  Incoming variables
    vec4 bottom - bottom pixel color (RGBA)
    vec4 top - top pixel color (RGBA)
    float bottomAlpha - opacity of bottom layer
    float topAlpha - opacity of top layer

*/

vec4 CompositeBottomAndTop(vec4 bottom, float bottomAlpha, vec4 top, float topAlpha)
{       
    //  Get the brightness of the top pixel
    float       brightTop =  top.a * (top.r + top.g + top.b) / 3.0;
    vec4        darkenedBottom = vec4(bottomAlpha) * bottom;
    
    //  Use topAlpha is our threshold- anything brighter than it should appears as the bottom image
    vec4        returnMe = (topAlpha < brightTop) ? (bottom * bottomAlpha) : (top);
    returnMe = mix(darkenedBottom, returnMe, top.a);
    returnMe.a = top.a;
    return returnMe;
    
}

vec4 CompositeBottom(vec4 bottom, float bottomAlpha)    {
    vec4        returnMe = vec4(bottomAlpha)*bottom;
    return returnMe;
}

vec4 CompositeTop(vec4 top, float topAlpha) {
    vec4        returnMe = vec4(topAlpha)*top;
    return returnMe;
}

Finally, test all the shader within VDMX by creating a test project with two layers and setting the composition mode of the top layer to your new blend mode. Load some sample movies or Quartz Composer patches and try setting the opacity of each layers to a variety of different values including both on, both off, top visible, bottom visible, and both at 50% opacity to see how it behaves at different settings.