Prerequisites:
- Read the tutorial on Layer Composition which covers opacity and blend modes.
- Read the tutorial on Adding an FX to a layer.
The project file for this tutorial can be downloaded here.
Step 1: In the Workspace Inspector create an extra layer, inspect it, and hide it by setting its opacity to 0 or clicking the 'Hide' button.
Tip: In the Layer Source controls choose a built-in source such as CoreImage Checkboard or Quartz Composer Gradient to create a test masking image.
Note the Show/Hide button state in the layer composition controls for this hidden 'Mask Layer'
Step 2: Add the "Layer Mask.fs" FX to the layer to be masked (the top layer), and in the "maskImage" pop-up menu, select the hidden layer's video stream.
Tip: The Layer Mask is in the Masking FX category.
Tip: Set the composition mode of the foreground layer to OpenGL Over or Source-Atop-VV for best results when compositing over another layer.
Previews of the Foreground and Masking layers at various stages of processing and the final output.