Step 1: In the Workspace Inspector create an extra layer, inspect it, and hide it by setting its opacity to 0 or clicking the 'Hide' button.
Tip: In the Layer Source controls choose a built-in source such as CoreImage Checkboard or Quartz Composer Gradient to create a test masking image.
Step 2: Add the "Layer Mask.fs" FX to the layer to be masked (the top layer), and in the "maskImage" pop-up menu, select the hidden layer's video stream.
Tip: The Layer Mask is in the Masking FX category.
Tip: Set the composition mode of the foreground layer to OpenGL Over or Source-Atop-VV for best results when compositing over another layer.