The Video Tracking plugin provides an interface for detecting faces, bodies, and hands and using their locations as data-source values and masks that can be used to control virtually any part of VDMX.
In this tutorial we will look at using the face and hand tracking options in the Video Tracking to create a fun example of controlling the parameters of a simple ISF generator.
The main panel for the Video Tracking plugin contains a pop-up menu for selecting which video feed to analyze and a status indicator for the analysis.
For situations where multiple bodies and / or faces are detected, the provided 'prev', 'next', and 'rand' buttons can be used to switch between which is currently being tracked. When previewing the body and face tracking video streams clicking on a bounding box region can also be to change the active tracking.
The tracking options are as follows:
Human Tracking: Detects human bodies and publishes the position as data-sources. This includes the center position, width, height, and the x/y coordinates of each corner of the box bounding detected people. A boolean 'Detected' data-source turns on / off when bodies are found.
Face Tracking: Detects faces and publishes the position as data-sources. This includes the center position, width, height, and the x/y coordinates of each corner of the box bounding detected face. A boolean 'Detected' data-source turns on / off when faces are found.
Hand Detect: Detects one or more hands and publishes the center position as data-sources. Includes options for specifying the maximum number of tracked hands and the chirality type (All, Left, Right, or Pairs). A boolean 'Detected' data-source turns on / off when hands are found.
Human Mask: Generates a mask image that can be used e.g. with the Layer Mask effect to remove the background from images. Includes a quality setting (accurate, balanced, or fast). Also see the 'Remove Background' effect.
Attention Saliency: Generates a mask image using an Attention Saliency algorithm.
Object Saliency: Generates a mask image using an Object Saliency algorithm.
Where available the 'Publish Preview' option can be enabled for each tracking mode. This will create a video feed with overlays showing bounding boxes and other visualizations related to the analyzed images.
The "Minimum Latency" option in the inspector can be enabled to reduce latency (masks will be tighter) but takes longer to process (framerate may be lower, depending on the overall system load). This is option is off by default.
Hand tracking in VDMX6
If you have a LeapMotion or UltraLeap, and want to try the GECO application to share OSC data with VDMX. You can find it here: https://uwyn.com/geco/ We used the first generation LeapMotion controller in this tutorial.
For the iPad you can use Shoots Pro for sharing over NDI or Elgato Epoccam