It has been just over TWELVE YEARS since the initial release of the HAP video codecs, and in this blog post we are going to be diving into the details behind the newest addition to the HAP codec family – HAP R, an ultra high quality GPU accelerated codec.
In these benchmarks we’ll be looking at how HAP R compares to the other HAP codecs in metrics like image quality and file size, and discussing some of the technical details for those who are curious.
Read MoreToday we are extremely excited to announce the first big round of updates to VDMX6 that begin to fully take advantage of our new Metal based rendering engine introduced last fall. This release adds a collection of powerful video analysis based plugins and FX built on top of Apple’s CoreML and Vision frameworks to help people bring their visuals to the next level.
Video Tracking
Faces, hands, and human bodies can be detected and their positions published as data-sources for controlling parameters throughout VDMX. It can also be used to generate masking images and visualizations that can be previewed and used as part of the video processing pipeline.
Scopes
The Scopes plugin generates and displays a variety of useful visualizations of the color data for a provided video stream including waveform and vectorscope modes.
OCR
Scan images for text and QR codes, publishes detected strings as data-sources to use with text fields, text generators, and anywhere else strings can be received.
Color Transfer FX
Automatically shift the color and brightness levels of a video stream to match a reference image.
Remove Background FX
Instantly remove the background (or foreground) of a video stream using person segmentation.
Blur and Overlay Faces FX: Uses face tracking to blur, pixellate, and overlay images on top of faces.
Segmented Color Transfer: Segmented version of the new Color Transfer FX that allows for selecting different references for the foreground and background. (macOS 14+ only)
New Segmented Blur FX: Applies different levels of blur to the background & foreground of a video stream. (macOS 14+)
You can see these features in action in the new set of video tutorials:
or try out these features directly in the VDMX6 demo with the bundled example setups that can be loaded from the Templates menu.
Today we are excited to announce the open source release of our latest codebase for working with Metal and rendering ISF shaders on macOS: ISFMSLKit
This codebase was initially written for the VDMX6 update last fall, and is now ready for use by other developers who are looking to support ISF in a Metal environment on the Mac. Along with this we’ve also open sourced VVMetalKit, our general purpose Metal framework which powers the rendering engine in VDMX6 and other utility apps.
Before we dive into ISF for Metal, let’s do a quick review of the ISF specification itself.
ISF, the Interactive Shader Format, is a cross platform standard for creating video generators, visual effects, and transitions. It was originally created for use in VDMX in 2013, and is now supported in over 20 different apps and creative coding environments.
ISF files contain two main components:
JSON Metadata
GLSL Code
These are used together by host applications to provide user interfaces to control parameters within the rendering pipeline.
An example ISF generator as code and as a source in VDMX.
The ISF documentation pages include a Quick Start overview, an in depth look at writing GLSL code for real-time visuals, and useful reference notes.
This new open source release has three main repositories:
ISFMSLKit: ISFMSLKit is a Mac framework for working with ISF files (and the GLSL source code they contain) in a tech stack that uses Metal to render content. At runtime, it transpiles GLSL to MSL, caches the compiled binaries to disk for rapid access, and uses Metal to render content to textures. Everything you need to get started with ISF shaders in a Metal environment.
VVMetalKit: VVMetalKit is a Mac framework that contains several useful utility classes for working with Metal, such as managing a buffer pool, rendering, populating, converting, and displaying textures.
ISFGLSLGenerator: ISFGLSLGenerator is a cross-platform c++ lib that provides a programmatic interface for browsing and examining ISF files as well as generating GLSL shader source code that can be used for rendering. This library doesn't do any actual rendering itself - it just parses ISF files and generates shader code.
These frameworks are the same code that we have running under the hood in VDMX6 and are free for anyone to use in their own software.
The ISFMSLKitTestApp running the Color Bars generator being processed by the Kaleidoscope filter.
All of the code needed for working with ISF in Metal can be found in ISFMSLKit.
This repository has several dependencies that are included, such as VVMetalKit and ISFGLSLGenerator.
The ISFMSLKitTestApp demonstrates the entire process of working with ISF under Metal:
Converting GLSL to MSL.
Caching converted shader code.
Loading, validating, and parsing ISF files.
Rendering ISF shaders to Metal textures as generators and FX.
Displaying the final image in a Metal view.
The three major classes you’ll be working with in ISFMSLKit are:
ISFMSLDoc: This class is a programmatic representation of an ISF document, and can be created either from the path to an ISF document or by passing it strings containing the ISF document's shaders. It performs basic validation, and parses the ISF's contents on init, allowing you to examine the ISF file's attributes. Behind the scenes, this class is basically a wrapper around VVISF::ISFDoc from the ISFGLSLGenerator library.
ISFMSLCache: This class is the primary interface for caching (and retrieving cached) ISF files. Caching ISF files precompiles their shaders, allowing for faster runtime access.
ISFMSLScene: A subclass of VVMTLScene used to render ISFs to Metal textures.
Detailed information on these classes and how to use them can be found in the ISFMSLKit readme file and documentation.
As ISF v2 is an OpenGL / GLSL based specification, in order to get ISF files working in Metal, first they need to be transpiled from GLSL to MSL. In a traditional OpenGL pipeline, behind the scenes ISF looks essentially like this:
This is where ISFMSLKit and SPIR-V come into play. SPIR-V is an intermediate format from The Khronos Group for converting shaders from one format to another. To convert from GLSL to MSL, this happens in two parts:
GLSLangValidatorLib which converts GLSL to SPIR-V.
SPIRVCrossLib which converts SPIR-V to MSL.
These are provided as precompiled binaries in ISFMSLKit to reduce compilation times, but the source is also available if you are curious to see how they work, or you'd rather build your own.
Our updated diagram with the conversion to SPIR-V and MSL in place of GLSL is now something like this:
When using ISFMSLKit inside of VDMX6 or the included ISFMSLKit sample application, on launch, every unconverted GLSL-based ISF file is first converted to MSL, and then tracked for changes during runtime. The original ISF is left intact. Error logs are generated for files that fail to compile or translate for whatever reason. Files that successfully transpile are cached for later use.
Example error log for a GLSL shader that fails to compile.
The process demonstrated in the ISFMSLKit repository is also a great starting point for developers looking to convert ISF files to other shader languages for other platforms, or as a reference for anyone looking for an example of SPIR-V in action.
Once the ISF files have been converted from GLSL to MSL, they are ready to be validated, parsed for input parameters, and rendered by the host application using ISLMSLKit or your own custom library.
In addition to providing the base for the ISF rendering classes in ISFMSLKit and its sample application, the new VVMetalKit contains a variety of general purpose utilities for working with Metal. New and experienced developers alike will find this framework useful when creating Metal based video apps on the macOS.
More information about ISF can be found on the isf.video website and examples can be found on the ISF Online Editor.
The set of open-source ISF files that is bundled with VDMX has over 200 shaders including useful utilities, standard filters, and a wide variety of FX for stylizing video, is completely free for other developers to use.
We are always excited to hear from people using ISF – if you have any questions, feedback, or just want to send us some cool links to your work for us to check out, please send us an email at isf@vidvox.net.
Developers who add support for ISF to their own software can send us a URL to their website and png icon to include on the list of apps that support ISF.
Hello VDMX Fans!
We’re thrilled to announce the official release of the biggest update to VDMX in over a decade: VDMX6!
What’s changed? Oh, just the entire rendering engine.
VDMX6 is now powered by Metal, fully transitioning away from the deprecated OpenGL technology. This update not only prepares us for the future of Mac hardware and OS updates but also opens new doors for upcoming features, improvements, and even more optimizations.
What’s else is new?
Read on! Looking to download the new release or buy a license? Visit the new Vidvox.net homepage!
One of the first new benefits of our new Metal rendering engine is we can add support for the latest addition to the HAP video codec family!
Following in the footsteps of HAP Q, we now have HAP R.
What do you need to know about HAP R?
The HAP R format creates files that are higher quality than HAP Q, with alpha channels, at about the same size.
The latest AVF Batch Exporter utility app in the Extras folder includes support for importing / exporting to the newest variation of HAP.
HAP R is more technically known as HAP 7A because it is based on the BC7 texture format.
Other developers looking to update their own software to support HAP R will find technical details of the format on the HAP Specification and the HAP in AVFoundation framework GitHub pages useful in getting started.
Stay tuned for more details and benchmarks on HAP R in a future blog post!
Along with overhauling the rendering engine we’ve rewritten the tempo tracking algorithm completely from scratch to run natively on both Intel and ARM64 processors. No more having to use Rosetta to take advantage of this crucial feature on newer Mac hardware!
Along with the performance boost of switching the rendering engine over to Metal we’ve optimized several other core parts of VDMX, making the app more efficient with an overall lower memory footprint.
One of the top things we hear from people who want to get started in working with live visuals is the financial barrier to entry. Along with adding feature improvements we understand that one thing we can do to make our software better is to make it more accessible to everyone and anyone, so we’ve decided to set the price on VDMX6 at 99 USD.
We have also introduced VDMX6 Plus at 249 USD for users who want to extend their experience by using TouchDesigner and Vuo compositions alongside the rest of the powerful features included in the standard version.
Students, teachers, and hobbyists can apply by email for our educational discounts of 25 USD on VDMX6 and 100 USD on VDMX6 Plus.
Existing VDMX5 users who purchased their license after September 12th, 2023 are entitled to a free update to VDMX6 Plus.
All other VDMX5 users can write in to receive a coupon to get 100 USD off on VDMX6 Plus.
Customers who have a license of VDMX6 can get the full price off of their original purchase if they choose to upgrade to VDMX6 Plus later.
Using TouchDesigner compositions in VDMX6
Yes indeed, as mentioned just above, in VDMX6 Plus you can now use TouchDesigner compositions as interactive generators, effects, and control data plugins.
Note: that using TouchDesigner compositions in VDMX6 Plus requires a TouchDesigner install, and a valid paid TouchDesigner license on the same computer (Education, Commercial, or Pro) Non-commercial will not work.
Example compositions can be found on the VDMX6 dmg in the Extras folder and additional detailed information can be found in the VDMX documentation.
No update to VDMX is complete without some new interactive video generators and FX to play with!
In this release we’ve got a new category of FX, Overlays, which include useful FX such as Doodler, Cursor, and Highlighter. These make it easy to add small elements on top of your video without having to do additional layer management.
There’s also some other new fun audio visualizers and glitch FX to play with:
Audio Waveform Shape
Pattern Glitch
Stylize Glitch
BlackMagic makes some of the most popular audio / video capture devices on the market, and we’ve just made using them with VDMX even better…
Use BlackMagic devices for simultaneous input and output, also known as full duplex mode, on supported devices.
Send audio streams to BlackMagic outputs.
Lower latency audio capture from BlackMagic inputs.
Apply key streams as alpha channels on supported devices.
Output alpha channels as external key streams on supported devices.
Find your favorite FX and sources even faster.
A major overhaul like this also deserves some new style, so we’ve included some new color themes and various minor interface tweaks throughout the app to improve overall usability.
One of the most useful UI additions is the new ‘Search’ field for quickly filtering the options in the layer source & FX menus.
Minimum requirements: macOS 12; Intel or ARM64 Mac
Recommended OS: macOS 13 or later; M1 processor or later.
Sadly, we had to drop support for older OS releases where that could not take advantage of certain features and optimizations to Metal that were introduced in macOS 12. But we will continue to host old versions of VDMX5 to support older systems. See (VDMX Versions for details).
While VDMX6 will run on macOS 12 and Intel machines, the biggest performance gains are found in the newest OS releases and hardware where Metal itself is more optimized. We strongly recommend updating to macOS 13 or later where possible.
With so many changes behind the scenes to create a stronger foundation for us to build on, one of the most existing parts of the VDMX6 update is what will come next… make sure to keep your eyes on this blog, tag and follow us on Twitter, YouTube, and Instagram, and sign up for our email list to get the latest news from VIDVOX!
We tried to make the process of updating from VDMX5 as seamless as possible: for example opening existing project files will create a copy that will be automatically updated to for the new version and GLSL based ISF shaders will be automatically transpiled as Metal shaders on launch.
Beyond that, we know that people updating from older versions have a ton of specific questions about the update process and we have created a VDMX5 upgrade FAQ page on this topic in the VDMX6 documentation. This page includes a listing of major changes to be aware of, and if needed we are also happy to answer questions you may have by email or on the forums.
With VDMX6 moving into Metal, some older plugins, custom FX, or ISF shaders may not be immediately compatible with the latest version. We recommend testing out the demo of VDMX6 first before making a full transition. If something is incompatible VDMX will do it’s best to give you a warning. And the best part is, you can have VDMX5, VDMX6, and VDMX6 Plus all installed on the same machine without conflict!
Once again we have traveled across the Atlantic to join the Mad Mapper team for the infamous Mapping Festival in Geneva! As always it was a fantastic time getting to spend time with amazing visual artists from all over the world, seeing some mind blowing performances, and participate in workshops showing off the latest techniques in live visuals and projection mapping. We leave having spent time reconnecting with old friends, meeting new ones, and eagerly awaiting our next chance to come back!
For those who could not make it to Geneva this year we have published the slides and accompanying notes from our two day workshop on getting started with using GLSL and ISF for creating real-time visuals and effects to use in apps like VDMX and Mad Mapper, which you can find here: https://bit.ly/isf-mapping-2024
We also have some photos to share! Geneva is a beautiful city and an absolute joy to visit.
And be sure to follow us on instagram for more live updates and pictures from events such as this!
As many of you know, a few years ago we released the https://www.interactiveshaderformat.com/ as a place where people could share their custom GLSL based generators and effects to use with VDMX and other software for live visuals. Today we are excited to announce the first major revision to the website, with a redesign to make it easier to find and create shaders.
Read MoreOnce again we'll be running our special sale on VDMX for students and teachers heading back to school over the next couple of weeks with an extra 100 USD off; and we're even more excited to extend this same discount to ALL customers!
This means that from January 13th through January 20th, everyone can get a license of VDMX for just 249 USD, and students / starving artists pay only 99 USD.
Along with making great software for VJs and our related open source projects, one of the areas we have tried to focus on is creating educational materials to help visual artists at all levels help improve their craft. Over the last few years our website tutorials section has served as the main outlet for our various lessons, demonstrations and conversations of the various techniques used for all aspects of live visual performance. We also recently launched VV Edu, a set of lesson plans for teachers to use as a starting point for teaching live visuals and VJing in the classroom.
Read MoreToday we are extra excited to announce an open source collaboration with the team over at Cycling74, creators of Max. Now all of the amazing ISF shaders that come with VDMX and on interactiveshaderformat.com can be used right alongside your other Jitter code using the jit.gl.isf object, available on Mac and Windows!
You can install jit.gl.isf directly from the Max package manager panel, accessible from the File menu. If you don’t already have any ISF shaders installed, also grab the free ISF Editor app for your appropriate platform. The free editor tool comes with the same standard set of 300+ generators, effects, and transitions we bundle with VDMX to get started with.
With this release Max joins the over a dozen apps and frameworks that support ISF as a standard for GLSL shaders. With ISF, there is no need to convert or translate code when moving between software. Write your generators and effects once, then use in Max, Motion, Final Cut Pro X, VDMX, on the web, and other video platforms. The specification includes conventions for working with multi-pass shaders with persistent frame buffers, allowing for the creation of complex compositions in a single easy to share file. The ISF documentation pages include detailed walkthroughs of the specification along with useful reference notes and a quick start for learning the basics of GLSL.
ISF for Jitter can be used out of the box with over 300 generators, effects, and transitions. Install the free ISF Editor to get the standard collection of shaders.
jit.gl.isf includes an extensive help patch, examples, and complete reference.
Use the Max package manager to download the jit.gl.isf object and access the launcher patch. Available on Mac and Windows.
Visit the ISF sharing website to find new generators and effects that can be used directly in with jit.gl.isf without any additional conversion. GLSL shaders written in other formats are often easily translatable to the ISF specification. See the ISF Documentation for useful tips and examples.
The ISF specification helps standardize techniques for creating advanced compositions like audio visualizers. See the include example patches and ISF Documentation for more information.
Want to learn more? Read more about it on the ISF for Jitter website, watch an introduction video tutorial, and visit the Max forums for more details from the Cycling74 crew. Developers curious to take a look under the hood can find the open source codebase in the jit.gl.isf repository. For the latest in ISF news, follow the @ISFVideo account on Twitter.
And don’t forget to check out some of our other tutorials describing techniques for using Max and VDMX together, such as using the OSCQuery Helper tool for simplifying OSC sync and sharing video streams via Syphon.
Once again we are getting in the spirit of capitalism with the yearly Black Friday / Cyber Monday sale!
The sale runs from Friday, November 28 through the morning of Tuesday, December 3rd!
During this time we’ll be taking 100 USD off every purchase from our online store!!!
And this sale stacks with our regular educational, just send us an email with a picture of your photo ID (or other similar proof of enrollment) to get your coupon code to double down your discounts and get VDMX for only 99 USD!!!
Or if you are not a student, we also offer a “starving artist discount” – do a small community giveback project, such as sharing videos loops or making a tutorial to get an extra discount.
Another year, another view! VDMX running in macOS 10.15 Catalina.
Hi everyone!
As usual we have a yearly update guide for people using VDMX who are looking to install the latest version of macOS on their computers.
You can read more about the specific details of the most recent release, including the list of new features and effects in the release announcement on the forums and discussion thread on upgrading to Catalina for more tips: https://discourse.vidvox.net/t/vdmx-macos-x-10-15-catalina-update-guide/942
The most noteworthy change for developers in 10.15 is the requirement for notarizing applications that are distributed outside the Mac App Store – if you are running version b8.7.1.6 or later, your copy of VDMX should be properly notarized.
If you run into any problems with updating VDMX to macOS 10.15, or just want to let us know about your experiences with Catalina, please send us feedback using the Report Bug option from the Help menu in VDMX, or send an email to support@vidvox.net.
With ISF for Motion, published variables from GLSL shaders can be animated using value key-frames, adjustable directly from the Motion timeline editor. Iterate over different visual possibilities from your shader creations using bezier curve tools instead of writing complex code.
GLSL shaders written as FX can be applied to layers, including movies files and other generative assets, and re-exported, adding a new range of possibilities for using code to create and process video. With ISF for Motion you can easily experiment with different ways of combining your code creations to take them to the next level.
Compositions containing GLSL shaders can be exported from Motion and FCP X as movie files, at super high resolutions and frame rates. A great way to showcase code that is too complex to be rendered in real-time, even at 8K Ultra HD and larger.
ISF for Motion can automatically create template files for loading shaders in Final Cut Pro X. Generators, filters and transitions show up in their appropriate categories along with their published input parameters.
With ISF for Motion, published variables from GLSL shaders can be animated using value key-frames, adjustable directly from the Motion timeline editor. Iterate over different visual possibilities from your shader creations using bezier curve tools instead of writing complex code.
Beyond the standard set of included assets, ISF for Motion makes it possible for you to write your own GLSL code and watch it live update in the Motion preview. Works with any standard text editor, as well as our free ISF Editor tools.
Today we are excited to announce ISF for Motion, a new way to use the same suite of real-time video generators and effects that we include with VDMX inside of Motion and Final Cut Pro X!
The ISF for Motion FxPlug enables usage of GLSL shaders written in the ISF specification within Apple Motion and FCP X alongside their standard generators, effects and transitions, opening up a world of new possibilities where creative code meets your traditional motion graphics and editing workflows.
Download and install ISF for Motion from the Mac App Store
Read MoreHey everyone!
For the first time ever we are doing a Black Friday / Cyber Monday sale, getting in the spirit and jumping on the bandwagon of this runaway train we call capitalism!!!
The sale runs from Friday, November 23rd through the morning of Tuesday, November 27th!
During this time we’ll be taking 100 USD off every purchase from our online store!!!
And this sale stacks with our regular educational, just send us an email with a picture of your photo ID (or other similar proof of enrollment) to get your coupon code to double down your discounts and get VDMX for only 99 USD!!!
Or if you are not a student, we also offer a “starving artist discount” – do a small community giveback project, such as sharing videos loops or making a tutorial to get an extra discount.
Along with making great software and open source projects, one of the areas we have tried to focus on is creating educational materials to help visual artists at all levels help improve their craft. Over the last few years our website tutorials section has served as the main outlet for our various lessons, demonstrations and conversations of the various techniques used for all aspects of live visual performance.
Today we are extremely excited to announce that we are taking the next big step in our educational offerings, and as you might imagine, we’re doing it open source style: https://vidvox.github.io/vvedu
Read MoreOne of the most useful techniques for artists working with audio and visual software is the OSC protocol for sending control information from one application to another. While MIDI is great for a lot of situation, OSC was designed to be more extensible and allow for a variety of different datatypes, greatly expanding the possibilities within the community of real-time digital art. Over the years the ecosystem of tools that support the protocol has grown and now we are excited to release a new extension to help the community take OSC to the next level.
The OSCQuery Protocol is an agreed upon specification that extends the base functionality of OSC to make it easier to construct impromptu or improvisational interfaces between different systems…
Read More
In another blast from the past we've just received an email from Angie Eng about a new visual music festival being organized in Boulder, CO. it sounds like an awesome time and you can find out more about the open call for works below.
But as a long time user of VDMX we weren't going to let Angie go without a quick interview of her own work as well...
Read MoreNext week we'll be heading to Geneva to join Mapping Festival to teach a few special workshops and presentations.
If you'll be in town for the festival we'd love to get a chance to say hi! Leave a comment below or send us a message to let us know who to look out for, or just show up at one of the events listed below.
On Wednesday, 4th May:
VJing: WTF is it? – a brief history and introduction to the world of VJing and its related fields.
On Thursday, 5th May:
VDMX Master Class – a two part class that will cover the basics of performing, producing and manipulating visual media with VDMX and advanced tips for pro users.
Writing ISF Shaders For Live Visuals – for this class we will look at the possibilities offered by the GLSL coding language to create live visuals (some examples can be seen on sites like interactiveshaderformat.com and shadertoy.com).
ProjectMilkSyphon is a free app to create sound reactive visuals from an audio input to your computer such as a built-in microphone or line in and mixed into your favorite VJ software over Syphon.
This tool is possible thanks to the projectM community who did all the years of hard work that are behind this project and generously made it available under the terms of the LGPL.
Read MoreIf you happen to be in London next weekend we'll be joining VJ London for a special meetup to talk about VDMX. They've also lined up some excellent AV sets of local talent to keep us entertained!
The event will be on 10th April 2016 starting at 7:30 PM at the Westbank Gallery (3-5 THORPE CLOSE London W10 5XL GB) – see flyer below, VJ London website and facebook for more details.
Hope to see lots of you there!
Happy New Year Everyone!
To celebrate 2016 we're running a special sale on VDMX through January 16th for a special price of only $249!
And as an added bonus, during the sale, students and starving artists can apply to get VDMX for as little as $99! Visit the buy page for more details, or send us an email with a scan of your school photo ID or a proposal for the starving artist rate.
This is also a great time to review some of your favorite tutorials or get started with learning the general fundamentals of VJing.
Slam Academy hosts DJ, Ableton and other music production classes.
This year the VDMX crew is heading back out to the wonderful Eyeo Festival and as a bonus this time we've lined up a special VJ hang out session hosted by our friends at Slam Academy.
The plan is for something informal – a chance to share some drinks, pizza and stories with other performing visual artists. And of course we'll have a projector set up for impromptu demos or jamming, so bring along laptops for showing off.
This event is free and open to the public – hope to see you there!
Date: June 1st, 2015
Time: 2-5 PM
