Today we are excited to announce the open source release of our latest codebase for working with Metal and rendering ISF shaders on macOS: ISFMSLKit
This codebase was initially written for the VDMX6 update last fall, and is now ready for use by other developers who are looking to support ISF in a Metal environment on the Mac. Along with this we’ve also open sourced VVMetalKit, our general purpose Metal framework which powers the rendering engine in VDMX6 and other utility apps.
What is ISF?
Before we dive into ISF for Metal, let’s do a quick review of the ISF specification itself.
ISF, the Interactive Shader Format, is a cross platform standard for creating video generators, visual effects, and transitions. It was originally created for use in VDMX in 2013, and is now supported in over 20 different apps and creative coding environments.
ISF files contain two main components:
JSON Metadata
GLSL Code
These are used together by host applications to provide user interfaces to control parameters within the rendering pipeline.
An example ISF generator as code and as a source in VDMX.
The ISF documentation pages include a Quick Start overview, an in depth look at writing GLSL code for real-time visuals, and useful reference notes.
Getting Started with ISF in Metal
This new open source release has three main repositories:
ISFMSLKit: ISFMSLKit is a Mac framework for working with ISF files (and the GLSL source code they contain) in a tech stack that uses Metal to render content. At runtime, it transpiles GLSL to MSL, caches the compiled binaries to disk for rapid access, and uses Metal to render content to textures. Everything you need to get started with ISF shaders in a Metal environment.
VVMetalKit: VVMetalKit is a Mac framework that contains several useful utility classes for working with Metal, such as managing a buffer pool, rendering, populating, converting, and displaying textures.
ISFGLSLGenerator: ISFGLSLGenerator is a cross-platform c++ lib that provides a programmatic interface for browsing and examining ISF files as well as generating GLSL shader source code that can be used for rendering. This library doesn't do any actual rendering itself - it just parses ISF files and generates shader code.
These frameworks are the same code that we have running under the hood in VDMX6 and are free for anyone to use in their own software.
The ISFMSLKitTestApp running the Color Bars generator being processed by the Kaleidoscope filter.
All of the code needed for working with ISF in Metal can be found in ISFMSLKit.
This repository has several dependencies that are included, such as VVMetalKit and ISFGLSLGenerator.
The ISFMSLKitTestApp demonstrates the entire process of working with ISF under Metal:
Converting GLSL to MSL.
Caching converted shader code.
Loading, validating, and parsing ISF files.
Rendering ISF shaders to Metal textures as generators and FX.
Displaying the final image in a Metal view.
The three major classes you’ll be working with in ISFMSLKit are:
ISFMSLDoc: This class is a programmatic representation of an ISF document, and can be created either from the path to an ISF document or by passing it strings containing the ISF document's shaders. It performs basic validation, and parses the ISF's contents on init, allowing you to examine the ISF file's attributes. Behind the scenes, this class is basically a wrapper around VVISF::ISFDoc from the ISFGLSLGenerator library.
ISFMSLCache: This class is the primary interface for caching (and retrieving cached) ISF files. Caching ISF files precompiles their shaders, allowing for faster runtime access.
ISFMSLScene: A subclass of VVMTLScene used to render ISFs to Metal textures.
Detailed information on these classes and how to use them can be found in the ISFMSLKit readme file and documentation.
How does ISF in Metal work under the hood?
As ISF v2 is an OpenGL / GLSL based specification, in order to get ISF files working in Metal, first they need to be transpiled from GLSL to MSL. In a traditional OpenGL pipeline, behind the scenes ISF looks essentially like this:
This is where ISFMSLKit and SPIR-V come into play. SPIR-V is an intermediate format from The Khronos Group for converting shaders from one format to another. To convert from GLSL to MSL, this happens in two parts:
GLSLangValidatorLib which converts GLSL to SPIR-V.
SPIRVCrossLib which converts SPIR-V to MSL.
These are provided as precompiled binaries in ISFMSLKit to reduce compilation times, but the source is also available if you are curious to see how they work, or you'd rather build your own.
Our updated diagram with the conversion to SPIR-V and MSL in place of GLSL is now something like this:
When using ISFMSLKit inside of VDMX6 or the included ISFMSLKit sample application, on launch, every unconverted GLSL-based ISF file is first converted to MSL, and then tracked for changes during runtime. The original ISF is left intact. Error logs are generated for files that fail to compile or translate for whatever reason. Files that successfully transpile are cached for later use.
Example error log for a GLSL shader that fails to compile.
The process demonstrated in the ISFMSLKit repository is also a great starting point for developers looking to convert ISF files to other shader languages for other platforms, or as a reference for anyone looking for an example of SPIR-V in action.
Once the ISF files have been converted from GLSL to MSL, they are ready to be validated, parsed for input parameters, and rendered by the host application using ISLMSLKit or your own custom library.
In addition to providing the base for the ISF rendering classes in ISFMSLKit and its sample application, the new VVMetalKit contains a variety of general purpose utilities for working with Metal. New and experienced developers alike will find this framework useful when creating Metal based video apps on the macOS.
Learning more about ISF
More information about ISF can be found on the isf.video website and examples can be found on the ISF Online Editor.
The set of open-source ISF files that is bundled with VDMX has over 200 shaders including useful utilities, standard filters, and a wide variety of FX for stylizing video, is completely free for other developers to use.
We are always excited to hear from people using ISF – if you have any questions, feedback, or just want to send us some cool links to your work for us to check out, please send us an email at isf@vidvox.net.
Developers who add support for ISF to their own software can send us a URL to their website and png icon to include on the list of apps that support ISF.