Bioscoop
Creative coding with video

Historically, video editing has progressed from destructive techniques, ie. cutting the negative, to non-destructive ones, ie. non-linear editing. The latter has become possible due to film being no longer analog but digital. In both cases, however, the creative process is manual, or mouse-based. A different approach, often overlooked, is programmatic: the editor spells out the edits he wants to apply, and those are being carried out by an underlying system. FFmpeg is such a system; capable of carrying pretty much any editing instruction you can think of.

FFmpeg is embedded in virtually every major platform and application that handles media, yet its use in the creative coding community to make video art is limited. Most video creators prefer to stay away from the notoriously complex and error-prone string-based syntax. Indeed, its usability is hindered by the information density of its textual interface.

Bioscoop unleashes FFmpeg from its shackles and puts programmability in the center of the creative process.

The standout feature in terms of implementation and design is that Bioscoop does both standalone compilation and macro expansion without code duplication. Early in the project, as soon as the grammar for the language was defined, I realized that I didn't want to choose between an internal or external DSL. I wanted both.

Typically, external DSLs use a parser that produces a custom abstract syntax tree (AST), which then gets processed by a dedicated transformation pipeline specific to that parser. Meanwhile, internal DSLs use macros that directly generate target code without going through an intermediate AST structure. This creates two completely separate code generation paths - one for parsed text and another for macro-expanded forms - leading to duplicated transformation logic, potential behavioral inconsistencies between the two modalities, and significant maintenance overhead as changes must be applied to both transformation systems independently.

Traditional DSL Implementation*:

• External DSL: Parser → AST → Transformer
• Internal DSL: Macro → Direct Code Generation
• Problem: Transformation logic is duplicated.

In Bioscoop, a single transform-ast multimethod handles both external and internal DSLs. The external parser processes text strings into a parse tree, which is then transformed into AST nodes. The internal macro converts Clojure forms into the exact same parse tree structure, enabling both input modalities to share the single transformation logic that converts the parse tree into the final AST structure. This convergence at the parse tree level ensures behavioral consistency while eliminating redundant transformation implementations.

AST Convergence Approach:

• External DSL: Parser → AST → Transformer
• Internal DSL: Macro → AST → Transformer
• Solution: Single transformation logic.

AST Convergence occurs when both the external DSL parser and the internal DSL macro system produce identical abstract syntax trees, enabling a single transformation pipeline to handle both input modalities.

A camera-ready LaTeX paper (for conference submissions) is available in the repository. It elaborates further on the concept of AST convergence.

The project is structured around several key components:

In FFmpeg, atomic editing operations—such as scaling, cropping, blending, and color correction—are implemented as filters. With over 500 filters available, FFmpeg provides extensive transformation capabilities.

When multiple filters are applied sequentially to source material, they form a filterchain, written as comma-separated commands. Filterchains can be labeled at their input and output points, allowing one chain's output to serve as another's input. These interconnected filterchains create a directed acyclic graph (DAG) structure, which FFmpeg calls a filtergraph. Filtergraphs are passed to FFmpeg using the -filter_complex parameter.

That string-based syntax maps closely with the underlying libavfilter that parses it. What is happening is the following: the parser (libavfilter/graphparser.c) tokenizes the string, identifying individual filters, their parameters, and the connections between them.

Each filter name in the string (like scale, overlay, colorkey) maps to a registered filter implementation in libavfilter. FFmpeg looks up these filters in its internal registry and instantiates them as AVFilterContext objects. Key-value pairs within each filter specification are parsed and passed to the filter's initialization function, which validates and stores them in the filter's private context structure.

The parser creates an AVFilterGraph object and connects the instantiated filters according to the semicolons (filterchain boundaries) and labels in the string. Each connection becomes an AVFilterLink that defines data flow between filter pads.

FFmpeg validates the complete graph topology, checking that input/output pad counts match, media types are compatible (audio vs. video), and that there are no cycles.

• Filter names → AVFilter structs registered in libavfilter
• Parameters (key=value) → Filter-specific configuration passed to AVFilter->init()
• Filterchains (comma-separated) → Linked sequences of AVFilterContext nodes
• Labels ([label]) → Named AVFilterLink references for graph routing
• Semicolons → Graph branching points that create multiple parallel paths

This one-way translation from string to internal structures is the core problem: there's no inverse mapping. Once parsed, the AVFilterGraph exists in memory, but there's no standardized way to serialize it back or manipulate it programmatically before the string parsing step. Developers must work in strings because libavfilter's graph construction API, while programmatically accessible, is complex and poorly documented compared to the string syntax.

The string format is essentially a convenience layer over libavfilter's C API—and it's become the only practical interface, despite its limitations.

This creates several challenges:

No programmatic structure: Unlike many modern tools that use JSON, YAML, or object-based APIs, filtergraphs cannot be easily constructed, validated, or manipulated programmatically. There's no schema to reference, no type checking, and no ability to introspect the graph structure before execution.

String concatenation dependency: Building dynamic filtergraphs requires manual string concatenation, making the code fragile and error-prone. A single misplaced comma, semicolon, or bracket can break the entire pipeline, with errors only surfacing at runtime.

Limited tooling support: Because filtergraphs lack a formal representation, IDEs cannot provide syntax highlighting, auto-completion, or validation. Developers must memorize the syntax or constantly reference documentation.

Debugging difficulty: When a filtergraph fails, error messages reference the string position rather than logical components, making it hard to identify which filter or connection caused the problem.

This string-only representation means filtergraphs are essentially "write-only" code—difficult to read, maintain, and programmatically generate at scale.

The DSL accepts a Lisp-like syntax with the following core constructs:

(scale 1920 1080)
(crop "iw/2" "ih" "0" "0")

(color {:color "blue" :size "1920x1080" :rate 24 :duration "10"})
(drawtext {:text "Hello World" :x 100 :y 50 :fontsize 24})

program = expression*
expression = atom | list | let-binding | map | compose | graph-definition | padded-graph
let-binding = '(' 'let' binding-vector expression+ ')'
graph-definition = '(' 'defgraph' symbol expression* ')'
padded-graph = '[' label+ expression* label+ ']'
compose = '(' 'compose' expression+ ')'
list = '(' expression* ')'
map = '{' mapentry* '}'
atom = number | boolean | symbol | string | keyword

The acts of the mind, wherein it exerts its power over simple ideas, are chiefly these three:

1. Combining several simple ideas into one compound one, and thus all complex ideas are made.
2. The second is bringing two ideas, whether simple or complex, together, and setting them by one another so as to take a view of them at once, without uniting them into one, by which it gets all its ideas of relations.
3. The third is separating them from all other ideas that accompany them in their real existence: this is called abstraction, and thus all its general ideas are made.

—John Locke, An Essay Concerning Human Understanding (1690)