In support of an Open Source Pipeline

It was interesting to read both VFX Law’s proposed new business model for visual effects as well as VFX Soldier’s take on it this week.

Let’s assume that something like this will eventually happen and films will start setting up ad-hoc dedicated VFX facilities for each production. (And why shouldn’t it — it’s definitely doable.) We’re left with the need for, as VFX Law describes it, “a portable infrastructure that can roam from project to project.” This possible future self-organising VFX-creating creature will start from Day 1 at the same place as all the existing VFX facilities started: nothing.

But it’s the 21st century now, folks and that’s not necessarily a deal-killer. Workstations and fileservers are rentable now. Highly refined, production-capable software is available off the shelf. Photoshop, Maya, Nuke, and all the rest of the long list of content creation applications are here and have been for quite some time. Numerous robust render-farm managers exist. Production tracking used to be custom-rolled at each studio, but Shotgun (and their competitors) are making that available now for a monthly fee.

Here’s the gotcha: Want access to some fancy in-house tool developed by one of the big 5 houses? You’re just going to have to hire them. I’m not proposing we throw out the whole existing system here. Or maybe you’ve got enough leverage to convince them to sell their in-house magic to a third party for wider distribution and maintenance. It’s been known to happen.

Well, okay you say! It’s all well and good to buy software and plug in machines and tell the artists to get working, but commodity software and hardware does not a facility make. You’re right. It doesn’t. But there are an awful lot of customised tool extensions available in places like Creative Crash and Nukepedia and many others. Many of the tools on those sites were developed and shared by artists from big fancy FX facilities. All of that comes for free. What we need is a framework to plug it into.

Have you ever done any research into building a pipeline for a VFX facility? I have. And I’m here to tell you there is almost nothing available. At a previous gig, I was involved in building a pipeline from the ground up for a small studio. I had no prior experience with big fancy pipelines, so I was very interested to learn about the sweet things a proper pipeline should have. So I did my research. And I learned that while there have been years of SIGGRAPH papers published detailing all manners of image-making magic, there is absolutely nothing published that talks about the nuts and bolts of getting all the hardware and software in a facility working together. I had to gather what little info I could find by talking to ex-employees of other facilities and from sifting mailing lists for tiny golden nuggets.

This strikes me as strange. I mean — this is the fundamental problem that any start-up (or ad-hoc) facility will run into. It’s almost equivalent to the operating-system for a facility. And there’s nothing that exists to solve the problem? I guess that’s the problem. Everyone creates their own unique system. And none of them are compatible. This was okay when there were just a few shops doing VFX. But this isn’t the same world. Even romantic comedies have multiple shops working on them these days. Most of the summer blockbusters this year had a dozen or so VFX vendors. Facilities work together now. They need to be able to communicate.

So here’s my proposal: open up and standardize. We’re going to be working together, whether in small VFX shops or in production-specific ad-hoc facilities. We could make it easier on ourselves (and on our coordinators) if our stuff worked interchangeably. We can also get up and running in a new facility if we can grab a job management system, or a directory structure or some glue-ware for the repository and just run it instead of having to develop yet another custom piece.

Just like VFX Law’s proposed new-world VFX facility is only going to happen when a production decides to give it a try, an open pipeline is only going to happen when someone shares. So I’m sharing. Here is my Open Source Pipeline on GitHub. It’s very much in development right now. My goal with it is to first provide a basic framework for setting up shows and file structures and then figure out where to go from there. I’d love to have some help. Anyone interested?

5 comments

  1. I like this idea. I was thinking modular, but then what exactly is a modular pipeline?

    I guess i use the word pipeline a lot and I know what i mean, but i really don’t know how to go about building one.

    Does a modular pipeline even make sense in this context?

  2. Awesome piece! :)

    I look forward to the next one.

  3. Great work… looking forward for further insight into pipeline creation. Wish they come fast..

  4. This post is exactly what I’ve been looking for the past 6 months!

    ” there is almost nothing available. ” Your right, and this needs to change. Strange how so many papers can be written on all these VFX subjects, but nothing on the pipeline?

    I’m in my last year of my BSc animation degree. Writing a technical paper on “building a vfx pipeline”. And how hard isn’t that without much pipeline experience. I had 2 months in a big studio last summer, and now I’m building some pipeline tools for a small vfx company as my final uni project.

    I will have a look at your github, maybe I can help.

  5. Hey Tim,

    i like!

    I am currently developing many nuke tools, resulting in a pipeline, when all combined together.

    Some of the best tools i’m wrote are so specific, i couldn’t share them, because there’s no use without the infrasctructure and the task to use.

    i see the same problem in a open pipeline, ’cause special tools are paired to the specific production needs, but there are some basic setup approaches an osp has to met, esspecially to create a portable a nuke pipeline, with automatic setted up artist and dev interface, loading all scripts and refining tools on the load. (this is what my current pipeline work is all about) portable was not a goal, but i managed it to be very flexible, interacting with the os)

    I’ll have a look on github proj. Feel free to leave me a message.

Leave a Reply

Your email address will not be published. Required fields are marked *