top of page

Back End Systems

Writer's picture: Tom MikotaTom Mikota

Updated: Dec 27, 2024


What you need to automate the way we do at CG Lumberjack.
What you need to automate the way we do at CG Lumberjack.


What’s Actually Required to Operate a Studio at Peak Efficiency?

Recently, I wrote a LinkedIn post that gained notable attention (at least for a pipeline-focused post). The core message was this:


The Animation industry has a vocabulary problem.

We use one word—“Pipeline”—to describe the incredibly complex ecosystem required to produce films. By contrast, industries like manufacturing have clear terms for their processes:

  • Hand Tool

  • Power Tool

  • Machine Tool

  • Assembly Line

  • Factory

  • Delivery Truck

  • Supply Chain


Even outsiders generally understand these concepts. Yet in film production, everything—no matter how nuanced—is called "Pipeline."

For example:

  • Hand Tool → Pipeline

  • Power Tool → Pipeline

  • Machine Tool → Pipeline

  • Assembly Line → Pipeline

  • Factory → Pipeline

  • Delivery Truck → Pipeline

  • Supply Chain → Pipeline


This lack of precision makes it challenging to even define a roadmap for improving studio operations. Ask ten people in a studio what “Pipeline” means, and you’ll get ten different answers.


--------------------------------


Defining the Systems of a Studio

When I recognized this issue, I also realized I didn’t have clear definitions for the essential systems in a studio or production ecosystem. This insight inspired me to write a book.

By analyzing my work on Alchemy—the system I’ve developed over 15 years—I identified and catalogued these systems. Below is a high-level overview of what’s needed.

Note: These terms are evolving, and I welcome feedback from others in the industry. Each system will eventually have detailed documentation linked here.


A Warning: Building This From Scratch Isn’t Cheap

The order below reflects how I would structure these systems if starting from scratch. However, I don’t recommend doing it all at once. Developing these systems has cost over $5M and taken 15 years to refine. Even with a clear roadmap, expect millions in development costs—plus the lost opportunity cost of inefficiency.


Opportunity Cost Example:

A studio with $10M in yearly revenue building these systems over three years might spend:

  • $5.6M per year on inefficiencies: artists searching for files, debugging issues, and stumbling through outdated workflows.

  • You're $15M just in lost efficiencies over 3 years not to mention the likelihood of spending $1.5-3M on development.


The takeaway?


Speed to delivery is everything. Investing in a pipeline isn’t just about saving money—it’s about enabling your studio to thrive.


Phase 1: Base Systems:


These are the systems you need to get going. Most studios don't currently have them. I still maintain that this is the baseline that you need to even start to automate anything. If you don't have these systems you are creating more work for yourself every time you write new code.




Software Stack


This is simply the tools you choose to purchase to operate your studio on. Maya, Unreal Engine, Nuke, etc... Your Software Stack will determine the foundation of your pipeline. And a lot of people call the choice of software a studio uses "the pipeline". This is a problem you've solved even if you're starting out as a blender artist.


Custom Tools


One of the first things people start to equate to pipeline is custom python scripts that help an artist do some task faster. Most studios never graduate from this. The entire pipeline is just one custom tool after another, many of them doing very similar things. Even with a full pipeline custom tools have a place, there's just not a huge need for them.


Git/Bitbucket


As soon as you start wanting to develop code you should be managing it with Git or Bitbucket. This is a non-negotiable if you want any kind of control over your code.


CI/CD


Continuous Integration and Deployment. You absolutely need systems that will automatically handle things like:


Unit tests on core pipeline code

Automatic Deployment of code to development servers and production servers

Automated Roll Back policies when things inevitably break


The best use case for this is - someone releases code - there's a bug somewhere - a developer should be able to fix the bug and get the code released within minutes of the bug being reported. You can only do this with CI/CD systems in place.


At the last studio we set this up for they went from releasing pipeline code once a month to releasing pipeline code multiple times per day.


File Management


Your file management system handles folder structure, file naming, and ensuring everything created in the pipeline follows the same standards. This is typically implemented as a python based set of modules for managing file patterns.


User workspaces


Your file management system needs to be able to handle the concept of user workspaces - folders with the user's name on them that allows them to work outside the publish area and ensures that files can be traced to the person who created them.


Version Control


Studios need two types of version control for files. Traditional version control where versions are tracked in numbered folders and version control like what you'd find in perforce. I've seen too many mishaps when studios try to do everything within perforce, so by default i recommend studios have version control built into their folder and file management systems.


Publish


Publish systems at their core simply mean that there is a predictable place to find files that have been approved for downstream consumption. You can't automate any kind of build process until you have predictable published locations for all Production Tasks in the studio.



Phase 2: Team Systems

These Systems Come into play when you're working with teams. Honestly i use them when working solo as well just to keep myself organized.




Project Management


Task Management


Artist GUI


Render Management


1 Click Reviews


Validation API



Phase 3: DCC Management

I don't know anybody that's built anything like this, but it's absolutely necessary to being able to manage a studio with one, or very few developers.




Custom Menu API


DCC Plugin API


Production Task API



Phase 4: Developer Systems & Workflow Automation

DCC management is honestly the first developer systems, but i section this off differently because you can sneak by with just raw code up until here. That's what i did when building Alchemy. Then you realize that you can manage things MUCH easier with some nice developer interfaces and some other really nice systems built on top of the stuff you built in the DCC management phase.




Workflow API


Developer GUI


Publish System


Render System


Build System


Review System


Start System


Refresh System


Phase 5: Scene Descriptions

This is hard if you don't know what you're doing. And it's hard to do in a scalable fashion. That's why i recommend waiting until phase 5.




Unified Shading System


Internal Scene Description


USD Implementation


Phase 6: Unreal Engine


Unreal Engine is a beast to get working at scale. It's incredibly good with experienced operators and very small teams. But when you want to work on it at scale with 20+ people it starts to break down very quickly. Especially Perforce Management.





Unreal Engine Integration


This is insanely hard. I spent 18 months in around 2019 on an experiment while working at the FSU film school as the head of a R&D Lab for Virtual Production. I wanted to know exactly what hooks I'd need in Unreal Engine in order to take a short film created in maya and rebuild the entire thing in Unreal Engine with the click of a button. Unreal is just built differently. It has the potential to create an entire film inside of it, and becomes unwieldy to manage. I wouldn't attempt to do this unless you've already got ALL of the systems listed here. Once you have that it's fairly straightforward and you can do what we've done with Alchemy - adjust it to work inside Unreal Engine and treat it as it's own file system.


Perforce Integration


Working with unreal engine without automation around perforce is a nightmare. Therefore you'll need command line tools that you can integrate into your artist's interface. This will make it easy to work with perforce. We've handled it much the same way as an IDE like PyCharm handles it. Simple icons to show the status of the Depot, and automations under the hood that connect to things like creating a new "location" for the film. We've developed these systems along side story departments.


Phase 7: Advanced Analytics


Once you have a well oiled machine that has all of these systems the next natural evolution is to create analytics around what you're doing so you can track it. This is dead last because without all these other systems you don't have accurate information to analyze anyway. The other reason is that when you do things like automate workflows you're saving SO much time there's not a lot of point to track it, you can see the savings in real time if you are developing and delivering fast enough.




10 views0 comments

Recent Posts

See All

Comments


GSL Newsletter

See it First

© 2035 By GSL Productions. Powered and secured by Wix

  • Instagram
  • Vimeo
  • YouTube
  • Facebook
  • Twitter
bottom of page