Ben's Comp Newsletter

Special Issue 05 -- AI Developments in Compositing.

Hey ,

AI in VFX is a controversial topic. I'm personally excited by it, as I don't believe it aims to replace artists. I see it as another tool to help us remove the friction in our creation process & it enables us to create better, more-unique imagery.

There are lots of tools in the wild aiming to automate things like roto and cleanup, but they're still quite a way off from being useful for final pixels. There are others that have scary implications outside of entertainment use-cases...

In this newsletter issue, I want to focus on the positive, and shine a light on some Compositors who are using these new tools in a productive and inspiring way.

Enjoy!

Freddy Chavez Olmos.

Freddy is a Compositor, Director and Filmmaker, and one of the most inspiring guys I've had the opportunity to work with.

A few months ago, he wrote a blog post about his rapid journey from being interested in AI, to how he levelled up in his career using the currently available tools.

There are plenty of great examples to be inspired by...

Check out Freddy's Blog Post.

Pranjal Choudhary.

Casting interactive light from CG elements onto plate photography is something we do all the time. Whether it's muzzle flashes, fire/explosions, or magical effects -- there are plenty of shots that require it.

Whether we're integrating IL passes from lighting, or manually rotoscoping & grading areas of a plate, it's always a time-consuming process.

To solve this, Beeble has released a free* app called Switchlight, which allows you to relight any image with an HDRI. It generates an albedo, roughness and normal map from your image in order to do this.

Pranjal has taken these parts, and utilizes CopyCat to generate a temporally-stable relit image.

Check out Pranjal's YouTube Tutorial.

For most users, Switchlight is online-only, which means we're unable to use it in production due to the sensitive nature of the content we're working with. They just released an API to open up use in different avenues (still via their servers), so it seems a Nuke plugin might not be out of the question in the future.

Mads Hagbarth Damsbo.

Mads is one of the guys who keeps doing innovative things with the available technology in Nuke. In this video, he demonstrates how you can manipulate facial performances with KeenTools & CopyCat.

Check out Mads' Video

(Skip to Timecode: 10:30)

Creating a 3D Hole using Nuke + Photoshop's Generative AI.

Alex Hanneman created this tutorial which is a great example of combining the new (generative AI) with the old (camera projection in Nuke).

There are a lot of implications to be made about the impact this could have on speeding up DMP / Environment work.

Perhaps this approach could be combined with NeRFs for fast & accurate clean plate generation?

Check out the Tutorial.

ChatGPT for Compositors.

Like many, I'm a big fan of ChatGPT. While we can't utilize it for summarizing dailies notes or emails (client content security), we can use it to remind ourselves of the things we've forgotten.

The best way I've found to prompt ChatGPT is by adding context & relevant detail. For example:

• "I'm a Compositor using Nuke. Please provide a Python script which will remove all nodes from my nuke script that don't contribute to the final output of my Write node."
• "I'm a Compositor using Nuke. What expression should I use to retime an animation curve on a knob?"

Adding that first sentence improves the accuracy of the responses I get. Being specific about what I'm trying to do, of course, clarifies the answer I get as well.

Try asking ChatGPT the same question in different ways, with different levels of context. You'll quickly find how to fine-tune the results to return what works for you!

Secondly, I use ChatGPT for a lot of non-Nuke things! While content security prevents us from summarizing dailies / meeting notes, it can still prove useful to sharpen the clarity of an email, or make suggestions for how to prioritize your work:

• "Below is an email I've written. Please shorten it to one small paragraph, change the tone to be more professional, and increase the clarity of the message.

<email pasted here>"

• "Help me prioritize the following three tasks. Ask follow-up questions to help your decision-making process."
• "I want to learn X, please provide 5 of the best tutorials which will get me up to speed quickly."

With these examples, we get the opportunity to provide follow-up prompts & greatly improve the results we get.

Wrapping up.

There is a lot happening in the AI space! I'm subscribed to the Ben's Bites Newsletter, which is a daily digest of the past 24-hours of AI news. While some of it is directly related to computer graphics, I get a lot of value reading about how other companies are using AI, and thinking about how those ideas could be translated to our industry...

For example:

• Superpowered.me can be hooked up to Google Meet, Teams & Zoom, and will summarize meeting notes for you.
• Vimeo released a text-based video editor, complete with options to detect and delete filler words and quiet spaces.
• HeyGen Ultra-Realistic Avatar to create personalized videos. I signed up to the waitlist, and received a video that repeated back some of the answers I gave in the Typeform survey. Pretty cool technology, but has some scary implications...

This is a big topic, hence the longer-than-usual newsletter. If you've seen other inspiring examples of AI being used in VFX, specifically by Compositors, please reply to this email and share them with me! I'd love to hear what's on your mind, and what you're excited about.

Happy Comping,

Ben