In a nutshell
- AI technology is advancing rapidly and is already being used in the visual effects industry.
- While AI is not yet advanced enough to generate full 4K accurate visual effects shots, it can be used in specific capacities such as background replacements and rotoscoping.
- The development of AI tools for visual effects has the potential to improve the speed and quality of visual effects. This will make high-end-looking, visually impressive results more accessible to low-budget filmmakers and film students.
It goes without saying: Advances in visual effects have progressively improved the quality of the media we consume. Filmmakers can now create otherwise impossible scenes for their projects. In turn, audiences have experienced some of the most memorable moments in cinema history. Along with the development of visual effects, there also comes a new form of technology that has everyone wondering when the two will combine. That technology is artificial intelligence (AI).
In this article, we’ll explore this question and explain the benefits of using AI in the creation of visual effects.
What does this mean for the visual effect industry?
We, like many of our visual effects colleagues, have been reading daily news about AI and wondering what this means for our jobs. The truth is, it’s hard to say. It could mean the end for all of us in the visual arts world. Or, it could be a valuable tool to make our lives easier. We hope for the latter.
Currently, AI is not advanced enough to generate full 4K accurate visual effects shots — not yet, at least. However, it is good enough to be used in a specific capacity within VFX.
The power of AI as a VFX artist
Apart from being a VFX artist, I, the author of this article, also take on the role of VFX editor on feature films and TV shows. In short, a VFX editor keeps track of the number of VFX shots in a project and creates temporary shots for the director and editor to use as a guide until the vendors start doing their work. Often, a VFX editor needs to put together concept shots to compile together as a rough guideline to help sell the idea of the scene. This can include things like background replacements or environmental images.
There have been many times when I have been personally asked to find a specific background for a greenscreen shot. Usually, the background the production needs doesn’t exist and isn’t easily found with a simple Google search. This makes me endure an hour or so of frustration and leaves me wondering why no one has ever created an image like the one I need. However, recently, I encountered the first instance in my professional career where AI came to my rescue.
I recently created a temporary background using an image generated by the AI tool Midjourney. The resulting shot was used for the majority of the cut. It even ended up being a template for the vendors to use when generating the full CG render. Below is an image I created with Midjourney to show you the creative possibilities this tool gives artists.
Evolving rapidly
AI is rapidly advancing in image generation. It’s constantly improving quality and its ability to understand user requests. This advancement will allow films and TV shows with lower budgets to have amazing-looking visuals much more easily. However, full CG-level animated renders have yet to reach this level of development. This includes character animations, rigid object animations, environmental simulations and live background building/tracking. This is not to say that there are no AI tools that can do basic versions of these types of renders. I have seen sneak peeks of some in development, but they are still at a basic level. They cannot compare to the work of a skilled animator paying attention to all the details involved. Moreover, human reviews and creative discussions among directors, editors and producers are still necessary to bounce ideas off of each other to reach their vision. That’s something that AI cannot do.
The possibilities of AI in visual effects
But there is one tool being developed by a friend of mine that I know will be a complete game-changer. Any VFX artist will dread the word “rotoscope.” It’s a task that, on many occasions, will involve complex masking and animating of that mask to clean out a scene. This could be either to single out an actor or prop or to clean up any background elements in the shot. Rotoscoping can be a very time-consuming task, so anything that would speed this up will be a widely welcomed tool.
This is where my friend’s software comes into play. It’s a tool that uses AI to identify and rotoscope an area of the shot that the user has selected. It will allow you to draw out where you want to roto (for example, an actor performing a stunt and needing the stunt wires removed) and will go by frame by frame and mask out that actor. Currently, this tool is still in development. That means there will still need to be some user cleanup following the processing. However, this is a small price to pay for time efficiency.
When looking at these tools and the speed at which they are developing, there really isn’t any doubt that within the next few years, we will see a large rise in the high-end-looking, visually impressive shows and films at a faster turnaround rate due to the use of AI tools. Not only this, but the options for low-budget filmmakers and even film students will grow beyond what once was imaginable.
Is AI a threat to VFX?
But as we mentioned earlier, many VFX artists might be wondering if AI threatens their jobs. The real answer is: We can’t say. But one bit of advice we can give as artists ourselves is that rather than being threatened by AI, we should learn how to embrace it and use it to benefit us. Many artists can use AI to speed up their workflow and create shots for their clients that they might not have been able to before. It also opens up the option for more in-house artists and concept compositors to help filmmakers better visualize what they are looking for in their projects.
The future of AI in visual effects
To conclude, artificial intelligence definitely has a place in the future of all filmmaking disciplines, not just visual effects. Due to the nature of AI and how it often times teaches itself, the development possibilities are virtually endless. With the ever-growing demand for content, there will always be a place for it. All we can do now is follow it along and do our best to control it rather than oppose it.
