The next Tuesdays, The edgeflagship podcast of The Vergecast presents a mini-series devoted to the use of artificial intelligence in often neglected industries, moderated by Edge senior reporter Ashley Carman. This week the series focuses on AI for the video world.
Specifically, we take a look at how AI is being used as a tool to help people streamline the process of creating video content. Yes, that could mean that software is playing a bigger role in the very human act of creativity, but what if instead of replacing us, machine learning tools could be used to help us in our work?
This is what Scott Prevost, vice president of Adobe Sensei – Adobe’s machine learning platform – envisions for Adobe’s AI products. “Sensei was founded on the firm belief that we have that AI will democratize and amplify human creativity, but not replace it,” says Prevost. “At the end of the day, allowing the creator to do things that he might not be able to do before. But also to automate and speed up some of the mundane and repetitive tasks that are part of creativity.
Adobe has already incorporated Sensei initiatives into its current products. Last fall, the company released a feature called Neural Filters for Photoshop, which can be used to remove artifacts from compressed images, change the lighting of a photo, or even change a subject’s face, giving it a smile instead of frowning, for example. , or by adjusting their “face age”. From a user’s perspective, this is all done by simply moving a few sliders.
Adobe also offers features like Content Aware Fill, which is built into its After Effects video editing software and can seamlessly remove objects from videos, a task that would take hours, if not days. to be done manually. Prevost shared the story of a small team of documentary filmmakers who had issues with their footage when they realized there were unwanted stains on their visuals caused by a dirty camera lens. With Content Aware Fill, the team was able to remove unwanted blemishes from the video after identifying the object as a single frame. Without software like Adobe’s, the team would have had to edit thousands of images individually or redo the footage entirely.
Another feature in Adobe called Auto Reframe uses AI to reformat and crop video for different aspect ratios, keeping important objects in the frame that may have been cropped using regular static cropping.
The technology in this area is clearly advancing for consumers, but also for professionals with a big budget. While AI video editing techniques like deepfakes haven’t really hit the big screen yet – most studios still rely on traditional CGI – the place where Hollywood directors and studios are. on the way to using AI is for voice acting.
A company called Flawless, which specializes in VFX and AI-powered movie making tools, is currently working on something they call TrueSync, which uses machine learning to create lifelike, lip-synced visualizations for actors to many languages. Flawless co-CEO and co-founder Scott Mann said The edge that this technique works much better than traditional CGI to reconstruct the movements of an actor’s mouth.
“You form a network to understand how a person speaks, so the mouth movements of an ooh and aah, different visemes and phonemes that make up our language are very person-specific,” Mann explains. “And that’s why it takes so much detail in the process to really get something authentic that speaks like this person spoke.”
One example Flawless shared that really stood out was a scene from the movie Forrest Gump, with a dubbing of the character of Japanese speaking Tom Hanks. The emotion of the character is still present and the end result is much more believable than a traditional overdub because the movement of the mouth is synchronized with the new dialogue. There are times when you almost forget that this is another backstage voice actor.
But as with any AI changing any industry, we also need to think about replacing jobs.
If someone creates, edits, and publishes projects on their own, Adobe’s AI tools should save them a lot of time. But in large production houses where each role is delegated to a specific specialist – retouchers, colorists, editors, social media managers – these teams can end up downsizing.
Adobe’s Prevost believes technology is more likely to displace jobs than destroy them altogether. “We think that some of the work that creatives used to do in production, they won’t do as much,” he says. “They can become more like art directors. We believe it actually allows humans to focus more on the creative aspects of their work and explore this larger creative space, where Sensei does some of the more mundane work.
Scott Mann at Flawless has a similar sentiment. While the company’s technology may reduce the need for screenwriters for translated films, it can open the door to new employment opportunities, he argues. “I would say, honestly, that this role is kind of a director. What you are doing is trying to convey that performance. But I think with technology and really with this process, it’s going to be about taking that side of the industry and developing that side of the industry.
Will screenwriters end up becoming directors? Or do photo retouchers end up becoming art directors? May be. But what we see for sure today is that many of these tools are already combining workflows from various points in the creative process. Audio mixing, coloring, and graphics all become part of the versatile software. So if you work in the visual media space, instead of specializing in specific creative talents, your creative work may instead require you will be more of a generalist in the future.
“I think the lines between images, video, audio, 3D and augmented reality will start to blur,” says Prevost. “There used to be people who specialize in stills and people who specialize in video, and now you see people working in all of those mediums. And so we think Sensei will have an important role to play in basically helping to connect these things together in a meaningful way. ”