IC Lip-Sync Contest 2021 — Cop Jokes Short Film
#iClone, #AccuLips, #Lip-SyncAnimation
Cop Jokes Short Film
Ever since I made my previous short film with a play on the Mandalorian, I have been thinking about how to make my next one. The most difficult part of the previous short was getting the voices and lip sync for iClone to render in Unreal Engine.
When Reallusion and Replica introduced this challenge I was super excited. I wanted to see how I could adapt the script and idea I had so AI characters could play the parts.
I actually started with my script. I had two characters. I knew they wanted to be cops. I started to think about the personalities of these two and began searching the Replica database for the best voices.
I ended up selecting Tony for the witty one and Trey for the more serious. Kind of an odd couple to be paired up together. Create some tension, right. I feed my script into the software and tweaked it until it was just right for each line. The key is to play around with the spelling. You can make the AI create a pretty compelling voice if you exaggerate some characters here and there.
After I had these voices down and scripted out, I left Replica Studio for a while for Character Creator 3. I needed two cops. I started to work on Tony first. Decided to just run with his Replica name.
Character Creator 3 is the go-to software for creating realistic characters. It’s just so intuitive and the outcome is unreal. I created the Tony character then started to work on Trey (also Replica name, so creative).
Now I wanted to get these two some uniforms so I started to hunt around and found this police officer on Sketchfab. His scale was much larger than my current characters so I had to bring them into Blender and match up the uniform with my character then back out to do the skin weights in CC3.
With both characters created, it was time to focus on the scene. I wanted an evening or night scene. Just wanted to have a slight twist at the end of the short and I think that needs to be that time of day. I also wanted to use Unreal Engine 5 to create the short. Wanted to try out the Lumen lighting as well as play around with some nanite objects. More on that later.
So I set up my scene. Added all the usual suspects to create the environment then started to look into the streetscape. I started with the road spline. I created a small section of road that I could use as a stretch of downtown road. Using the Brushify urban buildings pack I started to populate some quick buildings around the scene. Pretty random as I just needed to create what felt like a solidly built environment at the street level.
After I had the rough framework laid out I felt like the streetscape was not giving off the right vibe. So once again took to Sketchfab. I have been following Alban’s 3d scans of the streets in NYC and decided to use a couple of his scans. Also curious how the iPhone lidar scans would work in UE5. I brought these straight into the engine without doing any work to them at all.
I did the same for the Streetlights, the City trash can, and the Fire hydrant. All Sketchfab assets. After getting all these in the streets felt empty still, so I found one last object from Alban’s scans. The Bus. I think it along with all the graffiti adds a real mysterious look to the scene. I could continue to build this out and tweak the results but at some point, you have to animate.
For the character animation, I went back to CC3 and exported both characters out to iClone7. Once in iClone, I needed to position the police officers into the vehicle, but it was in UE5 so I brought the car into iClone. With the car in iClone I was able to work with the characters in the correct positions, they would be in for the animation. Mainly seated.
Now it was time to bring Replica Studios (affiliate link) back up and start to transfer the script over to the characters. Just working through the timeline I applied the voices and iClone did the rest. After I had the rough track I went in and added more facial animation and head, eye movement to try to bring the characters to life.
Since the Reallusion Unreal Livelink plugin is not yet out there for UE5, I exported the characters with animation to fbx files with the Unreal settings. This is where I ran into a slight snag. When bringing the character into UE5 as an fbx the eyes were getting blacked out. No matter what I did to the eye materials I would not get them to show up, so I deleted the characters and imported them into UE4 using the plugin and once in migrated the files over to my UE5 project. After doing that they worked perfectly. I then imported only the animation without the mesh of each character using the RL skeleton of each character that was migrated over.
Sounds like a lot of work, but it was a pretty easy workaround!
Now it was camera work and sequencer from here on out. I started by creating a new level sequence. I brought my various cameras in and animated the cuts adding a camera cut track to the editor. Imported my audio script file exported from iClone to sync up the animations. Added Trey and Tony to the sequencer and did slight adjustments.
Then rendered out to an AVI at 1920x1080. I wanted to export the film to 4k but was having issues with flickering and lighting at that resolution for some reason. After the footage was exported I added all the audio and sfx in premiere to complete the short film. You can watch the entire film (all 30 seconds of it) here.
