Even though I got into After Effects to make widgets for live streaming, that idea has fallen by the wayside in favor of learning how to do visual effects with real video. One effect that seems to be overwhelmingly common in video is the placement or replacement of elements in ways that make them look natural. For example, imagine that your actor is sitting in a coffee shop, ubiquitous Macbook open to the camera so we can see the blazing white Apple logo. Since we don’t have permission to use the Apple logo, but still want to imply that this is an “Apple-like” device, we’d have to put our own custom icon over the Apple logo. We could do that for real by slapping an opaque sticker over the logo, but if the intern forgot to do that, it’s up to the effects folks to dodge a hefty lawsuit by using planar tracking.
Although After Effects can do point and camera tracking, AE CC 2019 ships with a stripped down version of BorisFX’s planar tracking plugin, Mocha CC 2019.
Mocha’s job is to track the motion of elements in the scene that have been traced by the VFX artist. What we’re looking at above is a way to use Mocha to replace the contents of a TV screen with a custom video that wasn’t actually on the TV at the time. The image above shows the post-tracking results with a test logo in place. You can see how the grid seems to lay across the plane of the TV, and while it doesn’t look like much, as this is a video where the shot is less than stable — on purpose in order to test Mocha’s tracking ability. Tracking will allow our inserted video to move with the TV as it it were recorded “live” without having to constantly pause and update position by hand.
Note, however, that Mocha is not intended to be used for reskinning non-planer elements in a video. If we had a soda can on a turntable that spun slowly over time, Mocha would not be the tool we’d need to apply new content to the outside of that can.
I spent a lot of time on the BorisFX site looking through their tutorial videos, as well as their YouTube channel. My first stab at this process involved a powered-down TV which didn’t work so well, and it was through these videos that I realized my error.
The TV video above was shot in my computer room using my Pixel 3 phone at 1920×1080, 29.97FPS. The Guild Wars 2 video was recorded as part of my long-delayed tutorial on how to extract your character using third party post-processing software.
Let me start with a caveat for my future self. The first time I tried this project using the same TV, it failed miserably and I couldn’t figure out why. Because I didn’t want to have to deal with extracting content from the TV, I had recorded the scene with the TV turned off, which I ultimately learned was a mistake. As Mocha was tracking the shaky motion of the scene, the parallax reflection of my computer room in the TV’s screen was causing Mocha’s tracker to lose its mind, resulting in an extremely poor result.
Once I understood this, I decided to fire up Photoshop and create a tracking image.
I dragged this image into Chrome, and used the browser to display the image via the Chromecast on the TV.
This was enough to make Mocha happy when tracking.
I used the X-Spline tool to outline the greened portion of the display, and had Mocha track forward for the duration of the clip. The results were pretty solid.
Next step was to define the planar boundaries using the additional bounding box tool.
With the screen area now suitably marked off and tested using the Mocha logo (as seen above), the UI was closed and the tracking data was automatically sent back to After Effects.
At this point I got tripped up a bit. AE had the data — sort of. This effect properties panel was for the Mocha effect applied to the TV layer in the timeline stack, but I needed to have the tracking data applied to the GW2 video because that’s what needed to be cropped and pinned to the TV. The steps were these:
- Click Create Track Data.
- A new window will pop up:
This window was asking which layer from Mocha contains the tracking data. I hadn’t renamed it, so Layer 2 is correct. However, I didn’t know I had to click the column beneath the gear in order to select the layer — clicking the layer doesn’t do what needs to be done.
Once that was done I set Export Option to Corner Pin (Support Motion Blur) even though nothing was moving fast enough to blur, and selected the GW2 layer for Layer Export To. When I pressed Apply Export, the scene took shape.
And as you can see in the video above, the GW2 video tracks in its place even as the camera shake moves the TV around a bit.