Screen Tracking and Replacement Techniques in After Effects

Screen replacement is the process of taking a device screen in a video, such as a phone, laptop, or monitor, and swapping the original screen content with a new graphic. In many cases, the original footage is shot with a green screen inside the device display, so the screen area is easy to isolate and replace. Once the screen is tracked, a new image or video can be substituted so it appears locked to the device as it moves.

This technique is especially common in product demos where a company wants to showcase an app on a phone. It is also widely used in film and television, including sci-fi interfaces where multiple screens need to look animated and believable. The tracking method used for this is often described as corner tracking, four-point tracking, or perspective tracking because the corners of the screen are tracked so the replacement can match the screen’s angle and motion.

Choosing Footage for Practice

A typical screen replacement workflow may involve several different shots, including laptop screens, phones being scrolled, and cameras moving around a device. Some examples are easier than others. When nothing passes in front of the screen, screen replacement can be done without a green screen because the tracking can be based on visible detail in the display area.

Green screen becomes much more useful when there is occlusion, meaning something moves in front of the screen. A hand scrolling on a phone is a common example. A green screen is also helpful when the screen shape is not a perfect rectangle. For instance, a modern phone display may have rounded corners and a notch, which makes masking and edge cleanup more demanding.

For a first pass, a shot with no occlusion, such as a camera orbiting around a phone, is a practical choice because it focuses attention on the tracking and corner alignment process.

Trimming the Work Area to Speed Up Tracking

Longer clips take longer to track. One way to move faster is to shorten the composition duration so the tracker has fewer frames to analyze. Reducing a clip from roughly thirteen seconds to ten seconds can save time, especially during practice or testing.

It is worth noting that adjusting the composition duration does not necessarily change how much source footage is sent into the tracking tool. Even if the comp is shortened, the tracker may still display the full source clip. In that situation, the important detail is simply to ignore frames beyond the portion you intend to use.

Tracking the Device First, Then Defining the Screen

A common point of confusion is what you are actually tracking. In practice, it often works best to track the device body rather than only the screen area. The device provides more stable, detailed features for tracking, especially when the screen is a flat green color with minimal texture.

The tracking process begins by applying the Mocha AE effect to the video layer and launching the Mocha interface. Inside Mocha, the first step is creating a tracking layer by drawing a spline. A rectangular X-Spline is a convenient starting shape, and it can be adjusted point by point to fit the device at an angle.

The tracking layer does not have to match the screen precisely. In fact, making it slightly larger than the screen can help because it includes more pixels and more detail for the tracker to analyze. Including a bit of the surrounding background can also help if those pixels remain consistent relative to the device during the shot.

To keep the process simpler, it helps to create the tracking shape on the first frame, assuming the device remains visible throughout the clip. This avoids the need to track both forward and backward from a middle frame. Once the shape is set, it is best not to adjust it later. Changing the shape mid-track creates additional keyframes and alters which pixels are being tracked, which can lead to unstable results unless you have a specific reason to animate the shape.

Setting the Planar Surface for Corner Pin Data

The tracking layer defines what Mocha analyzes, but the screen replacement requires something more specific. That is where the planar surface comes in. The planar surface is a visual overlay that defines the four corners and center point of the surface you want to replace. This is the data that will drive the screen replacement back in After Effects.

The key distinction is that the outer spline can be larger than the screen, but the planar surface must match the screen’s dimensions. Those corner points are what will ultimately be applied to the replacement graphic. If the planar surface is misaligned, the replacement will not sit correctly and may show edges that do not match the real screen.

A phone screen introduces an extra challenge because its corners are rounded. There are no true square corners to snap to, so the alignment becomes an approximation. The goal is to align the planar surface edges with the sides of the screen area rather than chasing a literal corner point. Zooming in closely helps, especially when you are trying to avoid leaving a visible green fringe.

That green fringe often appears along curved or diagonal edges due to anti-aliasing, which is how pixel-based footage smooths curves and angled lines. When the replacement goes in, any leftover fringe can read as a thin green outline around the new screen. Positioning the planar surface just outside that fringe helps reduce the risk.

This stage often involves trial and error, including careful panning and zooming while moving individual corner points. It is easy to accidentally move the entire planar surface instead of one corner, so patience and precision matter. The idea is simple but the execution benefits from taking extra time. The closer this match is, the less rework you will need later.

Using a Grid Overlay to Check Perspective

Mocha provides a preview option that can make the planar surface easier to evaluate. By inserting a simple grid into the layer options, you can see the planar surface filled with a patterned guide rather than an empty outline. This does not affect the final output. It simply makes it easier to see whether the surface is fitting the screen correctly and whether the perspective feels right as the device moves.

A grid is particularly helpful when the camera angle changes because it makes skew and perspective shifts more obvious.

Adjusting Tracking Parameters for Accuracy

Before tracking, review the tracking settings. For screen replacement, it is common to track translation, scale, rotation, shear, and perspective. Translation represents position changes. Scale and rotation are self-explanatory. Shear and perspective help the planar surface adapt as the screen angle shifts relative to the camera.

Another important setting is the minimum percentage of pixels used. Mocha sets a value automatically, such as 50 percent, but it may be worth increasing when you want a more precise track. Raising the value to something like 90 percent tells Mocha to use more of the pixels inside the tracking area. This can improve accuracy, but it also increases tracking time. The tradeoff depends on your computer performance, the complexity of the shot, and how reliable the initial track looks.

Once tracking begins, the most important thing is to watch for drift. The tracking region should remain stable on the device, and the planar surface should continue to align with the screen area. If the track slides away or gradually drifts, it usually means the tracked region needs refinement or the pixel usage settings need adjustment.

Saving the Track and Preserving the Data

After tracking completes, save the Mocha project from within Mocha. Then close the Mocha window to return to After Effects and save the After Effects project as well. Both saves matter.

Mocha tracking data is stored inside the Mocha AE effect applied to the layer in After Effects. Saving inside Mocha preserves the tracking work you just created, and saving the After Effects project ensures the effect and its embedded data are preserved in your project file. When both are saved, the tracking data is protected even if After Effects crashes afterward.

A Strong Track Starts With Careful Setup

Screen replacement is a multi-step workflow, but each step has a clear purpose. The tracking layer gives Mocha enough detail to follow the device motion reliably. The planar surface defines the exact screen area that will receive the new content. Zooming in to align edges carefully helps prevent visible fringes and mismatched borders, especially on rounded phone screens. With solid tracking data saved correctly, the project is ready for the next stage, where the tracked corner information can drive a replacement graphic that moves naturally with the device.

This article is part of a continuing series on motion tracking and compositing techniques in After Effects.