Use reference frames in After Effects to guide Content-Aware Fill and prevent unwanted sampling from complex backgrounds. Learn how Photoshop integration enables precise cleanup by creating a tailored reference image before generating the final fill.
Key Insights
- Content-Aware Fill may pull unwanted details from nearby textures, and creating a reference frame allows users to define the desired result for more accurate fills.
- The reference frame workflow requires Photoshop, where users can edit a single extracted frame with tools such as the Remove tool, healing tools, or manual painting before saving and returning to After Effects.
- After Effects uses the cleaned reference frame to guide the fill process, and users can refine results by adjusting mask tracking, alpha expansion settings, fill methods like Surface, and the defined work area.
This lesson is a preview from our After Effects Certification Course Online (includes software & exam). Enroll in a course for detailed lessons, live instructor support, and project-based training.
This is a lesson preview only. For the full lesson, purchase the course here.
Content-aware fill is a powerful tool for removing unwanted elements from video, but it works by sampling whatever happens to be nearby in the frame. On a clean, simple background, that's usually fine. On a complex one, it can become a problem. If you're trying to remove graffiti from a wall that also has text, door seams, and other visual detail, the tool may sample from those adjacent areas and produce a result that looks nothing like the original surface.
This is where the reference frame feature becomes invaluable. Instead of letting the algorithm guess, you supply a single still image that shows exactly what the finished result should look like. After Effects uses that image as a guide during the fill process, dramatically improving accuracy on complicated backgrounds.
What You Need Before You Start
This workflow requires both After Effects and Photoshop. The "Create Reference Frame" command works by extracting a single frame from your composition and opening it directly in Photoshop, where you clean up the unwanted element. After Effects then imports that corrected image as the reference. If you do not have Photoshop installed, this particular workflow is not available.
Any recent version of Photoshop will work, including the public beta, which you can download from the Creative Cloud app under the Beta section. The beta often includes newer AI-powered tools that can make the cleanup step faster and more effective.
Choosing and Creating Your Reference Frame
Start by identifying a frame that represents what you want the final image to look like. This is typically the first or last frame of the clip, where the camera may be at a clean angle or the subject is positioned away from the area you want to fix. Open the Content-Aware Fill panel (found under the Window menu) and click "Create Reference Frame."
After Effects will extract that frame and open it in Photoshop automatically. From here, your goal is simply to remove the unwanted element and leave the background looking as natural as possible. You can use any of Photoshop's tools to do this: the healing brush, clone stamp, or the AI-powered Remove tool, which lets you paint over an area and have the software intelligently replace it.
The Remove tool is particularly effective for this kind of work. Increase the brush size with the bracket keys, paint over the area you want gone, and release the mouse. Photoshop analyzes the region and fills it in. If the result misses a spot, simply paint over it again.
Once the frame looks the way you want it, save the file with Command S (Mac) or Control S (Windows), then close Photoshop and return to After Effects. The reference frame will appear in your timeline as a single still image layer.
Masking and Tracking the Problem Area
With your reference frame in place, the next step is to create a mask over the element you want to remove on the video layer itself. Select your video layer, choose a mask tool appropriate to the shape (the Pen tool works well for irregular shapes), and draw a path that closely follows the contour of the object. Once closed, set the mask mode to Subtract so that the masked region becomes transparent, creating a hole in the footage.
If the object moves during the clip, you will need to track the mask. Right-click on the mask in the timeline and choose "Track Mask." Use position, rotation, and scale tracking, then run the track forward from the first frame. After Effects will follow the masked region through the clip and update the mask path accordingly. Keep an eye on the result to make sure the mask does not drift from the content you want to cover.
Running Content-Aware Fill with the Reference Frame
With your mask tracked and your reference frame layer visible in the composition, navigate one frame past the reference frame and open the Content-Aware Fill panel. Set your Alpha Expansion to around 10 pixels to give the fill a small cushion beyond the mask edge, which helps blend the result into the surrounding image.
Next, choose a fill method. "Surface" works well for elements placed against textured backgrounds, such as text or graphics on a wall, because it preserves the texture of the underlying surface. "Edge Blend" is faster but may soften textures, while "Object" is best suited for removing subjects in motion. When in doubt, Surface is a strong default for the kind of scenario described here.
Before running the full fill, it is worth testing on a shorter section first. Drag the work area bar in the timeline to cover just a few seconds of footage, then click "Generate Fill Layer." This lets you evaluate the quality of the result quickly without committing to a full render. If the output looks good, extend the work area to the full clip and run the fill again.
Understanding Render Time and Output Quality
The time content-aware fill takes to render depends on several factors: the length of the clip, the complexity of the region being filled, the resolution of the footage, and the speed of your computer. Surface and Object fill methods generally take longer than Edge Blend. On a modern machine, most clips should process in a reasonable amount of time, but longer or higher-resolution clips may require more patience.
When the fill layer is complete, toggle off your mask to see the final result in context. A well-constructed reference frame should produce a fill that accurately matches the surface texture, handles shadows cast across the area, and avoids sampling from unrelated parts of the background. The reference frame does the heavy lifting by telling the algorithm precisely what the finished frame should look like rather than leaving it to guess.
A Few Final Tips
If your shot requires removing multiple elements, you can create multiple reference frames and work through them in stages, though managing multiple fill layers does add some organizational complexity. When possible, handling everything in a single reference frame pass keeps the project simpler and easier to revise.
Also keep in mind that you do not necessarily need to remove every piece of text or detail from the reference frame, only what you want gone from the final output. Clean up what matters to you and leave the rest, since the fill will be guided by whatever the reference image shows.
Used thoughtfully, the reference frame workflow gives you precise control over object removal in complex environments, making content-aware fill far more reliable than it would be working from context alone.
This article is part of a continuing series on motion tracking and compositing techniques in After Effects.