The use of mattes dates back to long before electronic media. When I think of mattes I often picture the early 1900 black and white photo with the circular inset highlighting and highlighting the subject. Now we have track mattes, mattes that move and reveal the subject, mattes that spell part of the message.
How do you apply a matte? It's just another layer, of course! Since simplicity is our guide, create an oval matte much like one you would see framing a photograph. You can actually create a matte inside After Effects but for our friend simplicity , let's create one in Photoshop and import it.
In Photoshop create an image the same size as your After Effects composition. You really do not need to draw anything. Using the elliptical marquee tool draw the ellipse that will become your matte when you apply it in After Effects. Click on the 'Channels' pane, typically next to the 'Layers', then create a new channel using the 'create new' icon to the left of the delete icon. When you create your alpha channel the background will become solid black. Be sure white is the chosen color, choose the paint bucket tool, and click inside your selection. It will turn white; This is the 'alpha channel' area that will let your movie show through in After Effects.
I would save this file as a '.tif' type file with the default settings. That was a little bit of a side trip but After Effects and Photoshop definitely like to play together and creating alpha and getting cozy with transparency is one of your most important tools.
Now that you have your matte, import the file into After Effects. Create a composition by simply dragging the movie of your choice to the new composition icon. This will create a composition based on your movie type and length, an instant composition.
Drag the matte we created into your timeline area, making sure it is above the movie. The track matte setting and display is abbreviated 'TrkMat'; if you do not see it, the shortcut key 'F4' toggles the modes display. There is also a switch icon in lower left of your timeline panel 'Expand or Collapse the Transfer Control pane'. Transfer control is a good way of picturing this. Your matte overlay will 'control' the viewing area of your movie.
Making sure the matte file is above your movie layer, under the 'TrkMat' pane click the pull down option box and choose 'Luma'. You will immediately see your movie masked by your matte! Notice three other indicators. The 'eye' icon on the matte layer has 'shut off' and next to the name under 'source name' you see a little icon with a white ball in the center indicating this is an overlay. Similarly, your movie name now has a little icon showing the blank composition grid white gray pattern with a black ball in the center, indicating that it has a matte applied.
These are the basic steps for creating a simple matte and applying it in After Effects. What more can you do with mattes? Let's explore some more.
By the way, here's some interesting 'matte' history from Wikepedia:
http://en.wikipedia.org/wiki/Matte_ (filmmaking)