Advanced Masking

Layer Masks | Smart Objects | Advanced Masking | Image Sharpening | Burning and Dodging
Now that you’re comfortable working with Layer Masks and Smart Objects, it’s time to dive a little deeper into the advanced technique of layer masking.
In the Intro To Layer Masks article, you created layer masks by using Photoshop’s brush tool. This works well when brushing in a subtle correction, but can be time consuming if your correction is more pronounced or if you need to remove your subject from its original background and composite it with a new background.
Most advanced masking techniques require a combination of existing masking/selection techniques. Rarely does one technique work in all situations. For that reason, it is beneficial to be able to draw on a large bag of tricks. In this article, I’ll introduce you to some of the advanced techniques used by photographers and professional retouchers to take full control of the presentation and style of their images.

If you’d like to follow along, click on the Harborboats.jpg image link and right click on the image to save to your desktop. Do the same for GrassyMeadow.jpg. Open the images in Photoshop.

Creating Complex Masks Using Color Range

Harbor Boats

Regardless of the correction you wish to make, or the effect you’re trying to achieve, your goal is to isolate the portion of the photo you’d like to correct from the rest of the image. For example, in the Harbor Boats image (right) the sky is too purple.
This is a problem often arises when converting photos to CMYK for reproduction on a printing press. We know skies should be blue, not slightly purple. To fix this, we must first isolate the blue sky from the rest of the image. Attempting this correction with the lasso tool would be impossible due to the complicated tangle of lines from the boat masts and rigging lines. The Magic Wand tool may work and would be worth a try, but would be time consuming to select each portion of the sky between the boats. So, for this tutorial, I’m going to show you how to use the Magic Wand tool’s more efficient big brother, the Color Range command (Select>Color Range).

Color Range 1

The Color Range tool is significantly improved for Photoshop CS4 and can now be considered a valuable tool in your selecting and masking arsenal. Prior to CS4, the Color Range lacked a couple of key options that make selecting complex objects much easier. I’ll address these new features now and next offer advice for readers working in earlier versions of Photoshop.
Step 1. To begin, let’s take a look at the Color Range dialog box. Remember, in earlier versions of Photoshop, it looks a little different than my screen shots.

Color Range 2

Step 2. At the top of the dialog is the Select pull down menu. This allows you to choose from several predefined colors (red, green, yellow, etc.), tonal ranges (highlights, shadows, midtones) or create your own selection using the Sampled Colors option. This is the best way to get a clean selection in Color Range, so select Sampled Colors if it isn’t already selected.

Color Range 3

Step 3. For all versions of Photoshop, you can select colors in Color Range by creating a selection with the eyedroppers to the right of the preview and refining the selection using the Fuzziness slider. The eyedroppers allow you to click inside of the image or on the preview within the dialog to target the color you wish to select. Add to the selected color range with the Add to Sample eyedropper (the one with the + next to it) or subtract from the color range with the Subtract from Sample eyedropper (the one with the – sign). You can also add or subtract from the color range selection using the selection modifier keys. Holding the Shift key while clicking on the image or preview adds to the selection, using Alt/Option (Win/Mac) subtracts from the selection. I recommend clicking once to set your target color range, then Shift-clicking several times to add to the selection.

Color Range 4a

Color Range 4b

The image on the left shows the Color Range preview after a single click with the Eyedropper tool. The image on the right shows the same preview after Shift-clicking several times in the sky to select the entire sky. The whiter the area in the preview, the more the area is selected.
Step 4. After creating the selection, it is time to refine it using the Fuzziness slider, which helps you select the range of colors you want to affect with your mask. The Fuzziness slider determines how similar two colors need to be in order to be selected. For example, when using a low Fuzziness setting, the two colors need to be almost exact matches to be selected. A high Fuzziness setting will select a broader range of colors, all blues, for example. The trick is to use the lowest Fuzziness setting that still achieves the goal of your selection.
In the Harborboats.jpg image (download link at the beginning of the article), the goal is to select the sky. After Shift-clicking several times in the blue sky to select the sky, I’ll begin adjusting the Fuzziness slider up and down to refine the selection. Too high of a Fuzziness setting selects the sky and the water, which is too wide a color range for this correction. Too low of a setting leaves the sky along the horizon unselected. I found a Fuzziness setting of 65 to select all of the sky without selecting the water.

Color Range 5

If you’re using Photoshop CS3 or earlier, press OK to accept the selection. If you’re using CS4, stick around, we have a couple extra features to play with.
As I mentioned earlier, Color Range is significantly improved for CS4, both in the math Color Range uses to create the selection and the addition of the Localized Color Clusters. Given the name, the feature sounds complicated, but it isn’t. When you check the Localized Color Clusters option and click to set a selection point, Photoshop activates the Range slider as well. Localized Color Clusters allow you to select blues in one portion of the image without selecting all the blues in the image. For example, if you’ve taken a landscape photo of a meadow on the edge of a forest, you can select the green grasses in the foreground, without selecting the green trees in the background. The Range slider gives you this control—control over how close a color needs to be to your original selection point in order to be selected. A low Range setting will only select similar colors in close proximity to your selection point. A high Range setting will select colors throughout the image matching the color in your selection point.

Color Range 6a

Color Range 6b

In these two examples above, I placed a single sample point in the sky. The image on left the uses a Range setting of 26 to select only the blues within a short distance from the original sample point. The image on the right, shows the effect of a Range setting of 80. Notice how much larger the selection is.
The Fuzziness and Range commands are powerful tools for isolating specific colors within your photo. Fuzziness refines your selection based on color characteristics (Hue, Saturation and Luminance) while Range allows you to refine your selection by proximity to the original sample point.
CS4 users, use the Fuzziness and Range controls to see if you can improve upon the selection you made with Fuzziness alone. When you’re satisfied, press OK to create your selection.
Step 5. Now that you’ve created your selection, create a new Hue-Saturation adjustment layer from the Create New Fill and Adjustment Layer at the base of the Layers panel, or using the Hue-Saturation button in the Adjustments panel.

Color Range 7

Step 6. Set the Hue to -12 and the Saturation to +33. The sky is now a much cleaner, more vibrant blue than before. Toggle the layer preview button on and off to check your work. You shouldn’t be able to see any obvious lines that may give away your work. If so, you’ll want to feather the mask using the Masks panel (CS4 only) demonstrated at the end of the article.

Harbor Boats (corr)

Creating Complex Masks Using Color Channels

Grassy Meadow

Color Range can help make short work of difficult selections, provided there is enough color difference between the object you’re aiming to select and the rest of the image. In the GrassyMeadow.jpg image, the yellow in the flowers and the green of the grasses can be difficult to separate using traditional selection methods. The yellow flowers are far too numerous to select using the Lasso or Magic Wand and using Color Range can be tricky because the colors are so similar. For this image, we’re going to dig deep into the bag of selection tricks to create our selection from the red, green and blue color Channels to change the color of the yellow flowers.

What Are Channels?

The digital photos created by your scanner or digital camera stores tone and color information as three separate components, called channels; one each for red, green and blue. When merged together, these three color channels give you the full range of color and tone in your digital photos. When needed, we can examine these color channels separately to perform advanced corrections, like creating a complex mask. Consider the three color channels of the GrassyMeadow.jpg image. Notice in the red channel, the flowers appear distinctly separate from the grasses, while they are virtually the same tone in the green channel. In the blue channel, the grasses are a dramatically different tone than the sky. These differences can form the basis for complex masks.

Grassy Meadow (red)

Grassy Meadow (green)

Grassy Meadow (blue)

To view your photo’s color channels, open the Channels panel (Windows>Channel). Single-click on the composite RGB channel, then click on an individual color channel to view it. Look for any channels containing high contrast between the portion of the image you wish to adjust and the rest of the photo. For example, in the red channel of the GrassyMeadow.jpg image there is a high degree of contrast between the flowers and the grasses. The flowers are nearly white, while the grasses are middle gray.
If you remember from the Intro to Layer Mask article, a mask is white in the area affected by the correction and black in the unaffected areas. If you wish to adjust the color of the flowers, you simply need to force the flowers to white and the rest of the image to black. The red channel gives us a starting point for creating this complex mask.

Channels 1

Step 1. Drag the Red channel on the Create New Channel Icon at the bottom of the Channels panel. This creates a duplicate channel titled Red Copy.
In order for the Channel to be effective as a mask, we need to force the pixels in the channel to pure white and pure black, otherwise, the correction to change the yellow flowers will be applied to the rest of the image as well.
You can use Levels, Curves or the Threshold command to perform this correction. For this example, I’ll use Levels.

Channels 2

Step 2. Open the Levels dialog (Image>Adjustments>Levels) and begin moving the Highlight and Shadows sliders toward the center to force pixels to white and black. Adjust the midtone slider to push the midtones toward either the highlights or shadows.

Channels 3

Step 3. For this image, setting the Shadow slider to 151, the Midtone slider to 0.12 and the highlight slider to 179. This forces the grasses to black while keeping the flowers white. The sky and rainbow are still white, but you’ll take care of that in the next step.

Channels 4

Step 4. Select the Lasso tool (L) and drag a lasso around the white and gray portions of the sky. Be sure your lasso selection doesn’t extend into the flowers.

Channels 5

Step 5. Fill the selection with black (Edit>Fill>Black). This removes the white areas from the sky.

Channels 6

Step 6. The mask is now complete and ready to use. Click on the Load Channels as Selection icon at the bottom of the Channels panel. This creates a selection from the mask.
Click once on the composite RGB channel to return to the full-color image, then return to the Layers panel. Create a new Hue-Saturation adjustment layer using the Create New Fill or Adjustment Layer icon. Using the Hue, Saturation and Lightness controls, change the color of the flowers to your liking. I changed the flowers from yellow to pink by setting the Hue at -111, the Saturation at -49 and the Lightness at +33. To make the pink color blend with the detail in the flowers, change your layer blending mode from Normal to Color.

Channels 7

Channels 8

Grassy Meadow (pink)

This is an effective way to quickly change flower color, but the edges of the mask look fake and show a clear transition between the masked and unmasked areas. In order to convincingly change the color of the flowers, we need to refine the mask. Fortunately, the Refine Mask command in Photoshop CS4’s Masks Panel can help us do just that.

The Masks Panel

Masks Panel 1

The Masks Panel, introduced in Photoshop CS4, simplifies and consolidates many of the commands for refining and improving layer mask quality. You can replicate most of the commands found in the Masks panel through other methods, but the simplicity and convenience of the Masks panel is hard to beat.
With your Hue-Saturation adjustment layer still targeted in the Layer panel, open the Masks panel (Window>Masks). For many subtle adjustments, you can use the Density and Feather sliders to quickly refine the mask to suit your needs. The Density slider makes the mask more transparent?, allowing some of the correction to show through. This is helpful if you add a local correction, like a saturation adjustment, and decide a subtle saturation boost would be helpful throughout the photo. Increasing the Density slider allows a percentage of the saturation adjustment to apply to the masked pixels.
The Feather slider feathers the edges of the mask, blurring the transition between the white and black areas. This smoothes the edges of the mask, making it appear more natural. For this image, select the Mask Edge button in the Masks panel to open the Mask Edge dialog. The commands in this section are identical to those found in the Refine Edge command in Photoshop CS3 and CS4. The Mask Edge command is used for adjusting masks, the Refine Edge command is used for selections.
In the Refine Mask dialog, begin by setting all five sliders at zero. This displays the current state of your mask. Near the bottom of the Refine Mask dialog are five different preview modes. Select the one that best matches your current need. For example, if you are creating a cutout, previewing the masked pixels against either black or white allows you to see any color contamination from the background. For our sample image, the first option is the most effective because we can see the edges of the selection and the effect it has on the image.
Your controls in the Mask Edge dialog are as follows:

• Radius: The Radius command smoothes out the edges of the mask and uses built-in intelligence to attempt to automatically find the edges of a selected, or masked, area.
• Contrast: The counterpart to the Radius slider, the Contrast slider is most often used to improve the integrity of the edge when using a high Radius setting.
• Smooth: Eliminates any rough edges and rounds sharp corners created when using the lasso tool or hand-drawing masks.
• Feather: Blurs the edge of the mask slightly to make the mask edge blend more naturally with the surrounding area.
• Contract/Expand: Increases or decreases the size of the masked areas to reduce color contamination. The Contract option is particularly useful when performing cutouts as shrinking the selection slightly removes any remaining background pixels which may not have been masked properly.

Because you’ve already gone through and generated a highly detailed mask, you can safely ignore the Radius and Contrast sliders, focusing instead on the Smooth, Feather and Contract/Expand sliders.
When using the Smooth slider, begin increasing the value until problems, like rounded edges or mismatched mask edges begin to appear. Since this is a low-resolution image, and the mask already has a lot of variance along the edges, a Smooth setting around 4 appears to be ideal.

Masks Panel 2

The Feather command blurs the edges of the mask slightly to create a natural-looking mask. Because this is a low-resolution image containing lots of fine detail in the mask, a low Feather setting of 0.9 pixels blends the edges of the mask without allowing the original yellow of the flowers to show through.

Masks Panel 3

Finally, the Contract/Expand slider is used to expand the mask 17 percent to cover any of the yellow tips of the flowers still peeking through the mask.

Masks Panel 4

Pressing OK applies these changes to the mask. The new color of the flowers in the foreground appears quite natural.
Note: Due to the small image size of this image, a few yellow flowers were ignored by the mask. On a full-resolution image, the mask will be more complete due to the larger pixel sizes of each flower. Generally speaking, you’ll have better results building your mask on a full-resolution file, then resizing a final version for the Web.

Grassy Meadow (final)

Conclusion

This article introduces you to two of the many masking techniques professionals use to isolate specific elements within their photo, either for corrections, compositing or for cutting the subject out from the background. No single technique works in all situations, therefore, it is essential to have a variety of different masking techniques to use. When assessing a masking problem, you’ll evaluate a variety of masking methods, from simplest to most complex, to derive a solution. Often, more than one technique can be used to perform your correction. Select the one requiring the least amount of effort and providing the best result. The fewer manual brush strokes necessary to complete a complex mask, the better.
Your experiments with advanced masking techniques will pay dividends in other areas of your image corrections as well. For example, learning to look at your color channels for detail and contrast is useful when you have to recover highlight detail in saturated colors, or remove noise from an image taken at a high ISO setting.
These techniques, and others presented in this series, help pave the way for deeper mastery of Photoshop for correcting and enhancing your digital photos.