Using the Pen Tool to Create a Vector Mask in Adobe Photoshop

Masking is an essential part of working in Adobe Photoshop. It allows us to non-destructively edit something out or into a photograph or a number of other things. While masking can be a tricky concept to pick up during the first few minutes of learning, things begin to flow rapidly after a few moments of that learning.

I’m sure you’ve heard of layer masks and layer masking. I’ve talked about these things a bunch on this website. The thing is, what I’ve primarily be discussing is the pixel based form of masking, which is great, but has its pitfalls. The greatest pitfall is that these types of masks rely on the pixels of an image, which, as you know, can look distorted if an image is resized.

An alternative to pixel based masking is something called vector based masking, which doesn’t rely on pixels. Like vector images, these masks rely on mathematical algorithms to produce crisp, sharp lines no matter the size. Here’s what Adobe has to say about vector masks:

A vector mask is a resolution independent path that clips out the contents of the layer. Vector masks are usually more accurate than those created with pixel-based tools. You create vector masks with the pen or shapes tools.

While I won’t be getting into how to use the Pen Tool per se in today’s post, I will be discussing how to use that tool to mask out the background of an image. In all honesty, the Pen Tool is a beast unto itself and this post wouldn’t be big enough to handle all I have to say about it. As a compromise, I’ve decided that I’d use today to focus primarily on the topic of masking something out. I’d like to keep it simple.

In today’s post, I’m going to open a photo inside of Adobe Photoshop. Then, I’ll use the Pen Tool to go through the process of masking the background out of a photograph that contains a camera. I purposefully chose the camera picture because it’s got more straight edges than curved. The reason for this is because the topic of creating curves with the Pen Tool, while fairly simple and straightforward, needs space to breath. That will have to wait for another time. I may touch upon it below though. What I really want to impress upon you is the fact that vector masks exist and that they are extremely simply to take advantage of for all types of purposes. One such purpose might include cutting an object out of one photo to include in another. You’ll see what I’m referring to below.

Today’s Demo Photo

Again, for this post, I chose a photo of a camera. Notice the straight lines.

This image shouldn’t take too long to work with.

The Easiest Vector Mask Ever

Okay, so we already know that vector masks can be created with the Pen Tool. The issue is, there are many methods for using this tool in conjunction with the mask itself, so which one is best? In this section, I’m going to give you a quick example of how this camera photo can have its background “removed” from it.

The very first thing I’ll do (after opening the photo in Photoshop) is to head up to the Layer > Vector Mask > Reveal All menu item and click.

Once I click that menu item, I’ll notice a new vector mask thumbnail appear in the Layers panel, to the right of the layer thumbnail.

Now, both the thumbnail and the vector mask thumbnail should already be selected. If I move over to activate the Pen Tool in the left toolbar, I can begin working.

Here’s how it’s going to work. I’m going to click on the canvas with the Pen Tool a few times in an area that surrounds the camera. This will create what’s called a “pen path.” The first click won’t make a change. The second click will make everything disappear. The third click will bring back a portion of the photo and every successive click will shape the mask area. This is the final product.

While this method did the job, it didn’t do it well. First of all, who in the world clicks around a photo they can’t see? As I mentioned above, the image disappears after the second click. Also, because of the random way I masked this photo, I wasn’t near the target at all. I can see see a lot of the background and some of the camera has been removed. I’d have to spend way too much time editing the path if I did things this way. I do want to show you a few things before I go any further though.

First, take a look at the Layers panel. You can see the path I just made in the vector mask thumbnail.

Next, I’d like you to see the path in the Paths panel.

And finally, I want you to see how each handle along the path can be manipulated. If I choose the Path Selection Tool (the black arrow) from the left toolbar, I can click on any handle and move the entire path area.

And if I click on the Direct Selection Tool (the white arrow) in the left toolbar, I can select just one (or more) handle(s) along the path and move only it or them.

So, as you can imagine, I do have the ability to modify the path after the fact. The problem is, that’s a terrible job to have. I’d have to add all sorts of anchor points and then try to fit them into small spaces. Trust me when I say this – there’s a better way.

Creating a Vector Mask, the Better Way

In this section, I’m going to create the path before applying the vector mask. Things will be much better this way. I’ll need to remove the path I just made though, so I thought I’d show you the process for doing that.

Basically, there are two really easy ways to delete vector masks. First, you can simply click and drag the mask thumbnail in the Layers panel down to the little trash can at the bottom of the panel or you can click and drag the vector mask path layer in the Paths panel I showed you earlier down to the trash can at the bottom of that panel. It’s up to you. They both do the same thing.

What I’m going to do now is reverse things. I’m going to first start off tracing the camera with the Pen Tool and then I’ll apply the vector mask when I’m finished. And after that, I’ll show you a few tricks that can really help when it comes to dealing with curves and the edge between the object and its background. This last part is going to slightly delve into how to use the Pen Tool, but I’ll try not to go too far.

To create the outline I want around the camera, I’ll use the Pen Tool. I’ll click at every corner I come across as I work around the circumference. Actually, the more I click right now and the more anchor points I include, the more accurate the outline will be. Here’s what I quickly came up with. I didn’t go nuts with this because I’m trying to save time.

Now that I have the outline done, I can create the vector mask. To do so, I’ll head up to the Layer > Vector Mask > Current Path menu item and click.

Now look at the result. Isn’t this much better? With the menu item I just used, I told Photoshop that I’m working on a layer and that I’d like to create a vector mask within that layer. Then, I said that I’d like to use the current path I just created with the Pen Tool for that vector mask.

At this point, I can do whatever I want with this camera. I can drag it over into a new file or do something with it in the current file. It’s up to me.

Editing the Mask

When using paths with vector masks, I think you’ll find things quite flexible when it comes to editing. The first thing I’d like to discuss is what to do when you encounter a curve in whatever it is you’re attempting to mask. Here’s the rule; when you get to a curve, click and drag. That’s it. Take a look below.

If you look closely, you’ll notice that I’m beginning to create a path around a curve. I clicked to set the first anchor point and then when I clicked a second time, I didn’t let go of my mouse button. I continued holding it down and then I dragged away from the curve. When the handles appeared, I knew I was on the right track. This technique will take some practice, but you’ll get it soon enough. You’ll also likely need to click to adjust other handles as you work. To do this while you’re in Pen Tool mode, simply hold down the Ctrl key on your keyboard to switch from Pen Tool to the Direct Selection Tool. That’s the one you want to use when moving handles. Again, this line of thought is edging towards how to use these types of tools, which I don’t want to get into today. I just thought this one tip would help.

The next tip I have for you really will make your life easier when creating path based vector masks in Photoshop. As you can see from the camera above, the background is transparent. When it comes to refining masks (editing them later on), it helps to make the background somewhat opaque. To change how visible the masked background is, I’ll need to go to the Window > Properties menu item and click. I’ll need to make sure the mask thumbnail in the Layers panel is selected first though because Photoshop needs to know which Properties panel to open up.

Inside of this Properties panel is a Density slider. If I push the Density slider to the left, the background that’s been hidden will become more visible.

Now, if I wanted to use some other tools to edit the path so it more accurately follows the edge of the object I’m trying to outline, I can do that with no issue at all.

I think I’m going to stop here. Rest assured though, I’ll be covering the Pen Tool and the other vector based tools a lot more on this website, so get ready!

——

I hope I clearly explained how to create a vector mask in Adobe Photoshop with the Pen Tool. If you have any questions regarding this post, please leave them for me in the comment section below. Thanks for reading!