I’ve been using XenForo forum software for a few months now and I’ve come to learn a few things. I’ve learned that it’s some of the best software I’ve ever taken advantage of and that it sure does create a lot of pages. If one member registers for an account and makes one post, lots of pages are spun out from that set of circumstances. Living in the world of Google Panda, this isn’t a good thing. I’ve actually seen mixed effects though – some sites that allow all these extra pages to flood into Google’s index are doing quite well, but from that I’ve unearthed, there’s a definite theme to the most successful sites that use XenForo. The ones that do well can get away with their bad behavior because they’ve got tons of links pointing towards them. For the rest of us, we need to be careful.
There are two huge issues with the forum software, which surroung member profile pages as well as image pages. I’ll talk about the profile pages first.
As I mentioned above, every member gets their own profile page when they register for an account. From what I’ve discovered, spammers like to register at forums, drop a link back to their spammy site in their profile and then never return to the site. It’s important that the search engine crawlers never see that link, even if it does have the rel=nofollow attribute attached to it.
Each member profile on these websites offers a few additional pages attached to it. Not only is there the profile page itself, but there’s also a Profile Posts page, Latest Activity, Postings and About pages. I’m sure there’s more, but you get the idea. Some of these pages use the rel=canonical attribute and some use noindex, but some use nothing. Considering that this area is so confusing and considering that 99% of forum profiles are empty, it’s better to block these pages entirely. In XenForo, there’s an option in the user group permissions area to make the Unregistered/Unconfirmed visitors of the website unable to see any member page. If they click on a profile username, they’ll be directed to a login page and they’ll receive a 403 Forbidden error header code. That’s exactly what you want.
Tip #1: Don’t allow crawlers to see any member pages.
The way attachments work in this software is a bit strange. In some circumstances, uploaded images aren’t simply linked to as their native filenames. They’re linked to as odd URLs, like:
Do you see that “.7629/” after the filename? Do you know what that does? That makes the uploaded image its own page. If someone uploads ten photos to their post, the result is that the post is indexed as one page and the ten images are indexed as ten more pages. Those ten pages are all empty, except for the image. You definitely don’t want that.
The good news is, there’s also a setting in the user group permissions area that forces unregistered guests to register and login before viewing attachments, so that’s what you’ll want to set your software to. If someone or something, such as a crawler, tries to view an uploaded attachment, they’ll be presented with a login page again and that login page will use that same 403 error header code.
Tip #2: Don’t allow crawlers to see any attachment pages.
There are a bunch of other things that I may add to this post, but I think I’ll leave things here for now. I will say this though, you definitely don’t want to block /members and /attachments directories in the robots.txt file. Doing this will indicate that these pages do exist and that the search engines should index them, but not show any results for them. You don’t want them in the index at all. That’s the whole point. There are thousands of them and they’ll do your website no good at all. As of right now, this is all I have in my robots.txt file for my forum websites;
Do you have any thoughts on this? If you do, please share below. Thanks!
UPDATE: By all means, block the /whats-new/ directory in robots.txt as well.