Website SEO – They Why & How of the Robots.txt File
Setting up a robots.txt file for your WordPress website or blog is an SEO trick that sounds full of mystique but is really straight forward.
There’s a number of reasons they impact WordPress SEO but the largest one is that Google HATES (reads penalizes) duplicate content.
You may say “but I only post stuff once, ever”… however, there are certain components within websites that can give the opposite impression.
The biggest culprits are categories and tags which are important to use, but pose a risk because they get a readable page associated with them.
Let’s say I wasn’t paying real close attention and I have two tags: “social media resources” and “social media tools”. Now you can see how, because these are similar, there is a chance that if you opened them, they could show that the exact same posts had been tagged with those two tags. Unfortunately, when the page visually (rather than logically) looks similar…. Google starts slapping hands!
So, how is a Robots.txt File Going To Help WordPress SEO?
A robots.txt file is a simple little file that goes in the root directory of your blog/website and tells Google (and other “bots” or “spiders”) where they may and may not store information from. This essentially hangs a “do not disturb” sign on dangerous areas (although it does not stop them from reading necessary things, only from storing the data).
Think of a robots.txt file for your site as a “duplicate page filter” for the Search Engine Bots.
You can find a link to my robots.txt file here and then save the page if you do not want to fuss with opening notepad to create your own. This is a very generic robots.txt file and you can use it on just about any self-hosted wordpress blog.
What’s Inside the Robots.txt File?
The inside of the file looks simply like this. Should you chose to retype be careful about stray characters and spaces.
User-agent: *
Request-rate: 1/5 # Maximum request rate of 1 page every 5 seconds (12/minute)
Crawl-delay: 5 # Used by several bots (12/minute)User-agent: Googlebot
Disallow: /wp-content/
Disallow: /trackback/
Disallow: /wp-admin/
Disallow: /feed/
Disallow: /archives/
Disallow: /index.php
Disallow: /*?
Disallow: /*.js$
Disallow: /*.inc$
Disallow: /*.css$
Disallow: */feed/
Disallow: */trackback/
Disallow: /page/
Disallow: /tag/
Disallow: /category/
Disallow: /go/
Disallow: /recommends/User-agent: Googlebot-Image
Disallow: /wp-includes/
Allow: /wp-content/uploads/User-agent: Mediapartners-Google*
Disallow:User-agent: Adsbot-Google
Allow: /User-agent: Googlebot-Mobile
Allow: /User-Agent: Googlebot
Disallow: /link.php
Disallow: /gallery2
Disallow: /gallery2/
Disallow: /category/
Disallow: /page/
Disallow: /pages/
Disallow: /feed/
Disallow: /feedUser-agent: ia_archiver
Disallow: /User-agent: duggmirror
Disallow: /
You can see that there is a whole lot of disallow “do not disturb” signs! I don’t want to get into what each line does because you can find that on 100 other websites complete with the associated headache. All you really need to know is that pending special circumstances, this is all you need for now.
So… How To Get The Robots.txt File Onto My Blog Or Website
This is set-it-and-forget-it information and you will only generally do this once ever for a particular blog or website. Bookmark this post so you can find it again when you may wish to set up a second site!
Steps To Install A Robots.txt File
1. Open your self-hosted blog or website using either FTP or your hosting provider’s CPanel File manager.
2. Navigate to the root of your site. If you only have a single blog or site on the account, that is generally the inside of the public_html folder. If you have multiples, you may have a folder with the domain name inside of the public_html folder. (Accidentally putting this in the wrong place wont generally hurt anything, but it simply wont work.)
3. Either open the link to my robots.txt file and save it as a .txt file – or copy paste the contents from above into a .txt file using a simple editor like Notepad. Do NOT use Word. Save the File, naming it robots.txt, if you haven’t done so already.
4. Now upload the robots.txt file to your website host using either the FTP or File Manager. On FTP that is simply dragging it from the left side (your computers files) to the right side (your server’s files)
5. Sit back, content in knowing just how damn smart you are!
A Few Notes
1) Thesis theme has already built the most critical components into the theme. However, there is more you can accomplish by uploading this anyways.
2) This only works on a self-hosted WordPress.org blog… aka no you can not install it on a wordpress.com or blogger.com blog. There are however similar files for other self-hosted platforms such as Joomla.
3) There are a few different theories on what should be in a robots.txt file and you can read more on that here if you’re bored and up for scratching your head.
4) Yes, there are ways to further build on the robots.txt file but this is the basics – what you need to keep you out of trouble.
5) Yes there are plugins that do similar, but we don’t want to take an SEO penalty for further slowing down our site as every single plugin installed does.
Summary of the Robots.txt
The robots.txt file is a simple little file that gets placed in the root (home) directory of your particular site and keeps Google from indexing stuff it’s not smart enough to understand.
This benefits SEO in WordPress because it avoids us taking a penalty for having what appears to be duplicate content … considered SPAM by Google… on our sites.
Simply save the file. Upload it via FTP/File Manager to the root folder of your site. Be pleased with how SEO savvy you’ve become!
If you found this post helpful, I hope you’ll share it with your friends, followers and community! I look forward to your questions below and what other SEO tips you can share with the community here!
Kimberly
Keep up with all the latest social marketing changes and news to accelerate your business!
- WordPress, Facebook, Twitter, & Google+
- Social Marketing
- List Building & Affiliate Marketing
Article by Kim Castleberry
Subscribe to feed via RSS or EMAIL to receive our Newsletter full of articles, tips, reviews and helpful products for your business!
Subscribe to our You tube channel
Hey Kimberly
I us a WordPress plugin to handle my robots.txt file
PC Robots.txt
Take a look and tell me what you think
Peter Fuller MBA invites you to read…Why I started my own Home Business
Well – its correctly placing it at yourdomain.com/robots.txt – and is blocking a very extensive list of the bad bots. However, whoever wrote this is from of the other schools of thought on Google and duplicate content – because as you have seen I’m sure – there is no disallow of /tags/ /categories/ etc. Also if you are using any plugin to mask affiliate links and using perhaps /go/ or /recommends/ in the middle, those need restricted in your robots.txt too as I’ve done above. I’m really surprised to see so much restricted but not what most would consider the basics, so this must be yet another school of thought.
This is also just one more plugin to run, that will affect your load speeds for your entire blogs life, that you could replace with 10 minutes or less of work. This is the kind of plugin that is only necessary for someone that is deathly afraid of the FTP/File Manager or of getting someone to help them. If you’re going for speed, its about avoiding plugins like the plague!
Kimberly Castleberry invites you to read…WordPress SEO – The Why -amp How of the Robotstxt File
Actually you can edit the robots.txt right from the plugin admin panel. I do not accept all of the defaults.
In fact that is why I like it, because I can edit it easily enough online no matter where I am in this world
As for speed, I only have a core set of plugins that I use so it is not an issue.
Peter Fuller MBA invites you to read…Why I started my own Home Business
See thats the thing, if you’re only using a core, you’re probably 10 or less. That means one more is no real issue for you. However, there are lots and lots of plugin addicts in the community – yours truly included (hehe) – that can not give up any more site speed every single day when we can spend a few minutes once (or at absolute worst, once a year). Sounds like you’re on the right track!
Kimberly
Kimberly Castleberry invites you to read…WordPress SEO – The Why -amp How of the Robotstxt File
once again, a brilliant post with info that i didn’t even know i could use. thanks kimberly for keeping me up-to-date and spider free!
hope all’s well w/ you girlfriend !
Melissa McCloud invites you to read…Don’t Move To My Neighborhood
Of course you know I am techo challenged. But You my dear make it easy and always keep me posted to the latest and greatest. I rely on your posts for my information on this. You are a true go-giver and superwoman. In fact, when my clients ask me tech questions, I always steer them to YOU. You are always giving. I am thankful you are in my life.
Thanks so much
Donna
Donna Merrill invites you to read…MARK HOVERSON’S Q-A WEBINAR BLEW MY MIND-
Excellent Kimberly! While I’ve been using a robot.txt file for ages mine is very basic.
Actually, after checking Wassup I’ve just noticed it doesn’t have one at all. Damn! Just goes to show how the small things can slip you buy. Reckon I may use your sample and see how things go. Thanks so much Kim.
Told ya you can teach me stuff.
Sire invites you to read…Reviewing MyLikes As An Advertiser
Zomg! The infamous Sire… knight in shining armor that he is… has forgotten something? *feins horror* Cant be! Hehehe!
Well now you got a quick file you can save and toss up there next time you’re putzing with the FTP client!
Kimberly
Kimberly Castleberry invites you to read…WordPress Plugin Review- Homepage Excerpts
I continue to hear more and more on the imporatance of having your robots.txt set up correctly. Thanks for the straightforward post on how to do just that.
Glad it helped you Tracy! Once you set out to get it in place if you have any questions give me a shout!
Kimberly
I did not know this – THANKS – I will change my robots file now.
Soren
Soren Egstrup invites you to read…STOP spending a Single Minute …
I jumped over to your site and while I was getting ready to leave a comment, I checked and still don’t see your file. If you need any help be sure to give me a shoutout!
Weird, thanks, I try to upload again.
Soren Egstrup invites you to read…STOP spending a Single Minute …
Thanks Kimberly for this nice post and making it so simple to understand it, not only you have provided the details but also a quick brief summary that really helps, thanks again.
Yay! Glad it helped you! Thanks for the feedback!
Kimberly, Nice post! Thanks for taking the time to compose the post and sharing your knowledge. On to the CPanel…
Excellent article Kimberly. I hardly find such long and informative article about robot.txt file. Nice work.
I usually avoid touching robot.txt files unless require because, that’s reason of getting indexed or deindexed in search engines. It is also good way to block some weird search bots. There are also some nice tools available online to help create own robot.txt files.
What do you do if your WordPress install is not in the root folder, but in a subdirectory? I have my WordPress index file in the root, but the rest of my wordpress files in a subdirectory. Thanks.
A robots.txt file can make a huge impact on your WordPress blogs traffic and search engine rank. This is an SEO optimized WordPress robots.txt file. Keep in mind that if you mess up the robots.txt file by blocking too much, you could lose all of your rank.
seph invites you to read…Site Map
I’ve seen some of my friends whom are using simple robots.txt file, they are giving permission to all search engine bots and they are also giving permission to index all the pages, is this useful? or should I use advanced robots.txt file by using any plugins or software? please give me suggestion…
Elton invites you to read…Importance Of Robotstxt File In Search Engine Optimization
There is also a plugin in the official WordPress Plugin Directory called KB Robots.txt. That does the job for everyone and it’s very easy to learn robots.txt syntax and instructions, as you just shown us. Great post!