Article Directory :: Internet Marketing/Online Business Articles

Yellow Mouse Robots Spray: Repel Search Engine Robots From Indexing Your Pages

By Aldian Prakoso

Republish: EasyPublish™ Published: 10Aug2007 Word count: 494 Viewed: 308 time(s) Get Free Content For Your Site

Don't you hate it if your download page or thank you page can be found in Google, Yahoo or MSN? What about your eBook, is it listed in Google, too?

Your result of your hard work can be accessed by everyone because your page is simply indexed by search engines such as Google. You invested your money, time and energy to create your best product but your product can be found by everyone easily without even paying you. You could loose thousands of dollars. You don't want this happen to you, do you?

So here comes the yellow mouse robots spray. Repel search engine robots from indexing your pages!

Robots.txt

Robots.txt is a simple text file that tells the search engine robots not to crawl certain directories and pages of your site.

When a robot crawls your site it looks for the robots.txt file. If it doesn't find one it assumes automatically that it may crawl and index the entire site. Not having a robots.txt file can also create unnecessary 404 errors (page not found error) in your server logs, making it more difficult to track the real 404 errors.

Creating robots.txt file is easy.

1. Open your notepad or other text editor (not HTML editor).

2. Type this following to prevent all robots to crawl or index your certain directory:

User-agent:*

Disallow:/directory/

where directory is the directory that you don't want to be crawled or indexed by search engine robots.

3. Save the file as robots.txt.

4. Upload it to your domain root directory so that it looks like this: http://yourdomain.com/robots.txt

5. Confirm if everything is correct using robots.txt validator. (You can easily find robots.txt validator using Google Search).

For more information on robots.txt, you can visit http://www.robotstxt.org.

Are these two tricks enough to protect your product download page, thank you page and private page? Perhaps, because nothing is perfect in this world.

So I'll give you another trick to double protect your precious page.

Meta tags

Another trick, which may be more convenient for page usage, adds a META tag to an HTML page to tell search engine robots not to index the page.

To prevent all robots from indexing a page on your site, place this following META tag into the HEAD section of your page:

META NAME="ROBOTS" CONTENT="NOINDEX, NOFOLLOW"

(Add "" after the code).

That should helps protect your precious pages.

One more trick, exclusive to blog only!

Disallow Pings

All of those tricks above are useless if you still allow pings for your blog page or blog post.

Disallow pings for your precious page.

If you use Wordpress:

1. Expand your discussion tab.

2. Uncheck "Allow Pings".

These tricks should help you protect your product download page, thank you page and private page. However, these tricks don't protect your precious page if your buyer adds it to his Internet Explorer's Favorite, Firefox's Bookmarks or social bookmarks and share the link with his friends.

EasyPublish™ this article - publishers click here

More articles by Aldian Prakoso

Free Report!
Ten Essential Secrets Of Article Marketing ... Grab Your Free
Copy Now:

Need Content?
Regular Top Quality Content for your Blog, Ezine or Website ...
Delivered Direct,
For Free!
Click For Details