Lookout! Robots are Coming...
mkLINK Internet Marketing Tips(29 March 2007)

What Are They?

Known as robots, trawlers, crawlers, spiders, wanderers, worms and such; their job is to traverse the internet, retrieving and/or reviewing documents.

The Good ones help make sense of the Internet. Search Engines like Google use them to go through your website and help them decide how to classify your website and how to rank you. Others check your site for mistakes, missing links, updated content to report - the list is endless.

The Bad ones look for weaknesses to exploit so that the owner can hack you or spam you or 'harvest' your email address to sell.(And lots of other annoying things like that)

Why should you care?

Because you want to get lots of natural search engine traffic.

Although there are a few directories that historically reviewed websites by human operators(and some continue to do so) such as DMOZ, Yahoo etc - the vast majority of indexing is automated. Not surprising when you consider the sheer volume of stuff they have to classify and review.

Which means robots are reviewing your website.

So be nice to them!

What should you do?

Assuming you want the robots to trawl your site, you should make it as easy as possible for them to do so. They basically follow links, so :

• Make an HTML site map
• Add it to your Home Page
• Let the Robots do their Job

Remember, Search Engine Robots LIKE Links, Text, ALT Tags, Site Maps, Titles, Meta Tags and most other text based stuff.

They DON'T Like Javascript, FLASH, Animations, Images etc.

That's not to say you can't have multimedia - just be aware that Spiders need html and Links too.

Don't like Spiders?

You can also request robots stop spidering your website too, by making a file called robots.txt and adding various exclusions there - for more details visit http://en.wikipedia.org/wiki/Robots.txt

It may be that you don't want them to index certain parts of your site or show up in your server logs for example.

How do You know if You've been visited by a robot?
Easy - look at your server logs.

Check them for sites that retrieve lots of documents, esp. quickly.
If you notice a site repeatedly checking for the file '/robots.txt' chances are that is a robot too.

If you'd like to learn more about Website Marketing - Just Contact me!
01454 852414 or visit mklink.com/contact2.php?source=newsletter

'till next time,

Mike Knight.

Want to know More?
Simply reply to this email, call me on 01454 852414 or visit my site :