Are Search Engine Robots Useful?

Sometimes referred to as 'spiders' or crawlers, automated search engine robots seek out web pages for the user. Just how do they accomplish this and is this of importance? What is the real purpose of these robots?

Robots actually have the same basic functionality that earlier browsers had. Just like these early browsers, search engine robots do not have the ability to do certain things. Robots cannot get past password protected areas. They do not understand frames, Flash movies, nor Images or JavaScript. Even if you use a robot, you have to click the buttons on your website. They can cease to function while using JavaScript navigation or when indexing a dynamically generated URL. A search engine robot retrieves data and finds information and links on the web.

Spiders are able to determine the content of your page by looking at the visible text, the HTML code, and links. Based on the words it finds, the spider determines what the site is about using a complex algorithm to determine what is and isn't important. Spiders also collect links from websites to follow later, which allows them to effectively hop from site to site to site. Since the entire Internet is made up of links between websites, the robots use them to make their way through the Internet as they search.

By collecting and following links, robots manage tn transport themselves all over the Internet. Think of it as an Internet equivalent of the roads we use in our lives. Robots travel on the roads and read the signposts so they know what leads to where.

When the robots return, the information they gathered is assimilated into the search engine's database. Through a complex algorithm, this data is interpreted and web sites are ranked according to how relevant they are to various topics that would be searched for. Some of the bots are quite easy to notice - Google's is the appropriately-named Googlebot, where Inktomi utilizes a more ambiguous bot named Slurp. Others may be difficult to identify at all.

There may be robots that you do not want to visit your website such as aggressive bandwidth grabbing robots and others. The ability to identify individual robots and the number of their visits is useful. Information on the undesirable robots is helpful also. IP names and addresses of search engine robots are listed at the end of this article in a resources section. These robots read the pages on your website by visiting your page and looking at the text that is visible on the page, and then looks at the source code tags such as title tags, meta tags and others. They look at the hyperlinks on your page. From these links, the search engine robot can determine what your page is about. Each search engine has its own algorithm to determine what is important. Information is indexed and delivered to the search engine's database according to how the robot has been set up through the search engine.

Search engines don't update instantly from moment to moment. No, their database updates can vary in the exact timing. However, once you're in there, the bots will make a point to visit you frequently so as to pick up on updates and the like. If your site is down at the time the bot may not be able to update your site in the search engine database, so do keep that in mind. So, robots may be scary things in movies, but as you can see, as far as the Internet goes they're nothing but helpful tools to guide us in going from site to site. Embrace them, learn how to help them be more efficient, and work with them to get your web site highly-ranked so that you can maximize your visitors.

By Justin Harrison

About the author: Justin Harrison is an internationally recognised Internet Marketing Consultant who provides world class SEO Services to website owners. For more information visit: http://www.seorankings.co.za