A Google bot is a search bot which is used by Google to gather and collect information about your website in order to build a searchable user index for the Google Search Engine. There are two types of Googlebot, “deepbot” and “freshbot” each undertakes a different task when visiting your site.


The “freshbot” crawls through websites looking for new and fresh content. If the “freshbot” discovers new content it will be reported and submitted for inclusion in the “deepbots” crawl list. The “freshbot” visits a website more frequently than the “deepbot”, this allows Google to maximise its resources as the “freshbot” is only looking for changes. The rate at which a “freshbott” returns to the site is dependent on how frequently your sites content changes and how important Google perceives the content of your site. If you’re adding a new page every day and your sites content is perceived as being rich in information and important to Google’s search users, then the frequency at which the “freshbot” will visit your site will increase.


The “deepbot” crawls through your website, trying to follow every link on every page, downloading as many pages as it can and submitting them to the Google Indexers. The “deepbot” spends much longer periods of time on your site and is often responsible for an increase in your sites bandwidth. It also follows new and fresh content links which have been found and submitted by the “freshbot”. The “deepbot” is responsible for correctly submitting the pages of your site to Google for indexing.


In order to make sure your website is fully indexed in Google it is essential to make sure that every page of your site is linked from the main content and navigation. Goggle discovers pages by harvesting the links on every page and crawling each one individually.


After Google has successfully crawled your website it then proceeds to use the information to index its findings within the Google search engine index. This is where your pages will be displayed an made searchable to Google search users by keyword and search term. Google takes into account over 200 criteria, including page rank, in order to identify the context of your site and its content and to index each page in the correct location.

Technorati Tags: crawl, google, google bot, googlebot, Search Engine Optimisation, Search Engine Optimization, searchbot, SEO, seo help, SEO Help & Advice