Welcome to the google dance

 

Googlebot

 

 

Google bot the spider

 

A Googlebot is a search bot used by Google dance. It collects documents from the web to build a searchable index for the Google search engine.

If a webmaster wishes to restrict the information on their site available to a Googlebot, or another well-behaved spider, they can do so with the appropriate directives in a robots.txt file,[1] and by adding the meta tag <META NAME="Googlebot" CONTENT="nofollow"> to the webpage. [2] Googlebot requests to Web servers are discernible from their user-agent string 'Googlebot'.

Googlebot has two versions, deepbot and freshbot. Deepbot, the deep crawler, tries to follow every link on the web and download as many pages as it can to the Google indexers. It completes this process about once a month. Freshbot crawls the web looking for fresh content. It visits websites that change frequently, according to how frequently they change. Currently Googlebot only follows HREF links and SRC links. [3]

Googlebot discovers pages by harvesting all of the links on every page it finds. It then follows these links to other web pages. New web pages must be linked to from another known page on the web in order to be crawled and indexed.

A problem which webmasters have often noted with the Googlebot is that it takes up an enormous amount of bandwidth. This can cause websites to exceed their bandwidth limit and be taken down temporarily. This is especially troublesome for mirror sites which host many gigabytes of data. Google provides "Webmaster Tools" that allow website owners to throttle the crawl rate

A web crawler (also known as a Web spider or Web robot) is a program or automated script which browses the World Wide Web in a methodical, automated manner. Other less frequently used names for Web crawlers are ants, automatic indexers, bots, and worms (Kobayashi and Takeda, 2000).

This process is called Web crawling or spidering. Many sites, in particular search engines, use spidering as a means of providing up-to-date data. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine that will index the downloaded pages to provide fast searches. Crawlers can also be used for automating maintenance tasks on a Web site, such as checking links or validating HTML code. Also, crawlers can be used to gather specific types of information from Web pages, such as harvesting e-mail addresses (usually for spam).

A Web crawler is one type of bot, or software agent. In general, it starts with a list of URLs to visit, called the seeds. As the crawler visits these URLs, it identifies all the hyperlinks in the page and adds them to the list of URLs to visit, called the crawl frontier. URLs from the frontier are recursively visited according to a set of policies

WebCrawler is a metasearch engine that blends the top search results from Google, Yahoo!, Live Search (formerly MSN Search), Ask.com, About.com, MIVA, LookSmart and other popular search engines. WebCrawler also provides users the option to search for images, audio, video, news, yellow pages and white pages. WebCrawler is a registered trademark of InfoSpace, Inc.

WebCrawler was the first full text search engine. It went live on April 20, 1994 and was created by Brian Pinkerton at the University of Washington. It was bought by America Online on June 1, 1995 and sold to Excite on April 1, 1997. WebCrawler was acquired by InfoSpace in 2001 after Excite, then called Excite@Home, went bankrupt. InfoSpace also owns and operates the metasearch engines Dogpile, MetaCrawler and Excite.

WebCrawler was originally a separate search engine with its own database, and displayed advertising results in separate areas of the page. More recently it has been repositioned as a metasearch engine, providing a composite of separately identified sponsored and non-sponsored search results from most of the popular search engines

.
More info here>>

 

 

 

gdanceoogle 

whats this my special keyword

gdanceoogle

Google Dance Tool

The Google Dance Tool is provided as a tool to help you, the webmaster, determine when the Google search engine is spidering the internet. This is extremely useful if you want to know when your site(s) will rank in Google.

The name Google Dance is a term used to describe the index update of the Google search engine. By using the Google Dance Tool , you can easily determine if Google has started their monthly spidering of the internet. Type in a keyword you would like to search for and choose some of Google's data centers to perform the search query on. You will then see a page displaying the search results of all the Google data centers you have chosen. If any of the results between the different data centers contain discrepencies, then you know that Google has started to spider.

We have also written a nice little script that is FREE for you to use that has deamed itself useful. With the use of the Google Web API, it is now possible to perform search queries on the Google search engine to find information concerning a variety of information. These include how many sites link to yours site, search results on any particular keyword, your site's ranking in Google, and much more