1. Grab
Crawling is the process by which Googlebot discovers new web pages and updates them to add the web pages to the Google index.
We use many computers to fetch (or "crawl") the large number of pages on our website. The program that performs the acquisition task is called Googlebot (also known as a robot or information collection software). Googlebot uses an algorithm to crawl: a computer program determines which websites to crawl, how often, and how many pages to fetch from each site.
Google's crawling process is based on a list of web page URLs, which is formed during the previous crawling process and is continuously expanded with Sitemap data provided by the webmaster. As Googlebot visits each website, it detects the links on each page and adds them to its list of pages to crawl. New sites, changes to existing sites, and dead links are logged and used to update the Google index.
Google doesn't allow payment for crawling your site more frequently. We distinguish between our search business and our for-profit AdWords services.
2. Indexing
Googlebot processes each web page it crawls so that it compiles into a large indexed list all the words it finds and the positions of those words on each page. Additionally, we process information in key content tags and attributes, such as TITLE tags or ALT attributes. Googlebot can handle many types of content, but not all. For example, we are unable to process the content of certain Rich Media files or dynamic web pages.
3. Provide results
When a user enters a query, our computers search our index for matching pages and return results that we believe are most relevant to the user's search. Relevance is determined by more than 200 factors, one of which is the PageRank of a given web page. PageRank is a measure of how important a page is to incoming links from other pages. Simply put, individual links from other sites that point to pages on your site make up your site's PageRank. Not all links are of equal value: Google is committed to continually improving the user experience by calling out link spam and other behavior that can negatively impact search results. Links that are assigned based on the quality of the content you provide are the best links.
In order for your site to rank well in search results pages, it's important to ensure that Google is crawling and indexing your site correctly. Our Webmaster Guidelines outline some best practices that can help you avoid common problems and improve your site's rankings.
Google's Related Search, Spelling Suggestions, and Google Suggestions features are designed to help users save time searching by showing related terms, common misspellings, and common queries. Similar to our google.com search results, the keywords used in these features are automatically generated by our web crawlers and search algorithms. We only display these suggestions if we think they will save the user time. If a site ranks higher for a certain keyword, it's because we've algorithmically determined that its content is more relevant to the user's query.