Introducing Googlebot

Being found on the Internet is the cornerstone of inbound marketing. Typically, it takes time for the Googlebot to crawl your website and add your new content to the Google index. Here's how website content is crawled and indexed.

What is Googlebot?

Googlebot is Google's web crawling bot. Also referred to as a spider, Googlebot is a large number of computers that fetch, or crawl, billions of pages on the web. During this process, Googlebot collects information about new and updated pages, which it then adds to Google's searchable index. 

What is crawling?

Crawling is the process where Googlebot goes through web content to copy new and updated information to send back to Google. Googlebot finds what to crawl using links and sitemaps. As your site grows and you create more relevant content and get backlinks, your crawl rate increases.

What is indexing?

When Googlebot crawls your site, it detects new and updated pages and updates the Google index. During indexing, Googlebot processes the words on a page, and information such as H1 header tags and ALT image tags. Similar to an index in a library that lists information about everything in the library, Google's index lists all the website pages that Google knows about. To see which pages on your site are in the Google index, you can do a Google search for: site:mywebsite.com. 

How does the Googlebot find new content?

Googlebot starts with web pages indexed during previous crawl processes. As it browses those web pages, it detects links on those pages to add to the list of pages to be crawled. New content on the web is discovered through links and by submitting sitemaps to Google Webmaster Tools. It's important not to submit your sitemaps too frequently, as this could label your site a spam site and search engines will avoid indexing it. 

How long does it take for Google to index your new content?

Every website page is ranked differently depending on what terms a searcher enters. The most important thing is to produce quality content that uses relevant keywords. Indexing your content can take the Googlebot anywhere from several days to a month or more. Occasionally, a site can even take 3 months before it's indexed completely. For this reason, it's a best practice to analyze your inbound marketing campaigns at least 2 months after the campaign completion date.