Google Crawling, indexing, and ranking

Nerdcore Computers Google Crawling, Indexing and ranking

So let's talk about how search engines actually work. Sure, Google's algorithm is extremely complex, but in its simplest form, Google is really just a pattern detection program. When you search for a keyword phrase, Google is going to provide you with a list of websites that matches the pattern that's related to your search.

Most people don't realize that when they do a search on Google, they're not actually searching the live web. They're actually searching Google's index of the web… the stored copy of the sites that Google has crawled.

Google uses little programs called spiders to crawl the web. The spiders are how Google actually discovers content. Basically, a spider will start on a page and check out all the content on that page, and then is follows the links on that page and looks at the content on those pages. Then it follows the links on those pages to discover even more content, and then the links on those pages lead to more content… The spiders keep crawling at a mind-blowingly massive scale until most of the internet has been crawled.

As the spiders crawl all of these pages, a copy is stored on Google's servers. This is Google's index, and it's stored in a way that allows Google to quickly search through the billions of pages in the index. When you search for a certain phrase, Google checks the index to get a list of every page that's related to your search phrase.

Here's where ranking comes into play - this is the part that SEOs work to influence. Google uses an algorithm to look at that list of pages it just pulled from the index and rank the pages based on relevancy.

Let's say you did a search for “Denver omelette recipe” - Google's going to search the index and come up with a list of every website with a recipe for a Denver omelette. Then it's going to use the algorithm to sort that list so that the most relevant sites will be at the top.

There are hundreds of ranking factors, each with a different assigned weight or value. The algorithm will look at a few hundred different factors that influence relevancy, like the content on the page, the number of other sites that link to that page, and the overall quality of the website.

When we do SEO, we're hoping to influence those relevancy scores. We know that if we optimize the right signals, Google's algorithm will decide that the page is more relevant - that it's a better answer to the question being asked - and show that page higher than the other options in the index.

Nerdcore Computers Author