Understanding how search engines work is essential if you’re doing SEO. In fact, it’s probably difficult to optimize for something you don’t understand; you’ll never get the results you’re hoping for.
Objection: But everyone knows how Google Search works, and everyone knows how to search for something on Google. Sure, but to do SEO, you need to delve a little deeper. For example, you need to know that search engines work by scanning billions of pages using web crawlers.
Crawlers, also known as spiders or bots, navigate the web and follow links to find new pages. These pages are then added to an index from which search engines pull results.
Put like that, it seems simple, but in reality it is an organized and very complex operation, performed by extremely sophisticated algorithms bordering on artificial intelligence.
1. Search Engine Basics
What are search engines?
Search engines are searchable containers of web content. They consist of two main parts:
- Search Index. A digital library of web page information.
- Search algorithms . Computer programs that rank matching results from a search index.
What is the purpose of search engines?
Every search engine aims to provide the best and most relevant results for users. This is partly how they gain market share.
How do search engines make money?
Search engines have two types of search results:
- Organic results from the search index. You can’t pay to be listed here.
- Paid results from advertisers. You can pay to be listed here.
Every time someone clicks on a paid search result, the advertiser pays the search engine. This process is known as pay-per-click (PPC) advertising, which is why market share matters. More users mean more clicks on ads, and therefore more revenue for the search engine.
Paid results: Businesses pay Google to appear in these results.
Organic results : No one can pay Google to appear here.
2. How search engine indexes are built
Each search engine has its own process for building a search index. Here’s a simplified version of the process Google uses:
Let’s analyze this process to try to understand how it works.
URL
It all starts with the URL (Uniform Resource Locator), essentially the web address of a page on the Internet. Google discovers them in many ways, but the three most common are:
- From backlinks. Google has an index of hundreds of billions (yes, billions) of web pages. If someone links to a new page from a page already known to Google, Google will start finding it there.
- From sitemaps. Sitemaps tell Google which pages and files you consider important.
- From URL submissions. Google allows site owners to request crawling of individual URLs through its Google Search Console .
Crawling (Scanning)
Crawling is the process by which a computer bot called a spider visits and downloads known URLs. Google’s crawler is Googlebot.
Processing and rendering
Processing is how Google understands and extracts key information from crawled pages. To do this, it must render the page, which essentially runs the page’s code to understand how users search.
No one outside of Google knows every detail of this process, but that’s not important—what we really need to know is that it involves extracting links and storing content for indexing.
Indexing
Indexing is the process by which information processed from crawled pages is added to the search index.
The search index is essentially what people look for when they use a search engine. That’s why being indexed by major search engines like Google and Bing is so important. Users can’t find you unless you’re included in the index.
3. How pages are ranked by search engines
Discovering, crawling, and indexing content is just the first part of the puzzle. Search engines also need a way to rank matching results when a user performs a search. This is the job of search algorithms.
What are search algorithms?
Search algorithms are formulas that cross-reference and rank results they deem relevant from the index. Google uses many factors in its complex algorithms.
Key Google Ranking Factors
No one knows all of Google’s ranking factors because Google hasn’t disclosed them. But some key ones are known; let’s take a look.
Backlink
Backlinks are links from one page of a website to another. They are one of Google’s strongest ranking factors.
It’s not just about quantity, though; quality matters too. Pages with a few high-quality backlinks often outrank those with many low-quality backlinks.
Relevance
Relevance is the usefulness of a given result to those searching for it. Google determines this in several ways. At the lowest level, it searches for pages containing the same keywords as the search query, then it also examines interaction data to see if others found the result useful.
Freshness
Freshness is a query-dependent ranking factor. It’s strongest for searches that require new results.
Page speed
Page speed is a ranking factor on desktop and mobile. However, it’s more of a negative ranking factor than a positive one, as it negatively impacts slower pages rather than positively impacts faster ones.
Mobile-friendly
Mobile-friendliness has been a ranking factor on both mobile and desktop since Google moved to mobile-first indexing in 2019.
4. How search engine results are personalized
Google personalizes search results for each user. Information such as your location, language, and search history are used to personalize the results displayed for each user.
Position
Google uses your geographic location to personalize search results, aiming to offer the most locally relevant ones. That’s why, if we try to type “Italian restaurant,” we’ll see local restaurants closest to the location we’ve entered. Google’s assumption is that it’s unlikely that from a given geographic location, you’d fly halfway around the world for lunch.
Language
Google knows it doesn’t make sense to show results in English to Spanish or Italian users. That’s why it ranks localized versions of content (if any) for users who speak different languages.
Search history
Google saves the things you do and the places you go to offer you a more personalized search experience. You can opt out of this by opting out of cookies or by customizing Google’s privacy settings, but most people probably don’t care.
Please remember:
- Search engines consist of two main parts: index and algorithms.
- To build their index, they crawl known pages and follow the links to find new ones.
- The purpose of search algorithms is to return the best and most relevant results.
- The quality of search results is important for building market share.
- No one knows all of Google’s organic ranking factors.
- Key ranking factors include backlinks, relevance, and freshness.
- Google personalizes results based on your location, language, and search history.
