How Search Engines Work

Google, like other search engines, uses automated software to read, analyze, compare, and rank your web pages. So you need to know what elements and factors Google cares about, and how important these factors are in relation to each other.

This is an important concept: Google uses automated software that looks at code and text, not human beings. This means the visual elements of your website that may matter to you – like layout, color, animation, Flash, and other graphics, are ignored by Google. The Google search engine is like a blind person reading a book in Braille – anything that is graphical, spatial, or visual in nature is simply not seen.

As such, you need to start thinking like the Google search engine.

So What Is a Ranking?
A ranking on a search engine is a web page’s listing and relative placement on a results page (known as a SERP) for a certain search query. As an example, if you type “house plans” into the search box at Google, you will get those listings displayed (10 listings per page by default) that Google deems most relevant to the search phrase house plans, sorted in order of relative importance.

The most relevant and most important web pages are listed in descending order. For Google, page relevancy is dependent on how well a web page “matches” a specific word search. Page importance on the other hand is dependent on the quality and quantity of links that point to your web page from other websites.

If your site does not appear in the top 20 for your most important keywords (search terms), you might as well forget getting much traffic from Google or from any other search engine. Because many people never go past the first page for a search result, you really need to be in the top 10.

It is debatable how much more traffic a #1 ranking gets compared to say, a #3 or a #10 ranking. Those listings “above the fold” on a page (anything higher than #4 or #3 depending on your monitor size and resolution) do get clicked more than those below the fold. Above the fold is anything displayed on the page before you have to start scrolling downward.
A recent study provides some interesting numbers on the subject of ranking vs. percentage of clicks for that position. This study tracked the number of times people clicked on a listing on Google for a given search query:

First Page:
1st position: 30%
2nd position: 15%
3rd position: 7%
4th position: 5%
5th position: 4%
6th position: 4%
7th position: 2%
8th position: 2%
9th position: 3%
10th position: 5%

Second Page:
1stt position: 6%
2nd position: 4%
3rd position: 2%
4th position and beyond <1%

As you can see, if you aren’t on the first two pages, you might as well forget getting clicked. When was the last time you went to the third page of a search query versus just starting a new search query?

When Google Comes Visiting
To be listed in Google’s search database (or index), Google visits your site using automated programs called robots or spiders. Such programs “read” each and every page of your website, starting typically with your home page and then following each link to all other web pages on your site. When a search engine robot or spider visits your site, it is said to crawl or spider your site.

Website crawls are performed by the main Google spider, called Googlebot. The more “popular” your site, the more often it typically is crawled by Google. Highly ranked sites and sites that update content frequently (like news and blog sites) get crawled daily.

If interested, you can check your server log files for the user-agent “Googlebot”. This will tell you when Google crawls your site. You can also check by IP address although this method is not as accurate as Google uses different IP addresses for their robots, which can change over time.

Google updates its main index more or less continuously although major “updates” still happen several times a year. These major updates correspond to major ranking algorithm changes. These updates have all been named – you may have heard about Florida, Bourbon, Allegra or Jagger in the forums.

For new websites, I advise you to make your site live as quickly as possible, even before you are completed. Given that Google prefers sites that are older, it no longer makes sense to wait until every “i” is dotted and “t” is crossed before going live with a new site. Instead, create an overall skeleton of your site, with a reasonably finished Home page and other important pages and make it live. Add new content, or update the content, on at least a monthly basis. Google also prefers sites that add or update content regularly.

This strategy has to do with what is called the Google Sandbox or the aging factor. The Sandbox is a set of filters applied to new websites whereby the site cannot rank well (or at all) for any competitive keywords for 6 – 24 months. Also called the aging delay. New sites can rank well for very niche, unique keyword phrases, such as their company name, but that’s about it. It is for this reason that new sites need to be made live on the Web as soon as possible in order to “start the aging clock”.

**It is critical that your website is up and running when Google visits you by following a link from another site. If your site is down, your listing on Google may disappear until the next update! The reason is that Google thinks your site doesn’t exist and may remove it from the index after a couple of attempts.

How Google Ranks Pages
Google uses a sophisticated and proprietary algorithm for ranking Web sites that uses over 100 different criteria in the calculation, each of which is given a specific weighting which can change over time. Because the algorithm can change, specific techniques that used to work well may no longer work as well over time. This is important to remember when your site’s ranking seems to change for no apparent reason. For this reason, optimizing your site should not be considered as a one-time task. You should always try, test, and refine your efforts.

The Google algorithm can be broken down into two major groups of factors:
On-page (keyword) factors. Keyword factors involve how, where and when keywords are used. Meaning how well your website is optimized for your most important keywords, and if those same keywords appear in your content and in links. Keyword factors determine page relevance.

Off-page (link) factors. These include the quantity and quality of links that point to your site. Link factors determine page importance and are related to Google PageRank (PR). Links play a VERY important role in getting high rankings, particularly for competitive markets.

Very simply put, Google finds pages in its index that are both relevant and important to a search for a particular term or phrase, and then lists them in descending order on search results pages.

On-Page Factors and Page Relevance
Keywords are intrinsically related to search terms – words and phrases that people enter into a search engine to find specific information. Most people enter 2 to 5-word phrases in Google to find what they are looking for. Google in turn analyzes all pages in its index and lists the pages which contain those search terms. Each web page usually contains one or two keywords that are repeated more often than others throughout the site. These keywords dictate the “theme” of a website.

In addition, Google analyzes other sites that contain links to your site. Specifically, Google looks to see if the text of a link (the clickable portion) that points to your site also contain those same keywords.

Off-Page Factors and Page Importance
Page importance is all about links - their quantity, quality, and strength, which we will discuss later on. This part of the algorithm includes Google PageRank (PR).
Google looks for links that point to your site from other websites. Google believes a link from website A to website B is a “vote” for the importance of website B. In this way, other websites add votes for your website, which in turn helps increase a pages PageRank value on your site. Each page on your site has a PR value. Usually the PR value is the highest for the home page as most people will link to your home page rather than another page on your site.

The more web pages that link to your site, and the more important in turn those pages are, the more important Google thinks your site is and hence the higher your PageRank value. Moreover, it is the quality, as well as the quantity, of links that matter – not all links are valued the same. Keep in mind that PageRank is but a single (albeit important) factor used in ranking.

Sites that are highly optimized for on-page factors can outrank sites that are less optimized but have higher PageRank.

PageRank value is assigned after comparing every page in the Google index against one another. This is billions and billions of web pages. Note that PageRank does NOT factor in keywords or phrases used on your site.

Top Things Google Looks For
Although Google looks at over 100 different criteria (which can change in importance over time) for ranking pages, here are the top aspects or elements that are currently deemed a “must-do” if you are serious about a top ranking. Other elements will be discussed later on that are also important. The following are listed in approximate order of importance, with the first two items being more important than the others:

1. Keywords used in link text – both on your site and especially on other websites that point to your site. And the more links you have on other sites that point to your site and that contain your most important keywords, the better, all else being equal.
This is extra important if you are targeting broad, generic or otherwise “competitive” search terms.
2. Keywords used in the title of your Web pages (between the