Chapter 1

Surveying the Search Engine Landscape

IN THIS CHAPTER

Discovering where people search

Understanding the difference between search sites and search systems

Distilling thousands of search sites down to three search systems

Understanding how search engines work

Gathering tools and basic knowledge

You've got a problem. You want people to visit your website; that's the purpose, after all - to bring people to your site to buy your product, or find out about your service, or hear about the cause you support, or for whatever other purpose you've built the site. So you've decided you need to get traffic from the search engines - not an unreasonable conclusion, as you find out in this chapter.

So where do you start? You know you want to have your site appear in Google, of course. but as big as Google is, it isn't everything. A lot of searches are carried out at sites other than Google. But when you start to consider other search options, the field starts to get crowded. There's AOL.com, Yahoo.com, and Bing.com, of course. But there's more; what about DuckDuckGo (a search site focused on privacy), DogPile.com, Ask.com, Baidu.com, Yandex.com, StartPage.com, and SwissCows.com?

And don't forget the nontraditional "search engines." Many searches are carried out at Amazon, Craigslist, eBay, and other shopping-related sites. Then there's sites such as YouTube, Twitter, and Facebook.

So where do you direct your attention? Well, I've got some good news. While you definitely need to consider more than just Google, the overall picture of search engine "targets" can be simplified. The point of this chapter is to take a complicated landscape of scores, maybe hundreds, of search sites and whittle it down into the small group of search engines that really matter. (Search sites? Search systems? Don't worry; I explain the distinction.)

Investigating Search Engines and Directories

The term search engine has become the predominant term for search system or search site, but you need to understand the different types of search, um, thingies that you're going to run across.

Although out on the Interwebs you will hear the term search engine a lot, perhaps almost exclusively, I like to sometimes use the term search site. Why? Because many search sites don't have their own search engines; rather, they partner with a search engine to provide their site visitors with search results.

Take, for instance, AOL.com (www.aol.com). One may be forgiven for thinking that AOL.com is a search engine; after all, it has a big search box right at the top, and if you enter a phrase and press Enter, or click a colored SEARCH button, you get search results.

INDEX ENVY

A few years ago, Yahoo! and Google used to compete to see who had the largest index; Google used to even publish the number of indexed pages on its home page; at one point the statement under the search box said that Google had indexed 15 billion pages.

Oh, the good old days . how things have changed. Now Yahoo! no longer has its own index (it gets search results from Bing), and forget billions of pages; now Google has found trillions of pages! In 2015, Google reported that it had discovered 60 trillion pages, though not all were indexed; today Google's How Search Works page (www.google.com/search/howsearchworks) states that the index itself contains "hundreds of billions of pages," and contains about 100 million gigabytes of data. (It's been saying that for a couple of years at least, so we're getting into fuzzy number territory. The bottom line? The index is yuge!)

However, AOL doesn't own a search engine, despite the fact that you can search at the AOL site. (Indeed, many people do search at AOL, around a couple of hundred million times a month). Rather, AOL gets its search results from the Bing search engine. Another example is EarthLink.net; this site (owned by an Internet Service Provider that used to be one of the top companies back in the 1990s) has a search box, but the search results come from Google. Hence my desire to differentiate between search sites (places where you can search) and search engines (the systems that actually do all the work). It's an important distinction.

Search sites, indexes, and engines

Let me quickly give you a few simple definitions:

• Search site: A website where you can search for information on the web.
• Search engine: A system that collects pages from the web, saves them in a massive database, indexes the information, and provides a mechanism for people to search through the data.
• Search index: The index containing all the information that the engine collected and searches.
• Search directory: A system that contains some basic information about websites, rather than about collected and indexed web pages.

Large search-index companies own thousands of computers that use software known as spiders, searchbots, or robots (or just plain bots) to grab web pages and read the information stored in them. These systems use complex algorithms - calculations based on complicated formulae - to index that information and rank it in search results when people search. Google, shown in Figure 1-1, is the world's most popular search site.

Search directories

Before there were search engines, there were search directories. A directory is a categorized collection of information about websites. Rather than containing information from web pages, it contains information about websites. In fact, before Google was even a twinkle in its fathers' eyes, Yahoo! directory was America's dominant search site; "The Google of the 1990s," as I've seen it described.

FIGURE 1-1: Google, the world's most popular search engine, produced these results.

Directories are not created using spiders or bots to download and index pages on the websites in the directory; rather, for each website, the directory contains information, such as a title, description, and category, submitted by the site owner. The most significant search directories in recent years were owned by Yahoo! (http://dir.yahoo.com) and the Open Directory Project, affectionately known as DMOZ (pronounced "dee-moz") due to its original name - Directory Mozilla - and its domain name, www.dmoz.org; see Figure 1-2; the Open Directory Project was actually a volunteer-managed directory owned by AOL. (You can see an archived version at https://dmoz-odp.org/ if you're interested.) Google used to have a directory, based on DMOZ data, at http://dir.google.com, but that's long gone.

These directories had staff members who examined all the sites in the directory to make sure they were placed into the correct categories and met certain quality criteria; Yahoo! charged $299 a year for the privilege of being listed in their directory.

However, search directories are simply nowhere near as important today as in the past. Yahoo! directory has gone, Google stopped using DMOZ data, and not long after that DMOZ itself closed its doors.

These directories became irrelevant to average users; most users didn't even know they existed . and now they don't.

However, directories may still be useful to your SEO efforts. There are still thousands of small, specialized directories, focusing on particular industries, hobbies, jobs, sports, cities, and so on, and these directories can be an important way to get traffic to your site. Chapter 16 addresses this topic.

FIGURE 1-2: The Open Directory Project, back in its heyday.

Spidered directories

I wasn't sure what to call these things, so I made up a name: spidered directories. As I describe in the preceding section, directories don't contain a full index of the content of the sites pages; rather, they contain a little information about the site itself. In most cases the person who enters the site into the index provides this information, or sometimes a staff member does so. But a number of small search sites actually use spiders (searchbots) to grab a little background information about each site, and even pages within the site, such as titles, descriptions, and keywords. In some cases, this information comes from the meta tags pulled off the pages in the index. (I tell you about meta tags in Chapter 3.) So these are a form of directory, but they are generally created programmatically rather than by site owners requesting inclusion. (Yahoo! Directory and DMOZ were "hand built" using data submitted by site owners.) A number of the smaller systems discussed in Chapter 16 are of this type.

Pay-per-click systems

Many search sites provide pay-per-click (PPC) listings. When you search at Google, for instance,...