How Do Search Engines Work?

This is a very interesting question! Do you know that how a “Search Engine” works, because every search engine wants to keep their secret with them. As a SEO you should have the knowledge about How a search engine works? If you know how a Search Engine works then you can make your proper plans accordingly and you can easily optimize your site. In 1994 Wide Web Worm was the first web search engine, who claim that they have more then 110,000 index web pages and web accessible documents.
We have seen several improvements in Search Engines Program. In these days we have Crawler-based search engines. Which includes the following three major elements:-
1. The spider or Web Crawler or Robot,
2. The index, and
3. The software.
Each has its own function and together they produce Search Engine Results Pages


1.) The spider:- The term “Spider“ is also known as web crawler or robot. A search engine spider is an automated program that reads web pages and follows any links to other pages within the website. This is often referred to as a site being "spidered" or "crawled". The popular search engine spiders are Googlebot (Google), Slurp (Yahoo!) and MSNBot (MSN Search). Spiders start crawling web pages when you list your site with their search engine database, with a list of page URLs it visits these pages, crawling the code and copy, it adds new pages (links) that it finds on the page to its index. The spider returns to the sites index on a regular basis, scanning for any changes. How often the spider returns is up to the search engines to decide. Website owners do have some control in how often a spider visits their site by making use of a robot.txt file. Search engines first look for this file before crawling a page.


2.) The Growing Index:- An index is like a giant catalogue or inventory of websites containing a copy of every web page and file that the spider finds. If a web page changes, this catalogue is updated with the new information. The latest figure released by Google is 8 billion indexes pages. Spider sometimes takes a while for new pages to be added to its index. Thus, a web page may have been "spidered" but not yet "indexed." Until a page is indexed - added to the index - pages will not be available to those search engines for search engine Page Result (SERP).


3.) The Performing Search Engine or Search Engine Software: - At the end of the day a search engine is a software program designed to sift through billions of pages recorded in its index to find matches to a search query and rank them in an order that it believes is most relevant. Each search engine has developed a set of rules or mathematical equations, called algorithm. This algorithm differs between engines, that’s why different search engines may produce different results for the same query. Exactly how a particular search engine's algorithm works is a kept secret, but some general rules are clear that are often used to increase a website's ranking performance. After applying this algorithm to their index of sites, a search engine comes up with a list of the most relevant results according to the search conducted. This is referred to as search engine optimization. If we want to get good SERP so we have to follow the particular search engine guidelines. These guidelines are related to On Page and Off Page optimization.