Search Engine Optimization Programs | SEO Tutorial Program

History of Search Engines

When the World Wide Web came into being in the early 1990s, almost twenty years since the ‘ARPANET’ went public, it has thrown open the doors to a virtual and vast new world of information, which was both unfathomable and ever expanding. But, as the number of web pages and the number of people accessing the internet increased, accessing a particular resource from the sea of information became a problem, especially in the event of the person not knowing the URL of a particular website. Necessity has always been the mother of invention; the need for a tool to search the required resource from the millions of pages of information in the World Wide Web was gravely felt and the researches for the same culminated in what is now called the first amongst the first generation search engines, ‘Archie’.

‘Archie’ – created by Alan Emtage in 1990 – used the directory listings of all files existing in the public anonymous FTP websites to create a searchable database of file names. A year later, Gopher, which does the similar purpose of Archie, but with text files, was introduced by Mark McCahill at the University of Minnesota. The programs, ‘Veronica’ and ‘Jughead’, which followed Gopher, worked similarly, on plain text files sent via Gopher.

The world of web search took a decisive turn in 1993, when the MIT student Matthew Gray made the first robot, the World Wide Web Wanderer. It was initially meant to count web servers in the World Wide Web, but was later reprogrammed to search URLs as well, thus forming the first website database, Wandex. ALIWEB released in 1993 was also a robot, but with an added feature allowing users to submit their own URLs for indexing purposes.

Web robots are also called web crawlers, web spiders or web wanderers. In its earlier versions, it was criticized for taking too much bandwidth, often leading to a server crash. But the issue was sorted out in the later generations.

As the World Wide Web started to grow phenomenally by the mid 90s, realizing its financial potential, more players began throwing their hats in the ring. Excite was the first to make their presence felt and it used statistical analysis of word relations to spot a particular web resource. It was followed by Yahoo in 1994, as a website providing a listing of its favorite web pages. The format of the main link or URL followed by a brief description of what is in the site was first introduced by Yahoo. Within a year, funds started to flow in, and soon Yahoo became a corporation. 

Lycos, with its prefix matching, relevance retrieval, and word proximity, announced its entry in almost the same period as Yahoo. It was a large search engine and by 1996, it had indexed over 60 million web pages, the biggest by any search engine of that time. Alta Vista is another search tool that had made headlines – again in the same period - with advanced search options and allowing natural language inquires. It was also the first to provide multimedia search options to find music files, videos, and pictures. 

Irony it may be, the search engine that is touted as the most popular today – the Google – came into existence rather late in the scene, in 1997. It used inbound links to rank websites, but the results it produced were hugely accepted and in another one year’s time, it became the most sought after search engine in the world. Its nearest competitor, MSN Search, and Open Directory – the biggest hand edited catalogue of the web – came into being in 1998.

As of now, Google has the most number of patrons, followed by Yahoo and MSN.

This wind up a brief peep into history of search engines…

In the near future, as the scene consolidates, the number of search engines that remains in the race may come down to the current top few. But, then all will be providing more advanced and diverse features alongside searching, and it is the common netizens who is stand to gain from the final outcome. After all, this is how product economics responds to competition!

Tail Piece: The numbers of search engines that came and vanished without leaving a trace in between are many. But only the important ones have been mentioned in the article.