|
||||||||
 |
|
|
 |
|||||
|
1. Why should I read about search engine history?
Understanding where current search engines have came from and why and how they have developed through their history into the most popular tools on the internet helps you gain an overall understanding of what search engines are and what they are trying to achieve. Understanding this will help you to focus your website to the specifics of these search engines - understanding their requirements and why they have them is one of the fundamentals to Search Engine Optimization. To be able to optimize your site to a particular Search Engine you must understand that Search Engine, what it wants and why - this is fundamental, and there is no better place to start than the beginning - the past - the history.
Search engines are one of the unforseen results of the distributed computing network that is now called the World Wide Web (WWW or web). The Web is in fact just a single component of the Internet as a whole, but mostly when people refer to the "Internet" they actually mean the World Wide Web. In the early history of the World Wide Web the only way to pass information from one computer to another on the Internet was via File Transfer Protocol or FTP, this required that one computer knew the address of another. An FTP client could then contact an FTP server (or Deamon) using this exact address and a user could then browse and download the available contents of this distributed machine. This browsing was not as we know it today however, these files were available only as a simple list of files, no graphical browsing or links. The user would then have to disconnect and reconnect their FTP client to another machine to browse the contents there (and of course could only connect to FTP servers for which the address was known), it was a difficult, long winded and labourious task by todays standards. The only real way for someone to find a file, especially a new file, was by someone alerting them to it via email or perhaps a notice on a discussion board. The very first search engine, the granddaddy of them all, was created at McGill University in Montreal by Alan Entage in the early 1990's. It was to be a list of available FTP files and their addresses - an archive - Alan planned to call this list "Archives", but plumped for Archie, as the fad at the time amongst Unix users was to use shortened and sometimes cryptic program and file names. Archie would scour anonymous FTP sites (anonymous FTP sites are those that allow whole world access, those that do not use a username and/ or password to restrict access to the enclosed files) and create an index in a database of all available files. A user searching for a file could then use a simple interface and some regular expression pattern matching to find the location of the required file. This eliminated the requirement to browse around many sites in search of a file you wanted and was the first real leap into the world of indexing internet resourses for the purpose of searching.
As the web developed, DNS (Domian Name Server) was introduced - this is a system whereby a text address or URL (Universal Resourse Locator) and be converted into a numeric location identifier or IP (Internet Protocol) address for a specific computer. Phew - what a mouthful! Basically it works like this; each computer or component on the Internet is allocated an IP address, this is a numerical identifier which is unique to that component. For example: 192.34.55.44, this is not an easy thing to remember however, and so a way was needed to make Internet addresses easy to remember. DNS achieves this by converting domain names into IP adresses and vise versa; when you type the address of a website into your browser a request is sent to a local DNS server which converts it into an IP number, the computer then uses this IP locate the correct website. This introduced the possibility for linking within text; click a link, DNS looks up the associated IP address and directs you straight there - web pages as we know them now were born. And from humble beginnings the massive entangeled web of doucuments that we now call the World Wide Web was born. Matthew Gray's World Wide Web Wanderer was the first autonomous agent on the web, it was designed to track the growth of the internet, initially it counted only web servers but leter it started to capture URL's as well. The World Wide Web Wanderer was creating the first web database and Matthew called it Wandex, Wandex exploited the linked nature of the web, following one link to the next - exactly the same process that is used by modern Web Robots today (A robot is a program that automatically traverses the Web's hypertext structure by retrieving a document, and recursively retrieving all documents that are referenced in that document). There was a fairly significant problem at first though; Wandex was very heavy on resourses, it would often find the same website many times a day and created significant extra load on the web servers. In response to this Martijn Koster created ALIWEB, ALIWEB is the worlds first search engine. Martijn was addressing some of the issues that Wandex created, Wandex had difficultly knowing what a webpage was about - on what search terms should the page be searchable? and which text provides an adequate summary for someone needing a summary? Martijn proposed that the creators of websites should be the ones to help users to track and find their work. Automatic programs like Wandex cannot understand natural language and so ALIWEB requires webmasters to enter a description of their site and submit it. ALIWEB then regularly retrieves all these files from around the world, and combines them into a searchable database. This helps eleminate the problems of increased load that Wandex created and also means that each website has an accurate, natural language description. ALIWEB: "Because the database can be updated regularly (currently once a day) the data is very up-to-date. Since ALIWEB does all the work of retrieveing and combining these files, people only need to worry about descriptions of their own services; so the information is likely to be correct and informative. And as only these small description files need to be gathered there is little overhead." [1] There was clearly potential in these ideas, and as the web continued to grow, there was more and more need for accurate and relevant indexing for the purpose of searching the web. Robots (or spiders as they are also known) started to pop up more and more, some caused controversy by being badly written can causing havok with bandwidth. Three robot (spider) based search engines soon made appearances; JumpStation that crawled the web collecting title and header information, The World Wide Web Worm, which collected just the titles and the silghtly more sophisticated Repository Based Software Engineering Project Spider (RBSE). The RBSE was different because it was the first to place a value on relevance to keywords based on a search. The final problem that needed to be addressed was also a fairly simple one: These early spider based search engines lacked any sort of intelligence, and as such your searches have to be very accurate to get relevant results. If you didn't know exactly what it was you were looking for then you were very unlikely to find it, what was needed was a method of categorising sites so that users could browse through the index to find resourses in the appropriate "area" of their search. EINet Galaxy, (nowadays just Galaxy) was the origional browsable by category directory, this meant that it accepted only URL's that were submitted to a specific category, (in this sense is a true directory). The outcome of this approach is that Galaxy's search results are more accurate and because each site is categorised it is possible to find sites even if your not sure exactly what it is your searching for, this is because human editors can make subjective decisions about a websites location in the directory. The downside is of course that humans are very slow compared to robots and so the resulting directory is much smaller than it's spider based equlivalent. Hang in there, were getting up to date now with a few names that you may recognise..! In April 1994 two "Yahoos" David Filo and Jerry yang, PHd students at Stanford University, found that their personal website that contained lists of their favourite and most visited websites, were receiving many hits each day. People liked their simple method of categorisation and summary, adding a simple search engine Yahoo! was born. In more recent times Yahoo! has been overtaken in terms of useage by Google; the ultimate crawler based search engine. Google's phenominal technology has made it possible to index literaly billions of webpages, combined with their sophisticated PageRank™ system which places a value of "importance" on each page, and very, very quick searching has made Google the undisputed king of search engines. This combination of traditional crawler based search engines such and Google and traditional human based directories such as Yahoo! seems to be the way forward; combining the benefits of both. The mass indexing and huge database sizes of search engines with the human accuracy and categorisation of directories. In a short decade search engines have come a long way...
The revolution in communications that is the Internet has spawned many sub-industries, one of which is Search Engine Optimisation (SEO) and Search Engine Marketing (SEM). In a nutshell SEO is the business of optimising websites so that they return the best results possible from search engines and directories while conforming to the individual rules and regulations of those search engines and directories. In reality this means pushing and bending those rules and regulations as far as possible. SEO often appears at first to be somewhat of a "black art", with people whispering about secret methods and tactics that they use to get the best possible positioning for their pages. You'll find many companies and individuals offering services at often highly inflated rates - but they insist that their "art" remains secret. We hope to blow this apart somewhat, there are no "secrets" - I garuentee you that! Lets see how we arrived at this situation... Search engines rank their results in many many ways - Ideally of course you would like your site to be returned in place #1 on the search engine results page (SERP) for the search terms for which you would like your site to be found (though your goal should initially be a realistic top 10 position). The search engine wants the best/ most relevant page in place #1, so simply all you have to do is make your site the best/ most relevant site and hey presto! Slot #1 is yours. So it really comes down to knowing how a search engine defines "best" - this is the key piece of information that you need. SEO is full of contradiction; by changing your website to suit a search engine does this mean that you are trying to trick the Search engine into thinking that your site is the best? Why am I optimising my site for a search engine? Should I not be optimising for my visitors? Well, of course but if people cant find you then there is no point optimising for them! Search engines have historically kept their algorithms for defining "best" a closely guarded secret, and this has been a major factor in creating the "cloak and dagger" world of SEO, everyone wants to get an advantage on another. Part of this means that some people like to let others think that they know more than they actually do. The role and primary target for the search engine community is to ensure that how they define "best" is also how their users define "best" - and so to some degree the two go hand in hand, and this is a great thing. This means that you can concentrate on producing the best site you possibly can for your visitors - the search engines will recognise this, and send users your way. Until the day when this is reality you need to tweak, adjust and modify your website so that it is optimised for the top search engines - this is where you will get visitors, and of course visitors = sales = income. In a Nutshell - Search Engine Optimisation.
Links > Matthew Gray of the Massachusetts Institute of Technology
[1] Introduction to ALIWEWB (http://aliweb.emnet.co.uk/introduction.html)
|
||||||||||||||
 |
 |