Web hosting Glossary

All About Web Hosting >> Web hosting Glossary

Definitions of web crawler:

A web crawler is an automated program that accesses a web site and traverses through the site by following the links present on the pages.
matra.sourceforge.net/misc/glossary.php
bot that visits publicly accessible websites following all links it comes across collecting data for search engine ‘indexes’. A spider discovers new sites and updates information from sites previously visited. A spider can also be used to check links within a website.
members.optusnet.com.au/~webindexing/Webbook2Ed/glossary.htm
A Web indexing program that builds an index by following hyperlinks continuously from Web page to Web page.
net.pku.edu.cn/~webg/DigLib/new/glossary.html
this is an automatic program (sometimes called a "robot") which explores the World Wide Web, following the links and searching for information or building a database - such programs are often used to build automated indexes for the Web, allowing users to do keyword searches for Web documents - it is also the name used by one of the most popular publically available keyword searching engines - this can be accessed using the URL: http://www.webcrawler.com/ or http://webcrawler. ...
www.archivemag.co.uk/gloss/W.html
A Web crawler searches through all the Web servers it knows to find information about a particular topic requested by the user.
www.shiftnetworks.com/glossary.cfm
See WebCrawler for the specific search engine of that name.----A web crawler (also known as a web spider or ant) is a program which browses the World Wide Web in a methodical, automated manner. Web crawlers are mainly used to create a copy of all the visited pages for later processing by a search engine, that will index the downloaded pages to provide fast searches.
en.wikipedia.org/wiki/Web_crawler

Site Map