Web Crawling

+Web crawling is the systematic browsing of the [World Wide Web](/wiki/world_wide_web) by an automated program. These programs, often called [web spiders](/wiki/web_spider), discover and retrieve web pages, primarily for building and maintaining a [search engine](/wiki/search_engine)'s index.

+## See also

+- [Bot](/wiki/bot)

+- [Data Mining](/wiki/data_mining)

+- [Internet](/wiki/internet)