5 best open source web crawler projects.

We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.

Displaying 1 to 5 from 5 results

Scrapy - Web crawling & scraping framework for Python

Python

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

crawler web-crawler scraping text-extraction spider

pholcus - [Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

crawler spider multi-interface distributed-crawler high-concurrency-crawler fastest-crawler cross-platform-crawler web-crawler

Soup - Web Scraper in Go, similar to BeautifulSoup

soup is a small web scraper package for Go, with its interface highly similar to that of BeautifulSoup.

webscraper webscraping beautifulsoup scraping web-scraping crawler web-crawler

gopa - [WIP] GOPA, a spider written in Golang, for Elasticsearch

GOPA, A Spider Written in Go. First of all, get it, two opinions: download the pre-built package or compile it yourself.

spider crawler lightweight elasticsearch web-crawler crawling web-spider web-scraping scraping

antch - Antch, a fast, powerful and extensible web crawling & scraping framework for Go

Antch, inspired by Scrapy. If you're familiar with scrapy, you can quickly get started. Antch is a fast, powerful and extensible web crawling & scraping framework for Go, used to crawl websites and extract structured data from their pages.

web-crawler crawling scraping framework crawler web-spider