colly - Elegant Scraper and Crawler Framework for Golang

•    Go

---

Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

scraper framework crawler scraping crawling spider

QueryList - :spider: Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。

•    PHP

---

QueryList is a simple, elegant, extensible PHP Web Scraper (crawler/spider) ,based on phpQuery.

colly - Fast and Elegant Scraping Framework for Gophers

•    Go

---

Colly provides a clean interface to write any kind of crawler/scraper/spider.With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

scraper framework crawler scraping crawling spider parser

headless-chrome-crawler - Distributed crawler powered by Headless Chrome

•    Javascript

---

Crawlers based on simple requests to HTML files are generally fast. However, it sometimes ends up capturing empty bodies, especially when the websites are built on such modern frontend frameworks as AngularJS, React and Vue.js. Note: headless-chrome-crawler contains Puppeteer. During installation, it automatically downloads a recent version of Chromium. To skip the download, see Environment variables.

headless-chrome puppeteer crawler crawling scraper scraping chrome chromium promise headless

Soup - Web Scraper in Go, similar to BeautifulSoup

•    Go

---

soup is a small web scraper package for Go, with its interface highly similar to that of BeautifulSoup.

webscraper webscraping beautifulsoup scraping web-scraping crawler web-crawler

Norconex HTTP Collector - A Web Crawler in Java

•    Java

---

Norconex HTTP Collector is a web spider, or crawler that aims to make Enterprise Search integrators and developers's life easier. It is Portable, Extensible, reusable, Robots.txt support, Obtain and manipulate document metadata, Resumable upon failure and lot more.

crawler web-crawler web-spider search-engine

Norconex HTTP Collector - Enterprise Web Crawler

•    Java

---

Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a repositoriy of your choice (e.g. a search engine). It very flexible, powerful, easy to extend, and portable.

crawler webcrawler spider full-text-search searchengine search-engine

google-play-scraper - Node.js scraper to get data from Google Play

•    Javascript

---

Node.js module to scrape application data from the Google Play store.

google-play scraper crawler api nodejs google play

xcrawler - 快速、简洁且强大的PHP爬虫框架

•    PHP

---

快速、简洁且强大的PHP爬虫框架

crawler spider scraper

pholcus - [Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software

•    Go

---

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

crawler spider multi-interface distributed-crawler high-concurrency-crawler fastest-crawler cross-platform-crawler web-crawler

weibo_terminater - Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything

•    Python

---

Final Weibo Crawler Scrap Anything From Weibo, comments, weibo contents, followers, anything. The Terminator

scraper weibo sina corpus chinese chatbot

grab-site - The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

•    Python

---

grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses wpull for crawling. a dashboard with all of your crawls, showing which URLs are being grabbed, how many URLs are left in the queue, and more.

archiving crawl spider crawler warc

Open Search Server

•    C++

---

Open Search Server is both a modern crawler and search engine and a suite of high-powered full text search algorithms. Built using the best open source technologies like lucene, zkoss, tomcat, poi, tagsoup. Open Search Server is a stable, high-performance piece of software.

crawler webcrawler searchengine search-engine full-text-search spider

Gigablast - Web and Enterprise search engine in C++

•    C++

---

Gigablast is one of the remaining four search engines in the United States that maintains its own searchable index of over a billion pages. It is scalable to thousands of servers. Has scaled to over 12 billion web pages on over 200 servers. It supports Distributed web crawler, Document conversion, Automated data corruption detection and repair, Can cluster results from same site, Synonym search, Spell checker and lot more.

search-engine searchengine distributed web-crawler spider

node-crawler - Web Crawler/Spider for NodeJS + server-side jQuery ;-)

•    Javascript

---

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

go_spider - [爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework

•    Go

---

A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014).

spider crawler schedule pipeline

Crawler-Detect - 🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

•    PHP

---

CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent and http_from header. Currently able to detect 1,000's of bots/spiders/crawlers. Run composer require jaybizzle/crawler-detect 1.* or add "jaybizzle/crawler-detect" :"1.*" to your composer.json.

user-agent crawler spider bots detect

diskover - File system crawler, disk space usage, file search engine and file system analytics powered by Elasticsearch

•    Python

---

diskover is an open source file system crawler and disk space usage software that uses Elasticsearch to index and manage data across heterogeneous storage systems. Using diskover, you are able to more effectively search and organize files and system administrators are able to manage storage infrastructure, efficiently provision storage, monitor and report on storage use, and effectively make decisions about new infrastructure purchases. As the amount of file data generated by business' continues to expand, the stress on expensive storage infrastructure, users and system administrators, and IT budgets continues to grow.

elasticsearch crawler filesystem-visualization filesystem-analysis filesystem-indexer disk-space disk-usage storage-analytics storage filesystem file-indexing duplicatefilefinder metadata duplicate-files botnet file-tagging analytics aws-s3 tree-walker

dhtspider - Bittorrent dht network spider

•    Javascript

---

Bittorrent dht network infohash spider, for engiy.com[a bittorrent resource search engine]

dht bittorrent spider crawler

Pavuk

•    C

---

Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set.

web-grabber crawler web-crawler spider