We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.
supercrawler - A web crawler
- 18
Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use. When Supercrawler successfully crawls a page (which could be an image, a text document or any other file), it will fire your custom content-type handlers. Define your own custom handlers to parse pages, save data and do anything else you need.
https://github.com/brendonboshell/supercrawler| Tags | web-crawler robot crawler sitemap distributed-crawler spider supercrawler |
| Implementation | Javascript |
| License | Apache |
| Platform | OS-Independent |
Related Projects
pholcus - [Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software
[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.
crawler spider multi-interface distributed-crawler high-concurrency-crawler fastest-crawler cross-platform-crawler web-crawlerSquzer - Distributed Web Crawler
Squzer is the Declum's open-source, extensible, scale, multithreaded and quality web crawler project entirely written in the Python language.
crawler distributed-systems downloader file search spiderNorconex HTTP Collector - A Web Crawler in Java
Norconex HTTP Collector is a web spider, or crawler that aims to make Enterprise Search integrators and developers's life easier. It is Portable, Extensible, reusable, Robots.txt support, Obtain and manipulate document metadata, Resumable upon failure and lot more.
crawler web-crawler web-spider search-enginecrawler - A high performance web crawler in Elixir.
A high performance web crawler in Elixir, with worker pooling and rate limiting via OPQ. Below is a very high level architecture diagram demonstrating how Crawler works.
elixir crawler spider scraper scraper-engine offline filesGigablast - Web and Enterprise search engine in C++
Gigablast is one of the remaining four search engines in the United States that maintains its own searchable index of over a billion pages. It is scalable to thousands of servers. Has scaled to over 12 billion web pages on over 200 servers. It supports Distributed web crawler, Document conversion, Automated data corruption detection and repair, Can cluster results from same site, Synonym search, Spell checker and lot more.
search-engine searchengine distributed web-crawler spidernode-crawler - Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Web Crawler/Spider for NodeJS + server-side jQuery ;-)
Norconex HTTP Collector - Enterprise Web Crawler
Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a repositoriy of your choice (e.g. a search engine). It very flexible, powerful, easy to extend, and portable.
crawler webcrawler spider full-text-search searchengine search-engineStorm Crawler - Web crawler SDK based on Apache Storm
StormCrawler is an open source collection of resources for building low-latency, scalable web crawlers on Apache Storm. StormCrawler is a library and collection of resources that developers can leverage to build their own crawlers. The good news is that doing so can be pretty straightforward. Often, all you'll have to do will be to declare StormCrawler as a Maven dependency, write your own Topology class (tip : you can extend ConfigurableTopology), reuse the components provided by the project and maybe write a couple of custom ones for your own secret sauce.
web-crawler apache-storm distributed crawler web-scrapingQueryList - :spider: Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。
QueryList is a simple, elegant, extensible PHP Web Scraper (crawler/spider) ,based on phpQuery.
grab-site - The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses wpull for crawling. a dashboard with all of your crawls, showing which URLs are being grabbed, how many URLs are left in the queue, and more.
archiving crawl spider crawler warcNutch - Highly extensible, highly scalable Web crawler
Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc.
crawler webcrawler searchengine search-engine full-text-searchgo_spider - [爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework
A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014).
spider crawler schedule pipelinecolly - Elegant Scraper and Crawler Framework for Golang
Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.
scraper framework crawler scraping crawling spiderCrawler-Detect - 🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent
CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent and http_from header. Currently able to detect 1,000's of bots/spiders/crawlers. Run composer require jaybizzle/crawler-detect 1.* or add "jaybizzle/crawler-detect" :"1.*" to your composer.json.
user-agent crawler spider bots detectEx-Crawler
Ex-Crawler is divided into 3 subprojects (Crawler Daemon, distributed gui Client, (web) search engine) which together provide a flexible and powerful search engine supporting distributed computing. More informations: http://ex-crawler.sourceforge.net
Scrapy - Web crawling & scraping framework for Python
Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
crawler web-crawler scraping text-extraction spiderPavuk
Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set.
web-grabber crawler web-crawler spiderhuntsman - Super configurable async web spider
Huntsman takes one or more 'seed' urls with the spider.queue.add() method. Once the process is kicked off with spider.start(), it will take care of extracting links from the page and following only the pages we want.
spider crawler crawl huntsman robot ayncweibospider - :zap: A distributed crawler for weibo, building with celery and requests.
:zap: A distributed crawler for weibo, building with celery and requests.
weibospider data-analysis python3 distributed-crawler weibo sinaOpen Search Server
Open Search Server is both a modern crawler and search engine and a suite of high-powered full text search algorithms. Built using the best open source technologies like lucene, zkoss, tomcat, poi, tagsoup. Open Search Server is a stable, high-performance piece of software.
crawler webcrawler searchengine search-engine full-text-search spider
We have large collection of open source products. Follow the tags from
Tag Cloud >>
Open source products are scattered around the web. Please provide information
about the open source projects you own / you use.
Add Projects.
