We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.
Norconex HTTP Collector - Enterprise Web Crawler
Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a repositoriy of your choice (e.g. a search engine). It very flexible, powerful, easy to extend, and portable.
crawler webcrawler spider full-text-search searchengine search-engineNutch - Highly extensible, highly scalable Web crawler
Nutch is open source web-search software. It builds on Lucene Java, adding web-specifics, such as a crawler, a link-graph database, parsers for HTML and other document formats, etc.
crawler webcrawler searchengine search-engine full-text-searchGrub
Grub Next Generation is distributed web crawling system (clients/servers) which helps to build and maintain index of the Web. It is client-server architecture where client crawls the web and updates the server. The peer-to-peer grubclient software crawls during computer idle time.
crawler webcrawler searchengine search-engine full-text-search spiderOpen Search Server
Open Search Server is both a modern crawler and search engine and a suite of high-powered full text search algorithms. Built using the best open source technologies like lucene, zkoss, tomcat, poi, tagsoup. Open Search Server is a stable, high-performance piece of software.
crawler webcrawler searchengine search-engine full-text-search spiderArachnode.net
An open source .NET web crawler written in C# using SQL 2005/2008. Arachnode.net is a complete and comprehensive .NET web crawler for downloading, indexing and storing Internet content including e-mail addresses, files, hyperlinks, images, and Web pages.
crawler webcrawler searchengine search-engine full-text-searchASPseek
ASPseek is an Internet search engine software developed by SWsoft.ASPseek consists of an indexing robot, a search daemon, and a CGI search frontend. It can index as many as a few million URLs and search for words and phrases, use wildcards, and do a Boolean search. Search results can be limited to time period given, site or Web space (set of sites) and sorted by relevance (PageRank is used) or date.
crawler webcrawler searchengine search-engine full-text-search spidermnoGoSearch
mnoGoSearch for UNIX consists of a command line indexer and a search program which can be run under Apache Web Server, or any other HTTP server supporting CGI interface. mnoGoSearch for Unix is distributed in sources and can be compiled with a number of databases, depending on user's choice. It is known to work on a wide variety of the modern Unix operating systems including Linux, FreeBSD, Mac OSX, Solaris and others.
crawler webcrawler searchengine search-engine full-text-searchHeritrix
Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web crawler project. Heritrix is designed to respect the robots.txt exclusion directives and META robots tags, and collect material at a measured, adaptive pace unlikely to disrupt normal website activity.
crawler webcrawler searchengine search-engine full-text-searchhttp-agent - A simple agent for performing a sequence of http requests in node.js
Since http-agent is based on top of request, it can take a set of JSON objects for request to use. If you're looking for more documentation about what parameters are relevant to http-agent, see request which http-agent is built on top of.Each time an instance of http-agent raises the 'next' event the agent is passed back as a parameter. That allows us to change the control flow of pages each time a page is visited. The agent is also passed back to other important events such as 'stop' and 'back'.
http-agent iterator http webcrawlersitemapper - parses sitemaps for Node.JS
Parse through a sitemaps xml to get all the urls for your crawler.
sitemap sitemap-xml parse xml robots.txt sitemaps crawlers webcrawleralgolia-webcrawler - Simple node worker that crawls sitemaps in order to keep an algolia index up-to-date
Simple node worker that crawls sitemaps in order to keep an Algolia index up-to-date. It uses simple CSS selectors in order to find the actual text content to index.
algolia webcrawler indexing search-engine algolia-webcrawler web-crawler searchredditLikedSavedImageDownloader - Download all of your reddit Liked/Upvoted and Saved images to disk for hoarding!
This repository includes a simple web server interface. Unlike the main script, the server is supported in Python 3 only. To use it, install tornado via pip3 install tornado then run python3 LikedSavedDownloaderServer.py. The interface can be seen by visiting http://localhost:8888 in any web browser.
reddit praw imgur images webcrawler offline tumblrRcrawler - An R web crawler and scraper
Rcrawler is an R package for web crawling websites and extracting structured data which can be used for a wide range of useful applications, like web mining, text mining, web content mining, and web structure mining. So what is the difference between Rcrawler and rvest : rvest extracts data from one specific page by navigating through selectors. However, Rcrawler automatically traverses and parse all web pages of a website, and extract all data you need from them at once with a single command. For example collect all published posts on a blog, or extract all products on a shopping website, or gathering comments, reviews for your opinion mining studies. More than that, Rcrawler can help you studies web site structure by building a network representation of a website internal and external hyperlinks (nodes & edges). Help us improve Rcrawler by asking questions, revealing issues, suggesting new features. If you have a blog write about it, or just share it with your collegues.
r rpackage crawler scraper webcrawler webscraping webscraper webscrapping crawlerspilgrim - Bookmarklet and manual webcrawler to aid in web research
Pilgrim is a prototype tool for assisting in web-based research. This project was initiated with generous support from the Knight Foundation Prototype Fund.
bookmarklet webcrawler are.na readabilitykrawler - A web crawling framework written in Kotlin
Krawler is a web crawling framework written in Kotlin. It is heavily inspired by crawler4j by Yasser Ganjisaffar. The project is still very new, and those looking for a mature, well tested crawler framework should likely still use crawler4j. For those who can tolerate a bit of turbulence, Krawler should serve as a replacement for crawler4j with minimal modifications to existing applications. Using the Krawler framework is fairly simple. Minimally, there are two methods that must be overridden in order to use the framework. Overriding the shouldVisit method dictates what should be visited by the crawler, and the visit method dictates what happens once the page is visited. Overriding these two methods is sufficient for creating your own crawler, however there are additional methods that can be overridden to privde more robust behavior.
webcrawler kotlin framework crawler4j link-checker web-crawler web-crawlingspiderman - your friendly neighborhood web crawler
Spiderman is a Ruby gem for crawling and processing web pages. Spiderman works with ActiveJob out of the box. If your crawler class inherits from ActiveJob:Base, then requests will be made in your background worker. Each request will run as a separate job.
http crawler spider web-crawler nokogiri web-scraping webcrawler webscraping spider-framework crawler-engine httprbScrapio - Asyncio web crawling framework. Work in progress.
Asyncio web scraping framework. The project aims to make easy to write a highly performant scrapers with little knowledge of asyncio, while giving enough flexibility so that users can customise behaviour of their scrapers. It also supports Uvloop, and can be used in conjunction with Splash the JavaScript rendering solution from ScrapingHub. The project can be installed using Pip.
python3 asyncio webcrawler
We have large collection of open source products. Follow the tags from
Tag Cloud >>
Open source products are scattered around the web. Please provide information
about the open source projects you own / you use.
Add Projects.
