We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.
colly - Elegant Scraper and Crawler Framework for Golang
- 214
Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.
http://go-colly.org/https://github.com/gocolly/colly
| Tags | scraper framework crawler scraping crawling spider |
| Implementation | Go |
| License | Apache |
| Platform | Windows MacOS Linux |
Related Projects
colly - Fast and Elegant Scraping Framework for Gophers
Colly provides a clean interface to write any kind of crawler/scraper/spider.With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.
scraper framework crawler scraping crawling spider parserScrapy - Web crawling & scraping framework for Python
Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
crawler web-crawler scraping text-extraction spiderruia - Async Python 3.6+ web scraping micro-framework based on asyncio.
Ruia is an async web scraping micro-framework, written with asyncio and aiohttp, aims to make crawling url as convenient as possible.
asyncio aiohttp asyncio-spider crawler crawling-framework spider uvloop ruiaferret - Declarative web scraping
ferret is a web scraping system aiming to simplify data extraction from the web for such things like UI testing, machine learning and analytics. Having its own declarative language, ferret abstracts away technical details and complexity of the underlying technologies, helping to focus on the data itself. It's extremely portable, extensible and fast. The following example demonstrates the use of dynamic pages. First of all, we load the main Google Search page, type search criteria into an input box and then click a search button. The click action triggers a redirect, so we wait till its end. Once the page gets loaded, we iterate over all elements in search results and assign the output to a variable. The final for loop filters out empty elements that might be because of inaccurate use of selectors.
query-language data-mining scraping scraping-websites dsl cdp crawling scraper crawler chromeheadless-chrome-crawler - Distributed crawler powered by Headless Chrome
Crawlers based on simple requests to HTML files are generally fast. However, it sometimes ends up capturing empty bodies, especially when the websites are built on such modern frontend frameworks as AngularJS, React and Vue.js. Note: headless-chrome-crawler contains Puppeteer. During installation, it automatically downloads a recent version of Chromium. To skip the download, see Environment variables.
headless-chrome puppeteer crawler crawling scraper scraping chrome chromium promise headlessQueryList - :spider: Simple, elegant, extensible PHP Web Scraper (crawler/spider),Use the css3 dom selector,Based on phpQuery! 简洁、优雅、可扩展的PHP采集工具(爬虫)，基于phpQuery。
QueryList is a simple, elegant, extensible PHP Web Scraper (crawler/spider) ,based on phpQuery.
grab - Web Scraping Framework
Project Grab is not abandoned but it is not being actively developed. At current time I am working on another crawling framework which I want to be simple, fast and does not leak memory. New project is located here: https://github.com/lorien/crawler First, I've tried to use mix of asyncio (network) and classic threads (parsing HTML with lxml on multiple CPU cores) but then I've decided to use classic threads for everything for the sake of simplicity. Network requests are processed with pycurl because it is fast, feature-rich and supports socks5 proxies. You can try new framework but be aware it does not have many features yet. In particular, its options to configure network requests are very pure. If you need some option, feel free to create new issue.
web-scraping http-client framework pycurl asynchronous networkcrawler - A high performance web crawler in Elixir.
A high performance web crawler in Elixir, with worker pooling and rate limiting via OPQ. Below is a very high level architecture diagram demonstrating how Crawler works.
elixir crawler spider scraper scraper-engine offline filesLulu - [Unmaintained] A simple and clean video/music/image downloader 👾
Sorry for this. Lulu is a friendly you-get fork (⏬ Dumb downloader that scrapes the web).
downloader video python3 crawler scraper crawling scrapingSoup - Web Scraper in Go, similar to BeautifulSoup
soup is a small web scraper package for Go, with its interface highly similar to that of BeautifulSoup.
webscraper webscraping beautifulsoup scraping web-scraping crawler web-crawlergrab-site - The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns
grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses wpull for crawling. a dashboard with all of your crawls, showing which URLs are being grabbed, how many URLs are left in the queue, and more.
archiving crawl spider crawler warcwebmagic - A scalable web crawler framework for Java.
A crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simply the development of a specific crawler.
crawler scraping frameworkgo_spider - [爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework
A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014).
spider crawler schedule pipelinedht - BitTorrent DHT Protocol && DHT Spider.
See the video on the Youtube.It contains two modes, the standard mode and the crawling mode. The standard mode follows the BEPs, and you can use it as a standard dht server. The crawling mode aims to crawl as more metadata info as possiple. It doesn't follow the standard BEPs protocol. With the crawling mode, you can build another BTDigg.
dht spider bittorrent-dht-protocolYioop - Open Source Search Engine Software
Yioop is an open source, PHP search engine capable of crawling, index, and providing search results for hundred of millions of pages on relatively low end hardware. It can index a variety of text formats HTML, RSS, PDF, RTF, DOC and images GIF, JPEG, PNG, etc. It can import data from ARC, WARC, Media-Wiki, Open Directory RDF. It is easily localized to many languages. It has built-in support for new feeds, discussion groups, blogs, and wikis. It also supports mixing indexes to create mash ups.
searchengine search-engine crawler spider searchGrub
Grub Next Generation is distributed web crawling system (clients/servers) which helps to build and maintain index of the Web. It is client-server architecture where client crawls the web and updates the server. The peer-to-peer grubclient software crawls during computer idle time.
crawler webcrawler searchengine search-engine full-text-search spiderGoutte - Goutte, a simple PHP Web Scraper
Goutte is a screen scraping and web crawling library for PHP. Goutte provides a nice API to crawl websites and extract data from the HTML/XML responses.
Norconex HTTP Collector - A Web Crawler in Java
Norconex HTTP Collector is a web spider, or crawler that aims to make Enterprise Search integrators and developers's life easier. It is Portable, Extensible, reusable, Robots.txt support, Obtain and manipulate document metadata, Resumable upon failure and lot more.
crawler web-crawler web-spider search-enginenewspaper - 💡 News, full-text, and article metadata extraction in Python 3. Advanced docs:
Newspaper has seamless language extraction and detection. If no language is specified, Newspaper will attempt to auto detect a language. Check out The Documentation for full and detailed guides using newspaper.
news crawler crawling scraper news-aggregator
We have large collection of open source products. Follow the tags from
Tag Cloud >>
Open source products are scattered around the web. Please provide information
about the open source projects you own / you use.
Add Projects.
