We have collection of more than 1 Million open source products ranging from Enterprise product to small libraries in all platforms. We aggregate information from all open source repositories. Search and find the best for your needs. Check out projects section.
colly - Fast and Elegant Scraping Framework for Gophers
Colly provides a clean interface to write any kind of crawler/scraper/spider.With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.
scraper framework crawler scraping crawling spider parserartoo - artoo.js - the client-side scraping companion.
artoo.js is a piece of JavaScript code meant to be run in your browser's console to provide you with some scraping utilities. The library's full documentation is available on github pages.
scraping datamining webscraperEmbed - Get info from any web service or page
PHP library to get information from any web page (using oembed, opengraph, twitter-cards, scrapping the html, etc). It's compatible with any web service (youtube, vimeo, flickr, instagram, etc) and has adapters to some sites like (archive.org, github, facebook, etc). This package is installable and autoloadable via Composer as embed/embed.
opengraph twitter-cards embeds scraping oembedscrape-it - :crystal_ball: A Node.js scraper for humans.
A Node.js scraper for humans. Please post questions on Stack Overflow. You can open issues with questions, as long you add a link to your Stack Overflow question.
scraper node-scraper scrape it a scraping module for humansScrapy - Web crawling & scraping framework for Python
Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.
crawler web-crawler scraping text-extraction spiderrequests-html - Pythonic HTML Parsing for Humans™
This library intends to make parsing HTML (e.g. scraping the web) as simple and intuitive as possible.
html scraping requests http kennethreitz lxml pyquery css-selectors beautifulsoupscraperjs - A complete and versatile web scraper.
Scraperjs is a web scraper module that make scraping the web an easy job. Try to spot the differences.
scraper scraping webnode-unfluff - Automatically extract body content (and other cool stuff) from an html document
Automatically extract body content (and other cool stuff) from an html document
content-extraction html scraping scrape web-page body-textheadless-chrome-crawler - Distributed crawler powered by Headless Chrome
Crawlers based on simple requests to HTML files are generally fast. However, it sometimes ends up capturing empty bodies, especially when the websites are built on such modern frontend frameworks as AngularJS, React and Vue.js. Note: headless-chrome-crawler contains Puppeteer. During installation, it automatically downloads a recent version of Chromium. To skip the download, see Environment variables.
headless-chrome puppeteer crawler crawling scraper scraping chrome chromium promise headlesscolly - Elegant Scraper and Crawler Framework for Golang
Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.
scraper framework crawler scraping crawling spiderinstagram-scraper - Scrape the Instagram frontend. Inspired from twitter-scraper by @kennethreitz.
This scraper uses requests_html which requires python 3.6 or higher runtime.
instagram no-authentication client scraping python-3-6 requests-htmlwebmagic - A scalable web crawler framework for Java.
A crawler framework. It covers the whole lifecycle of crawler: downloading, url management, content extraction and persistent. It can simply the development of a specific crawler.
crawler scraping frameworkferret - Declarative web scraping
ferret is a web scraping system aiming to simplify data extraction from the web for such things like UI testing, machine learning and analytics. Having its own declarative language, ferret abstracts away technical details and complexity of the underlying technologies, helping to focus on the data itself. It's extremely portable, extensible and fast. The following example demonstrates the use of dynamic pages. First of all, we load the main Google Search page, type search criteria into an input box and then click a search button. The click action triggers a redirect, so we wait till its end. Once the page gets loaded, we iterate over all elements in search results and assign the output to a variable. The final for loop filters out empty elements that might be because of inaccurate use of selectors.
query-language data-mining scraping scraping-websites dsl cdp crawling scraper crawler chromefacebook_data_analyzer - Analyze facebook copy of your data with ruby language
Analyze Facebook copy of your data. Download zip file from Facebook and get info about friends, ranking by message, vocabulary, contacts, friends added statistics and more. It won't work if you use different language because of date formatting, different titles on pages. This script uses nokogiri internally to parse data.
data-science facebook-data-analyzer facebook script scraping facebook-data english-language statistics conversation data-visualization ruby-gempanther - A browser testing and web crawling library for PHP and Symfony
Panther is a convenient standalone library to scrape websites and to run end-to-end tests using real browsers. Panther is super powerful, it leverages the W3C's WebDriver protocol to drive native web browsers such as Google Chrome and Firefox.
scraping e2e-testing webdriver selenium selenium-webdriver symfony chromedriverLulu - [Unmaintained] A simple and clean video/music/image downloader 👾
Sorry for this. Lulu is a friendly you-get fork (⏬ Dumb downloader that scrapes the web).
downloader video python3 crawler scraper crawling scrapingCasperjs - Navigation scripting and testing utility for PhantomJS and SlimerJS
CasperJS is a navigation scripting & testing utility for PhantomJS and SlimerJS (still experimental). It eases the process of defining a full navigation scenario and provides useful high-level functions, methods & syntactic sugar for doing common tasks such as Filling forms, Clicking links, Capturing screenshots of a page, Downloading resources, even binary ones, Writing functional test suites, exporting results as JUnit XML (xUnit) and lot more.
phantomjs headless-browsers headless-testing slimerjs test testing scraping headless-browser automation web-testing screen-captureImageScraper - :scissors: High performance, multi-threaded image scraper
A high performance, easy to use, multithreaded command line tool which downloads images from the given webpage. Note that ImageScraper depends on lxml, requests, setproctitle, and future. If you run into problems in the compilation of lxml through pip, install the libxml2-dev and libxslt-dev packages on your system.
pypi scraper scraping terminal command-line commandline-toolSoup - Web Scraper in Go, similar to BeautifulSoup
soup is a small web scraper package for Go, with its interface highly similar to that of BeautifulSoup.
webscraper webscraping beautifulsoup scraping web-scraping crawler web-crawlerActive Statement Generator (moved to activestatement project)
!!!! Unfortunately, I must have made something really bad to the Team Foundation Server project, because I started having problems with semi-disappearing files on the server. I chose to move the files to the activestatement project !!!! I'm using this project both to handle a...
active-statements ioc microsoft-money ofx regex scraping
We have large collection of open source products. Follow the tags from
Tag Cloud >>
Open source products are scattered around the web. Please provide information
about the open source projects you own / you use.
Add Projects.
