Displaying 1 to 8 from 8 results

marmot - 💐Marmot | Web Crawler/HTTP protocol Download Package 🐭

  •    Go

If you go get difficult, you can move those files under GOPATH in this project to your Golang env's GOPATH. HTTP Download Helper, Supports Many Features such as Cookie Persistence, HTTP(S) and SOCKS5 Proxy....

awesome-web-scraper - A collection of awesome web scaper, crawler.


A collection of awesome web scaper, crawler. Please, read the Contribution Guidelines before submitting your suggestion.

scrala - :whale: :coffee: :spider: Scala crawler(spider) framework, inspired by scrapy.

  •    Scala

scrala is a web crawling framework for scala, which is inspired by scrapy. You will get the jar in ./target/scala-<version>/.

AlipaySpider-Scrapy - AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

  •    Python

AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

NScrapy - NScrapy is a

  •    CSharp

Below is a sample of NScrapy, the sample will visit Liepin, which is a Recruit web site Based on the seed URL defined in the [URL] attribute, NScrapy will visit each Postion information in detail page(the ParseItem method) , and visit the next page automatically(the VisitPage method). It is not necessary for the Spider writer to know how the Spiders distributed in different machine/process communicate with each other, and how the Downloader process get the urt that need to download, just tell NScrapy the seed URL, inhirt Spider.Spdier class and write some call back, NScrapy will take the rest of the work NScrapy support different kind of extension, including add your own DownloaderMiddleware, config HTTP header, user agent pool.

We have large collection of open source products. Follow the tags from Tag Cloud >>

Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.