AlipaySpider-Scrapy - AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

  •        21

AlipaySpider on Scrapy(use chrome driver); 支付宝爬虫(基于Scrapy)

https://github.com/sunhailin-Leo/AlipaySpider-Scrapy

Tags
Implementation
License
Platform

   




Related Projects

scrapy-examples - Multifarious Scrapy examples

  •    Python

Multifarious scrapy examples with integrated proxies and agents, which make you comfy to write a spider. There are several depths in the spider, and the spider gets real data from depth2.

scrapyrt - Scrapy realtime

  •    Python

HTTP server which provides API for scheduling Scrapy spiders and making requests with spiders. Allows you to easily add HTTP API to your existing Scrapy project. All Scrapy project components (e.g. middleware, pipelines, extensions) are supported out of the box. You simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you to schedule your spiders and get spider output in JSON format.

scrapy-redis - Redis-based components for Scrapy.

  •    Python

Redis-based components for Scrapy. You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls.


Scrapy - Web crawling & scraping framework for Python

  •    Python

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

scrapy-proxies - Random proxy middleware for Scrapy

  •    Python

Processes Scrapy requests using a random proxy from list to avoid IP ban and improve crawling speed. For older versions of Scrapy (before 1.0.0) you have to use scrapy.contrib.downloadermiddleware.retry.RetryMiddleware and scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware middlewares instead.

scrapy-redis - Redis-based components for scrapy that allows distributed crawling

  •    Python

Redis-based components for scrapy that allows distributed crawling

scrapy-zhihu-github - scrapy examples for crawling zhihu and github

  •    Python

scrapy examples for crawling zhihu and github

scrapyd - A service daemon to run Scrapy spiders

  •    Python

Scrapyd is a service for running Scrapy spiders. It allows you to deploy your Scrapy projects and control their spiders using an HTTP JSON API.

django-dynamic-scraper - Creating Scrapy scrapers via the Django admin interface

  •    Python

Creating Scrapy scrapers via the Django admin interface

distribute_crawler - 使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

  •    Python

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

quotesbot - This is a sample Scrapy project for educational purposes

  •    Python

This is a Scrapy project to scrape quotes from famous people from http://quotes.toscrape.com (github repo). This project is only meant for educational purposes.

HttpProxyMiddleware - A middleware for scrapy. Used to change HTTP proxy from time to time.

  •    Python

A middleware for scrapy. Used to change HTTP proxy from time to time. Initial proxyes are stored in a file. During runtime, the middleware will fetch new proxyes if it finds out lack of valid proxyes.

scrapy_jingdong - 用scrapy写的京东爬虫

  •    Python

用scrapy写的京东爬虫

scrapy_doc_chs - scrapy中文翻译文档

  •    Python

scrapy中文翻译文档

ants-go - open source, distributed, restful crawler engine in golang

  •    Go

I wrote a crawler engine named ants in python base on scrapy. But sometimes, dynamic language is chaos. So I start to write it in a compile language.