house-renting - Possibly the best practice of Scrapy and renting a house (可能是 Scrapy 和租房的最佳实践)

•    Python

Possibly the best practice of Scrapy and renting a house (可能是 Scrapy 和租房的最佳实践)

scrapy-redis - Redis-based components for Scrapy.

•    Python

Redis-based components for Scrapy. You can start multiple spider instances that share a single redis queue. Best suitable for broad multi-domain crawls.

haipproxy - :sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

•    Python

:sparkling_heart: High available distributed ip proxy pool, powerd by Scrapy and Redis

Scrapy - Web crawling & scraping framework for Python

•    Python

Scrapy is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It can be used for a wide range of purposes, from data mining to monitoring and automated testing.

awesome-scrapy - A curated list of awesome packages, articles, and other cool resources from the Scrapy community

A curated list of awesome packages, articles, and other cool resources from the Scrapy community. Scrapy is a fast high-level web crawling & scraping framework for Python. scrapyscript Run a Scrapy spider programmatically from a script or a Celery task - no project required.

scrapyrt - Scrapy realtime

•    Python

HTTP server which provides API for scheduling Scrapy spiders and making requests with spiders. Allows you to easily add HTTP API to your existing Scrapy project. All Scrapy project components (e.g. middleware, pipelines, extensions) are supported out of the box. You simply run Scrapyrt in Scrapy project directory and it starts HTTP server allowing you to schedule your spiders and get spider output in JSON format.

scrapy-proxies - Random proxy middleware for Scrapy

•    Python

Processes Scrapy requests using a random proxy from list to avoid IP ban and improve crawling speed. For older versions of Scrapy (before 1.0.0) you have to use scrapy.contrib.downloadermiddleware.retry.RetryMiddleware and scrapy.contrib.downloadermiddleware.httpproxy.HttpProxyMiddleware middlewares instead.

scrapy-redis - Redis-based components for scrapy that allows distributed crawling

•    Python

Redis-based components for scrapy that allows distributed crawling

scrapy-zhihu-github - scrapy examples for crawling zhihu and github

•    Python

scrapy examples for crawling zhihu and github

scrapyd - A service daemon to run Scrapy spiders

•    Python

Scrapyd is a service for running Scrapy spiders. It allows you to deploy your Scrapy projects and control their spiders using an HTTP JSON API.

scrapy-examples - Multifarious Scrapy examples

•    Python

Multifarious scrapy examples with integrated proxies and agents, which make you comfy to write a spider. There are several depths in the spider, and the spider gets real data from depth2.

ants-go - open source, distributed, restful crawler engine in golang

•    Go

I wrote a crawler engine named ants in python base on scrapy. But sometimes, dynamic language is chaos. So I start to write it in a compile language.

elves - 🎊 Design and implement of lightweight crawler framework.

•    Java

crawler framework

WechatSogou - 基于搜狗微信搜索的微信公众号爬虫接口

•    Python

基于搜狗微信搜索的微信公众号爬虫接口

django-dynamic-scraper - Creating Scrapy scrapers via the Django admin interface

•    Python

Creating Scrapy scrapers via the Django admin interface

distribute_crawler - 使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

•    Python

使用scrapy,redis, mongodb,graphite实现的一个分布式网络爬虫,底层存储mongodb集群,分布式使用redis实现,爬虫状态显示使用graphite实现

quotesbot - This is a sample Scrapy project for educational purposes

•    Python

This is a Scrapy project to scrape quotes from famous people from http://quotes.toscrape.com (github repo). This project is only meant for educational purposes.

HttpProxyMiddleware - A middleware for scrapy. Used to change HTTP proxy from time to time.

•    Python

A middleware for scrapy. Used to change HTTP proxy from time to time. Initial proxyes are stored in a file. During runtime, the middleware will fetch new proxyes if it finds out lack of valid proxyes.

scrapy_jingdong - 用scrapy写的京东爬虫

•    Python

用scrapy写的京东爬虫

scrapy_doc_chs - scrapy中文翻译文档

•    Python

scrapy中文翻译文档