awesome-spider - 爬虫集合

  •        1

爬虫集合

https://github.com/facert/awesome-spider

Tags
Implementation
License
Platform

   




Related Projects

go_spider - [爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework

  •    Go

A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014).

Spider Compiler

  •    

Spider Compiler parses the input of a spider programming source file and compiles it (with help of csc.exe; the C#-Compiler) to an exe-file. This project is developed in C#.

SPIDER on Rails

  •    Java

SPIDER on Rails (new name of J2EE Spider) is a open source tool for rapidly developing form-based web applications. See more: http://www.infoq.com/news/2008/03/J2EE-Spider

node-rolling-spider - A library for controlling a Parrot Rolling Spider drone via BLE.

  •    Javascript

There are a few steps you should take when getting started with this. We're going to learn how to get there by building out a simple script that will take off, move forward a little, then land.To connect you need to create a new Drone instance.

node-readability - Scrape/Crawl article from any site automatically

  •    Javascript

In my case, the speed of spider is about 1500k documents per day, and the maximize crawling speed is 1.2k /minute, avg 1k /minute, the memory cost are about 200 MB on each spider kernel, and the accuracy is about 90%, the rest 10% can be fixed by customizing Score Rules or Selectors. it's better than any other readability modules.


Monkey-Spider

  •    Python

The Monkey-Spider is a crawler based low-interaction Honeyclient Project. It is not only restricted to this use but it is developed as such. The Monkey-Spider crawles Web sites to expose their threats to Web clients.

dhtspider - Bittorrent dht network spider

  •    Javascript

Bittorrent dht network infohash spider, for engiy.com[a bittorrent resource search engine]

scrapy-examples - Multifarious Scrapy examples

  •    Python

Multifarious scrapy examples with integrated proxies and agents, which make you comfy to write a spider. There are several depths in the spider, and the spider gets real data from depth2.

php-spider - A configurable and extensible PHP web spider

  •    PHP

The easiest way to install PHP-Spider is with composer. Find it on Packagist. This is a very simple example. This code can be found in example/example_simple.php. For a more complete example with some logging, caching and filters, see example/example_complex.php. That file contains a more real-world example.

not-awesome-es6-classes - A curated list of resources on why ES6 (aka ES2015) classes are NOT awesome

  •    

Reverse-inspired by all of the awesome lists on GitHub, like Awesome, Awesome Awesomeness, Awesome JavaScript, Awesome React, Awesome Cycle.js, Awesome Go, Awesome Elixir, Awesome Elm, etc. While ES6 brings several useful and syntactically pleasing new features to JavaScript, there are many people in the JS community who feel that adding class syntax to the language was a mistake. I share this sentiment, but I have encountered quite a few programmers in the wild who don't agree or simply don't seem to understand why some of us have this opinion. So, I wanted to create an online reference where people could come to learn specifically about this issue and why they might not actually need class syntax in JavaScript.

awesome-linux - :penguin: A list of awesome projects and resources that make Linux even more awesome

  •    

We all know that Linux is awesome, but here's a list of especially awesome things related to the Linux ecosystem. You might also want to check awesome-c, awesome-shell, awesome-sysadmin, and awesome-security.

awesome-dotnet-core - :honeybee: A collection of awesome

  •    CSharp

A collection of awesome .NET Core frameworks, libraries, tools, resources and software.Inspired by awesome, awesome-dotnet, awesome-nodejs, frontend-dev-bookmarks.

awesome-android - A curated list of awesome Android packages and resources.

  •    

A curated list of awesome Android libraries and resources. For general Java libraries have a look at awesome-java. My personal recommendation is (for now) to use the android api to build a native app. Scala can help to build this native apps with cleaner code but it adds to many methods (Multidex required). Kotlin is a modern language with 100% interoperatibility with java projects without multidex. But there are also use cases where alternatives like cross-platform development can be useful.

awesome-yara - A curated list of awesome YARA rules, tools, and people.

  •    

A curated list of awesome YARA rules, tools, and resources. Inspired by awesome-python and awesome-php. YARA is an ancronym for: YARA: Another Recursive Ancronym, or Yet Another Ridiculous Acronym. Pick your choice.

Fork-Awesome - A fork of the iconic font and CSS toolkit (Font-Awesome 4.7)

  •    CSS

Fork Awesome is a full suite of 718 pictographic icons for easy scalable vector graphics on websites, originally created by Dave Gandy and now maintained by a community. Following concerns regarding the development of Font Awesome, the PR Freeze since Oct 2016 and the direction Fort Awesome is taking with the version 5.0 of their project, we are forking Font Awesome (4.7), in order to build on this incredible tool Dave Gandy has given us, while at the same time allowing this project to be run by a distributed community of contributors.

vue-awesome - Awesome SVG icon component for Vue.js, built-in with Font Awesome icons.

  •    Javascript

Awesome SVG icon component for Vue.js, with built-in Font Awesome icons. Vue-Awesome an SVG icon component for Vue.js, with built-in icons courtesy of Font Awesome.

awesome-dotnet - A collection of awesome .NET libraries, tools, frameworks and software

  •    

A collection of awesome .NET libraries, tools, frameworks, and software. Inspired by awesome-ruby, awesome-php, awesome-python, frontend-dev-bookmarks and ruby-bookmarks.

tarantula - a big hairy fuzzy spider that crawls your site, wreaking havoc

  •    Ruby

a big hairy fuzzy spider that crawls your site, wreaking havoc

node-crawler - Web Crawler/Spider for NodeJS + server-side jQuery ;-)

  •    Javascript

Web Crawler/Spider for NodeJS + server-side jQuery ;-)





We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.