dht-spider - 一个简单的基于DHT协议的BT磁力链接爬虫

  •        56

一个简单的基于DHT协议的BT磁力链接爬虫

https://github.com/neoql/dht-spider

Tags
Implementation
License
Platform

   




Related Projects

dht - BitTorrent DHT Protocol && DHT Spider.

  •    Go

See the video on the Youtube.It contains two modes, the standard mode and the crawling mode. The standard mode follows the BEPs, and you can use it as a standard dht server. The crawling mode aims to crawl as more metadata info as possiple. It doesn't follow the standard BEPs protocol. With the crawling mode, you can build another BTDigg.

dhtspider - Bittorrent dht network spider

  •    Javascript

Bittorrent dht network infohash spider, for engiy.com[a bittorrent resource search engine]

findcl - Nodejs 实现磁力链接获取 DHT BT爬虫 磁力链接解析 种子解析 资源搜索

  •    Javascript

Nodejs 实现磁力链接获取 DHT BT爬虫 磁力链接解析 种子解析 资源搜索

Bt - Java BitTorrent Done Right!

  •    Java

Bt is a lightweight framework for P2P-lovers and enthusiastic BitTorrent researchers, perfect choice for light enterprise and home usage and experimentation. It offers good performance, reliability and is highly customizable. With Bt you can create a production-grade BitTorrent client in a matter of minutes. Bt is still in its' early days, but is actively developed and designed with stability and maintainability in mind.

peer-tweet - Decentralized feeds using BitTorrent's DHT

  •    Javascript

BitTorrent's DHT is probably one of the most resilient and censorship-resistant networks on the internet. PeerTweet uses this network to allow users to broadcast tweets to anyone who is listening. When you start PeerTweet, it generates a hash @33cwte8iwWn7uhtj9MKCs4q5Ax7B which is similar to your Twitter username (ex. @lmatteis). The difference is that you have entire control over what can be posted because only you own the private key associated with such address. Furthermore, thanks to the DHT, what you post cannot be stopped by any government or institution. Once you find other PeerTweet addresses you trust (and are not spam), you can follow them. This configures your client to store this user's tweets and broadcasts them to the DHT every once in a while to keep their feed alive. This cooperation of following accounts, allows for feeds to stay alive in the DHT network. The PeerTweet protocol also publishes your actions such as I just followed @919c.. or I just liked @9139.. and I just retweeted @5789... This allows the possibility for new users to find other addresses they can trust; if I trust the user @6749.. and they're following @9801.., then perhaps I can mark @9801.. as not spam. This idea of publicly tweeting about your actions also allows for powerful future crawling analysis of this social graph.


bootstrap-dht - DHT bootstrap server

  •    C++

The DHT bootstrap server can be used as an introducer to the bittorrent DHT network. Like the ones running at router.utorrent.com and router.bittorrent.com. BitTorrent clients can use this server to join the DHT, assuming some number of clients are agreeing on using the same server.

dht - Kademlia/Mainline DHT node in Go.

  •    Go

This is a golang Kademlia/Bittorrent DHT library that implements BEP 5. It's typically used by a torrent client such as Taipei-Torrent, but it could also be used by a standalone DHT routers, or for other more creative purposes.

magnetico - Autonomous (self-hosted) BitTorrent DHT search engine suite.

  •    Python

Autonomous (self-hosted) BitTorrent DHT search engine suite. Both programs, combined together, allows anyone with a decent Internet connection to access the vast amount of torrents waiting to be discovered within the BitTorrent DHT space, without relying on any central entity.

BitDHT

  •    C++

BitDHT is a LGPL'd general purpose C++ Distributed Hash Table library. It is designed to take hassle out over creating your own DHT. BitDHT is compatible with bitttorrent's DHT and can leverage this network to bootstrap your own personal DHT.

bittorrent-dht - Simple, robust, BitTorrent DHT implementation

  •    Javascript

Simple, robust, BitTorrent DHT implementation

kadoh - The Kademlia DHT in Javascript for Node.js and Browsers

  •    Javascript

KadOH is a framework to build P2P applications for browsers and node.js. By implementing the basis of the Kademlia DHT, KadOH lets you build distributed web applications for mobile and desktop devices. With its flexible and extensible design, you can easily adapt KadOH to fit your needs. KadOH is available under the MIT License. See the wiki for more informations ! Also take look at our report and be aware that this document may be outdated.

dhtcrawler2 - dhtcrawler is a DHT crawler written in erlang

  •    Shell

dhtcrawler is a DHT crawler written in erlang. It can join a DHT network and crawl many P2P torrents. The program save all torrent info into database and provide an http interface to search a torrent by a keyword. dhtcrawler2 is an extended version to dhtcrawler. It has improved a lot on crawling speed, and much more stable.

Spider Compiler

  •    

Spider Compiler parses the input of a spider programming source file and compiles it (with help of csc.exe; the C#-Compiler) to an exe-file. This project is developed in C#.

SPIDER on Rails

  •    Java

SPIDER on Rails (new name of J2EE Spider) is a open source tool for rapidly developing form-based web applications. See more: http://www.infoq.com/news/2008/03/J2EE-Spider

node-rolling-spider - A library for controlling a Parrot Rolling Spider drone via BLE.

  •    Javascript

There are a few steps you should take when getting started with this. We're going to learn how to get there by building out a simple script that will take off, move forward a little, then land.To connect you need to create a new Drone instance.

node-readability - Scrape/Crawl article from any site automatically

  •    Javascript

In my case, the speed of spider is about 1500k documents per day, and the maximize crawling speed is 1.2k /minute, avg 1k /minute, the memory cost are about 200 MB on each spider kernel, and the accuracy is about 90%, the rest 10% can be fixed by customizing Score Rules or Selectors. it's better than any other readability modules.

Monkey-Spider

  •    Python

The Monkey-Spider is a crawler based low-interaction Honeyclient Project. It is not only restricted to this use but it is developed as such. The Monkey-Spider crawles Web sites to expose their threats to Web clients.

scrapy-examples - Multifarious Scrapy examples

  •    Python

Multifarious scrapy examples with integrated proxies and agents, which make you comfy to write a spider. There are several depths in the spider, and the spider gets real data from depth2.