Crawwwler - Open source large scale web crawler

  •        4682

This project is still in its absolute infancy. craWWWler will be a large scale web crawler written in C++ (no MFC). It currently has a very basic plugin architecture controlled by a purposely thin manager. The manager, however, is designed to be more like an ignition switch, occasional pump, and emergency shutdown. The manager is responsible for allowing one or mores plugins to subscribe to the output of other plugins. In this way, the plugins do not have to pass large amounts of data to other plugins via the manager class. Data is only passed on to interested parties. WARNING DO NOT let this loose on the web. Test it on a site you've downloaded onto your local machine and don't let it get anywhere near the WWW yet! It is NOT STABLE! We don't want to go around crashing sites because we don't yet know what we're doing! There are a lot of laws and caveats out there we need to be very aware of. The primary purpose of this software is availability of information, so lets keep it both legal and helpful. p.s. its an eclipse managed build project because I'm new to coding in Linux and don't know any better. Anyone have guidance on this?

http://code.google.com/p/crawwwler

Tags
Implementation
License
Platform

   




Related Projects

vim-olive - Vim Mode Line Verifier


Vim Mode Line Verifier

vim-mash - Motion Activated Search Highlighter for Vim


Motion Activated Search Highlighter for Vim

vim-grillz - Flash your wicked grillz!


Flash your wicked grillz!

vim-foist - Complete whole lines from any partial therein


Complete whole lines from any partial therein

vim-efmc - Vim Error Format Compiler


Vim Error Format Compiler


uzbl-utrs - mkng uzbl tlrbl


mkng uzbl tlrbl

toycsv - A toy CSV parser written in ruby + lexr + racc


A toy CSV parser written in ruby + lexr + racc

tiktok - TikTok provides a simple asynchronous timer object for VimL.


TikTok provides a simple asynchronous timer object for VimL.

tabby - Using Vim's Tabs the Right Way


Using Vim's Tabs the Right Way

SohiVila - GNU source-highlight Vim language


GNU source-highlight Vim language

SinTax - A DSL for generating Vim syntax highlighting files


A DSL for generating Vim syntax highlighting files

rkdots - Generate a graphviz visualisation of the given javascript statement.


Generate a graphviz visualisation of the given javascript statement.

RelNumBar - Show relativenumbers alongside normal numbers in Vim


Show relativenumbers alongside normal numbers in Vim

Punisher - Punisher hurts you where it hurts most - your time.


Punisher hurts you where it hurts most - your time.

noisy - Don't chat quietly (weechat channel noises)


Don't chat quietly (weechat channel noises)

newlisp-manual - Asciidoc version of the newLISP manual


Asciidoc version of the newLISP manual

Land-of-newLISP - Selected snippets from Land of Lisp rewritten in newLISP


Selected snippets from Land of Lisp rewritten in newLISP

Fossilise - Poor man's collaborative editing in Vim


Poor man's collaborative editing in Vim

firstly - Convert Between Numeric, Spelt, and Short & Long Ordinal Forms of Numbers


Convert Between Numeric, Spelt, and Short & Long Ordinal Forms of Numbers