Displaying 1 to 4 from 4 results

http-agent - A simple agent for performing a sequence of http requests in node.js

  •    Javascript

Since http-agent is based on top of request, it can take a set of JSON objects for request to use. If you're looking for more documentation about what parameters are relevant to http-agent, see request which http-agent is built on top of.Each time an instance of http-agent raises the 'next' event the agent is passed back as a parameter. That allows us to change the control flow of pages each time a page is visited. The agent is also passed back to other important events such as 'stop' and 'back'.

algolia-webcrawler - Simple node worker that crawls sitemaps in order to keep an algolia index up-to-date

  •    Javascript

Simple node worker that crawls sitemaps in order to keep an Algolia index up-to-date. It uses simple CSS selectors in order to find the actual text content to index.

pilgrim - Bookmarklet and manual webcrawler to aid in web research

  •    Javascript

Pilgrim is a prototype tool for assisting in web-based research. This project was initiated with generous support from the Knight Foundation Prototype Fund.