Displaying 1 to 4 from 4 results

supercrawler - A web crawler

  •    Javascript

Supercrawler is a Node.js web crawler. It is designed to be highly configurable and easy to use. When Supercrawler successfully crawls a page (which could be an image, a text document or any other file), it will fire your custom content-type handlers. Define your own custom handlers to parse pages, save data and do anything else you need.

robotparser-scala - robotparser-scala implements a parser for the robots.txt file format in Scala.

  •    Scala

robotparser-scala implements a parser for the robots.txt file format in Scala. And then, you have RobotsTxt instance. By default, character encoding is UTF-8.

X.Web.Sitemap - Simple sitemap generator for .NET and .NET Core

  •    CSharp

Below is a more comprehensive example that demonstrates how to create many sitemaps and how to add them to a sitemap index file in a unit-testable fashion.

blog-generator - static blog generator for my blog at https://zupzup.org/

  •    Go

A static blog generator using a configurable GitHub repository as a data-source. The posts are written in markdown with yml metadata attached to them. This is an example repo for the blog at https://zupzup.org/. The tool can be configured using a config file called bloggen.yml. There is a bloggen.dist.yml in the repository you can use as a template.