yacy_grid_crawler - Crawler Microservice for the YaCy Grid

•    Java

The Crawler is a microservices which can be deployed i.e. using Docker. When the Crawler Component is started, it searches for a MCP and connect to it. By default the local host is searched for a MCP but you can configure one yourself. Every loader and parser microservice must read this crawl profile information. Because that information is required many times, we omit a request into the cawler index by adding the crawler profile into each contract of a crawl job in the crawler_pending and loader_pending queue.

scala-parser-combinators - simple combinator-based parsing for Scala

•    Scala

This library is now community-maintained. If you are interested in helping please contact @gourlaysama or mention it on Gitter. As of Scala 2.11, this library is a separate jar that can be omitted from Scala projects that do not use Parser Combinators.

sitemap-php - Library for generating Google sitemap XML files

•    PHP

For the 90's people, i'm keeping this repository as 5.2 compatible. If you need PSR-0 and Composer compatible version, here is a fork that maintained by Evert Pot. Include Sitemap.php file to your PHP document and call Sitemap class with your base domain.

Mvc Xml Sitemap

MVC Sitemap makes it a snap for your ASP.NET MVC based web site to expose a sitemap xml file to search engine crawlers. Simply place a [Sitemap] attribute on all Actions you want crawled and create an action for the sitemap - it's that easy.

sitemap.js - Sitemap-generating framework for node.js

•    Javascript

sitemap.js is a high-level sitemap-generating framework that makes creating sitemap XML files easy.Description specifications. Required fields are thumbnail_loc, title, and description.

sitemap - Sitemap and sitemap index builder

•    PHP

Sitemap and sitemap index builder. After that, make sure your application autoloads Composer classes by including vendor/autoload.php.

parboiled2 - A macro-based PEG parser generator for Scala 2.10+

•    Scala

parboiled2 is a Scala 2.11+ library enabling lightweight and easy-to-use, yet powerful, fast and elegant parsing of arbitrary input text. It implements a macro-based parser generator for Parsing Expression Grammars (PEGs), which runs at compile time and translates a grammar rule definition (written in an internal Scala DSL) into corresponding JVM bytecode.PEGs are an alternative to Context-Free Grammars (CFGs) for formally specifying syntax, they make a good replacement for regular expressions and have some advantages over the "traditional" way of building parsers via CFGs (like not needing a separate lexer/scanner phase).

colly - Fast and Elegant Scraping Framework for Gophers

•    Go

Colly provides a clean interface to write any kind of crawler/scraper/spider.With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

SiteMap Generator

•    ASPNET

An automatic ASP.NET sitemap generator, including dynamic url's, completely configurable. JUST SET AND FORGET!

SiteMap Editor for Microsoft Dynamics CRM 2011

SiteMap Editor for Microsoft Dynamics CRM 2011 helps developer and customizers to configure the Site Map in a graphical way. You'll no longer have to create solution, add component, export, update Xml and reimport the solution to update the SiteMap.

SiteMapMVC

An ASP.NET MVC breadcrumbs & SiteMapPath with controller, action and routeValues implementation

pholcus - [Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software

•    Go

[Crawler for Golang] Pholcus is a distributed, high concurrency and powerful web crawler software.

argonaut - Purely functional JSON parser and library in scala.

•    Scala

Argonaut is a JSON library for Scala, providing a rich library for parsing, printing and manipulation as well as convenient codecs for translation to and from scala data types. Argonaut is licenced under BSD3 (see LICENCE). See more at http://argonaut.io.

Sitemaps.NET

•    ASPNET

Sitemaps.NET is a website plugin that automatically generates an XML sitemap of your content. Sitemaps.NET reuses ASP.NET's sitemap functionality and automatically mirrors changes in your site to search engines. Features include: - Quickly generate XML sitemaps for search eng...

DNNspot Sitemap

Need the easiest way to add a sitemap to your DotNetNuke module? After a very simple install process, drop this on a page and voila - you have a sitemap. Confi

WatchersNET.SiteMap

WatchersNET.SiteMap - A Modern SiteMap / TreeView Module and Skin Object for DotNetNuke®

SimpleSiteMenu - A nested SiteMap UL list

•    ASPNET

Simple nested UL/LI emitting composite web control which you can bind to a SiteMap provider. I have provided some basic CSS and jQuery scripts to style it into a tree view. Code has been derived from this sample http://bryantlikes.com/archive/2006/02/17/4839.aspx

jekyll-sitemap - Jekyll plugin to silently generate a sitemaps

•    Ruby

💡 If you are using a Jekyll version less than 3.5.0, use the gems key instead of plugins. Because the sitemap is added to site.pages, you may have to modify any templates that iterate through all pages (for example, to build a menu of all of the site's content).

laravel-sitemap - Create and generate sitemaps with ease

•    PHP

This package can generate a sitemap without you having to add urls to it manually. This works by crawling your entire site. The generator has the ability to execute JavaScript on each page so links injected into the dom by JavaScript will be crawled as well.

scrooge - A Thrift parser/generator

•    Scala

Scrooge is a thrift code generator written in Scala, which currently generates code for Scala, Java, Cocoa, Android and Lua.It's meant to be a replacement for the apache thrift code generator, and generates conforming, compatible binary codecs by building on top of libthrift. It integrates with the finagle project, exporting stats and finagle APIs, and makes it easy to build high throughput, low latency, robust thrift servers and clients.