scrape - Depth controllable Web scraper and Sitemap Generator in Go

  •        2

Scrape is minimalistic depth controlled web scraping project. It can be used as command-line tool or integrate it in your project. Scrape also supports sitemap generation as an output. Once the Scraping is done on given URL, the API returns the following structure.

https://github.com/vedhavyas/scrape

Tags
Implementation
License
Platform

   




Related Projects

web-scraper-chrome-extension - Web data extraction tool implemented as chrome extension

  •    Javascript

Web Scraper is a chrome browser extension built for data extraction from web pages. Using this extension you can create a plan (sitemap) how a web site should be traversed and what should be extracted. Using these sitemaps the Web Scraper will navigate the site accordingly and extract all data. Scraped data later can be exported as CSV. When submitting a bug please attach an exported sitemap if possible.

SiteMap Generator

  •    ASPNET

An automatic ASP.NET sitemap generator, including dynamic url's, completely configurable. JUST SET AND FORGET!

sitemap.js - Sitemap-generating framework for node.js

  •    Javascript

sitemap.js is a high-level sitemap-generating framework that makes creating sitemap XML files easy.Description specifications. Required fields are thumbnail_loc, title, and description.

Mvc Xml Sitemap

  •    

MVC Sitemap makes it a snap for your ASP.NET MVC based web site to expose a sitemap xml file to search engine crawlers. Simply place a [Sitemap] attribute on all Actions you want crawled and create an action for the sitemap - it's that easy.

sitemap-php - Library for generating Google sitemap XML files

  •    PHP

For the 90's people, i'm keeping this repository as 5.2 compatible. If you need PSR-0 and Composer compatible version, here is a fork that maintained by Evert Pot. Include Sitemap.php file to your PHP document and call Sitemap class with your base domain.


WatchersNET.SiteMap

  •    

WatchersNET.SiteMap - A Modern SiteMap / TreeView Module and Skin Object for DotNetNuke®

laravel-sitemap - Create and generate sitemaps with ease

  •    PHP

This package can generate a sitemap without you having to add urls to it manually. This works by crawling your entire site. The generator has the ability to execute JavaScript on each page so links injected into the dom by JavaScript will be crawled as well.

sitemap - Sitemap and sitemap index builder

  •    PHP

Sitemap and sitemap index builder. After that, make sure your application autoloads Composer classes by including vendor/autoload.php.

SimpleSiteMenu - A nested SiteMap UL list

  •    ASPNET

Simple nested UL/LI emitting composite web control which you can bind to a SiteMap provider. I have provided some basic CSS and jQuery scripts to style it into a tree view. Code has been derived from this sample http://bryantlikes.com/archive/2006/02/17/4839.aspx

SiteMap Editor for Microsoft Dynamics CRM 2011

  •    

SiteMap Editor for Microsoft Dynamics CRM 2011 helps developer and customizers to configure the Site Map in a graphical way. You'll no longer have to create solution, add component, export, update Xml and reimport the solution to update the SiteMap.

SiteMapMVC

  •    

An ASP.NET MVC breadcrumbs & SiteMapPath with controller, action and routeValues implementation

PHP sitemap generator

  •    PHP

A simple php solution that can automatically generate your website xml sitemap periodically from your database, it can compress your sitemaps and ping search engines after generation. It can be used with many popular RDBMS like mysql postgresql mssql etc

XML Utility for DNN Blog Module

  •    

This is a mini-project which allows you to easily add or generate an XML site map to your DotNetNuke® website for the search engines to use to index your blog. You can also generate and use custom Atom RSS feeds for your blog.

DotNetNuke Search Engine Sitemaps Provider

  •    

The iFinity DotNetNuke Search Engine Sitemaps Provider project generates Search Engine Sitemaps for DotNetNuke installs.

sitemap generator

  •    ASP

A fast and simple sitemaps creater written in ASP,support utf8 and gb2312.you can test new version on http://www.vtalkback.com/sitemap,Type your url ,press submit.save the popup page as 'sitemap.xml'. Upload to your server and tell google where it is.

SharePoint Site Map Generator

  •    

The SharePoint Site Map Generator will automatically generate a sitemap.xml file in the root of your site collection using your Internet zone URL. This will help the content in your public facing web site be discovered by major search engines.

Sitemaps.NET

  •    ASPNET

Sitemaps.NET is a website plugin that automatically generates an XML sitemap of your content. Sitemaps.NET reuses ASP.NET's sitemap functionality and automatically mirrors changes in your site to search engines. Features include: - Quickly generate XML sitemaps for search eng...

DNNspot Sitemap

  •    

Need the easiest way to add a sitemap to your DotNetNuke module? After a very simple install process, drop this on a page and voila - you have a sitemap. Confi

jekyll-sitemap - Jekyll plugin to silently generate a sitemaps

  •    Ruby

💡 If you are using a Jekyll version less than 3.5.0, use the gems key instead of plugins. Because the sitemap is added to site.pages, you may have to modify any templates that iterate through all pages (for example, to build a menu of all of the site's content).

.NET CMS

  •    ASPNET

Microsoftreg; Enterprise class .net CMS multilingual+FORUM+GALLERY+DYNAMIC PDF generator+Responsive Web Design+WIKI+COMMUNITY: Easy user-friendly,ultra fast (10X),dynamic sitemap;multi site in different domain;multi server;HTML5 microdata schema.org; PayPal IPN,SEO,chat,Adsense,templates,plugins. Support: http://cmsaspnet.com/