googlebot - Express middleware that returns the resulting html after executing javascript, allowing crawlers to read on the page

  •        4

This module implements a middleware for express that allows to render a full Html/JS/Css version of a page when JS is not available in the client and the site relies heavily on it to render the site, like when using ember/angular/jquery/backbone; I needed to code this for work to be able to deliver a SEO friendly version of the website to the Google Crawler, and found no solution available. Google will replace the hashbang (or the url) with ?_escaped_fragment_= and append the rest of the url there and expects a different, completely rendered version of the site, the middleware will realize when the request has this and instead of retrieving the normal response it will return the full rendered version that phantomJS creates.

https://github.com/dvidsilva/googlebot

Dependencies:

node-phantom : 0.2.5

Tags
Implementation
License
Platform

   




Related Projects

rendora - dynamic server-side rendering using headless Chrome to effortlessly solve the SEO problem for modern javascript websites

  •    Go

Rendora can be seen as a reverse HTTP proxy server sitting between your backend server (e.g. Node.js/Express.js, Python/Django, etc...) and potentially your frontend proxy server (e.g. nginx, traefik, apache, etc...) or even directly to the outside world that does actually nothing but transporting requests and responses as they are except when it detects whitelisted requests according to the config. In that case, Rendora instructs a headless Chrome instance to request and render the corresponding page and then return the server-side rendered page back to the client (i.e. the frontend proxy server or the outside world). This simple functionality makes Rendora a powerful dynamic renderer without actually changing anything in both frontend and backend code. Dynamic rendering means that the server provides server-side rendered HTML to web crawlers such as GoogleBot and BingBot and at the same time provides the typical initial HTML to normal users in order to be rendered at the client side. Dynamic rendering is meant to improve SEO for websites written in modern javascript frameworks like React, Vue, Angular, etc...

angular-seo - SEO for AngularJS apps made easy.

  •    

SEO for AngularJS apps made easy. Based on PhantomJS and yearofmoo's article. You will need PhantomJS to make this work, as it will render the application to HTML.

Revenant - A high level PhantomJS headless browser in Node.js ideal for task automation

  •    Javascript

A headless browser powered by PhantomJS functions in Node.js. Based on the PhantomJS-Node bridge.This library aims to abstract many of the simple functions one would use while testing or scraping a web page. Instead of running page.evaluate(...) and entering the javascript functions for a task, these tasks are abstracted for the user.

phearjs - PhearJS - render dynamic Javascript webpages to JSON with PhantomJS

  •    CoffeeScript

PhearJS renders webpages. It runs a server which supervises a set number of PhantomJS workers that do the actual parsing and evaluation. Many websites rely on AJAX and front-end rendering. When a machine requests a page from such a website it sees a completely different page than you would see when viewing it in a browser.

prerender-node - Express middleware for prerendering javascript-rendered pages on the fly for SEO

  •    Javascript

Google, Facebook, Twitter, Yahoo, and Bing are constantly trying to view your website... but they don't execute JavaScript. That's why we built Prerender. Prerender is perfect for AngularJS SEO, BackboneJS SEO, EmberJS SEO, and any other JavaScript framework. This middleware intercepts requests to your Node.js website from crawlers, and then makes a call to the (external) Prerender Service to get the static HTML instead of the JavaScript for that page.


SiteManager - A Simple SEO Driven CMS

  •    

This is a simple CMS (content management system) built for small business websites. Our primary focus was to keep the HTML wirefame dynamic, content SEO optimized, and provide easy configuration for Google Webmasters, sitemap generation, and Google Analytics.

prerender_rails - Rails middleware gem for prerendering javascript-rendered pages on the fly for SEO

  •    Ruby

Google, Facebook, Twitter, Yahoo, and Bing are constantly trying to view your website... but they don't execute javascript. That's why we built Prerender. Prerender is perfect for AngularJS SEO, BackboneJS SEO, EmberJS SEO, and any other javascript framework. This middleware intercepts requests to your Rails website from crawlers, and then makes a call to the (external) Prerender Service to get the static HTML instead of the javascript for that page.

django-meta - Pluggable app to allow Django developers to quickly add meta tags and OpenGraph, Twitter, and Google Plus properties to their HTML responses

  •    Python

This pluggable app allows Django developers to quickly add meta tags and OpenGraph, Twitter, and Google Plus properties to their HTML responses. django-meta is now maintained by Nephila on github. Old bitbucket repository won't be updated anymore.

prerender-spa-plugin - Prerenders static HTML in a single-page application.

  •    Javascript

Flexible, framework-agnostic static site generation for sites and SPAs built with webpack. 👉 This is the stable 3.x version of prerender-spa-plugin based on puppeteer. If you're looking for the (now-deprecated) 2.x version, based on PhantomJS, take a look at the v2 branch.

phantomjs-gem - Phantomjs via Rubygems: Auto-install phantomjs on demand for current platform

  •    Ruby

I am lazy as hell, and wanted to be able to install PhantomJS via Rubygems/Bundler when using poltergeist. It keeps installations of phantomjs in $HOME/.phantomjs/VERSION/PLATFORM. When you call Phantomjs.path, it will return the path to the phantomjs executable in there. If that is not present, it will first fetch and install the prebuilt packages suitable for the current plattform (currently Linux 32/64 or OS X supported).

NCrawler

  •    DotNet

Simple and very efficient multithreaded web crawler with pipeline based processing written in C#. Contains HTML, Text, PDF, and IFilter document processors and language detection(Google). Easy to add pipeline steps to extract, use and alter information.

mocha-phantomjs - :coffee: :ghost: Run client-side mocha tests in the command line through phantomjs

  •    Javascript

Ariya Hidayat archived phantomjs on March 3rd, 2018. (See this tweet for more info). phantomjs served us all as great headless browser for years, but now with Electron and headless modes for both Chrome and Firefox, we have much better options. mocha-chrome is a project inspired by mocha-phantomjs, so migration should be easy, and you will be running your tests on the same browser that 50% of your users actually use! I highly recommend it. I will accept pull requests still, but I won't be answering issues or doing feature work myself.

sitecheck

  •    Python

Modular web site spider for web developers.

VosaoCMS - simple CMS for Google App Engine

  •    Java

Vosao (vo-za) is a content management system (CMS) that enables you to build web sites and online applications on the Google App Engine platform for Java.

SEOstats - SEOstats is a powerful open source PHP library to request a bunch of SEO relevant metrics

  •    PHP

SEOstats is the open source PHP library to get SEO-relevant website metrics. SEOstats is used to gather metrics such as detailed searchindex & backlink data, keyword & traffic statistics, website trends, page authority, social visibility, Google Pagerank, Alexa Trafficrank and more.

prerender - Node server that uses Headless Chrome to render a javascript-rendered page as HTML

  •    Javascript

Prerender is a node server that uses Headless Chrome to render HTML, screenshots, PDFs, and HAR files out of any web page. The Prerender server listens for an http request, takes the URL and loads it in Headless Chrome, waits for the page to finish loading by waiting for the network to be idle, and then returns your content. The Prerender server can be used in conjunction with our Prerender.io middleware in order to serve the prerendered HTML of your javascript website to search engines (Google, Bing, etc) and social networks (Facebook, Twitter, etc) for SEO. We run the Prerender server at scale for SEO needs at https://prerender.io/.

phantomjs-node - PhantomJS integration module for NodeJS

  •    Javascript

See examples folder for more ways to use this module. v1.0.x used to leverage dnode to communicate between nodejs and phantomjs. This approach raised a lot of security restrictions and did not work well when using cluster or pm2.

phridge - A bridge between node and PhantomJS

  •    Javascript

A bridge between node and PhantomJS. Working with PhantomJS in node is a bit cumbersome since you need to spawn a new PhantomJS process for every single task. However, spawning a new process is quite expensive and thus can slow down your application significantly.

phantom-lambda-template - The bare minimum for a phantomjs app to run on Amazon Lambda.

  •    Javascript

This is a reference implementation of running PhantomJS on AWS Lambda deployed with AWS CodePipeline. PhantomJS needs to be compiled for the OS you plan on running it and this can be painful because of that fact. This could be circumvented with a build server, which is a very personal decision and it was hard to commit to a build server for this little project. Now with AWS CodeBuild, this has become a trivial matter. So we are using AWS Developer tools 100%, AWS created a great walk through: Automating Deployment of Lambda-based Applications I have done my best to automate the walk through, so its simple and repeatable.