awwwards-stream - scrape Awwwards data

  •        13

Creates a readable stream of Awwwards.com data by scraping their HTML.⚠️ This is fragile and should only be used for offline experimentation / artistic purposes. It is not an official API and you should rate limit your requests to keep stress off the Awwwards servers. It may break at any point and should not be used in a live Node.js server.

https://github.com/Jam3/awwwards-stream

Dependencies:

cheerio : ^0.20.0
defined : ^1.0.0
from2 : ^2.1.1
got : ^6.1.1
moment : ^2.11.2
url-join : 0.0.1

Tags
Implementation
License
Platform

   




Related Projects

autoscraper - A Smart, Automatic, Fast and Lightweight Web Scraper for Python

  •    Python

This project is made for automatic web scraping to make scraping easy. It gets a url or the html content of a web page and a list of sample data which we want to scrape from that page. This data can be text, url or any html tag value of that page. It learns the scraping rules and returns the similar elements. Then you can use this learned object with new urls to get similar content or the exact same element of those new pages. It's compatible with python 3.

Polr - A modern, powerful, and robust URL shortener

  •    PHP

Polr is an intrepid, self-hostable open-source link shortening web application with a robust API. It allows you to host your own URL shortener, to brand your URLs, and to gain control over your data. Polr is especially easy to use, and provides a modern, themable feel.

grab-site - The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

  •    Python

grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses wpull for crawling. a dashboard with all of your crawls, showing which URLs are being grabbed, how many URLs are left in the queue, and more.

ASPseek

  •    C++

ASPseek is an Internet search engine software developed by SWsoft.ASPseek consists of an indexing robot, a search daemon, and a CGI search frontend. It can index as many as a few million URLs and search for words and phrases, use wildcards, and do a Boolean search. Search results can be limited to time period given, site or Web space (set of sites) and sorted by relevance (PageRank is used) or date.

url-pattern - easier than regex string matching patterns for urls and other strings

  •    CoffeeScript

easier than regex string matching patterns for urls and other strings. turn strings into data or data into strings.a pattern is immutable after construction. none of its methods changes its state. that makes it easier to reason about.


VosaoCMS - simple CMS for Google App Engine

  •    Java

Vosao (vo-za) is a content management system (CMS) that enables you to build web sites and online applications on the Google App Engine platform for Java.

ness - Deploy web sites and apps to your own cloud account effortlessly.

  •    TypeScript

Ness is the easiest way to stand up a production-ready web site on your own cloud infrastructure. Ness ships with two commands: deploy and destroy. By default, the deploy command will stand up a simple S3 website and output the site URL. Running destroy will tear it down and put your AWS account back in the state that it was prior to deploy.

ISO-3166-Countries-with-Regional-Codes - ISO 3166-1 country lists merged with their UN Geoscheme regional codes in ready-to-use JSON, XML, CSV data sets

  •    Ruby

These lists are the result of merging data from two sources, the Wikipedia ISO 3166-1 article for alpha and numeric country codes, and the UN Statistics site for countries' regional, and sub-regional codes. In addition to countries, it includes dependent territories. The International Organization for Standardization (ISO) site provides partial data (capitalised and sometimes stripped of non-latin ornamentation), but sells the complete data set as a Microsoft Access 2003 database. Other sites give you the numeric and character codes, but there appeared to be no sites that included the associated UN-maintained regional codes in their data sets. I scraped data from the above two websites that is all publicly available already to produce some ready-to-use complete data sets that will hopefully save someone some time who had similar needs.

URL-Detector - A Java library to detect and normalize URLs in text

  •    Java

The url detector is a library created by the Linkedin Security Team to detect and extract urls in a long piece of text.Note: Keep in mind that for security purposes, its better to overdetect urls and check more against blacklists than to not detect a url that was submitted. As such, some things that we detect might not be urls but somewhat look like urls. Also, instead of complying with RFC 3986 (http://www.ietf.org/rfc/rfc3986.txt), we try to detect based on browser behavior, optimizing detection for urls that are visitable through the address bar of Chrome, Firefox, Internet Explorer, and Safari.

malsub - A Python RESTful API framework for online malware analysis and threat intelligence services

  •    Python

malsub is a Python 3.6.x framework that wraps several web services of online malware and URL analysis sites through their RESTful Application Programming Interfaces (APIs). It supports submitting files or URLs for analysis, retrieving reports by hash values, domains, IPv4 addresses or URLs, downloading samples and other files, making generic searches and getting API quota values. The framework is designed in a modular way so that new services can be added with ease by following the provided template module and functions to make HTTP GET and POST requests and to pretty print results. This approach avoids having to write individual and specialized wrappers for each and every API by leveraging what they have in common in their calls and responses. The framework is also multi-threaded and dispatches service API functions across a thread pool for each input argument, meaning that it spawns a pool of threads per each file provided for submission or per each hash value provided for report retrieval, for example. Most of these services require API keys that are generated after registering an account in their respective websites, which need to be specified in the apikey.yaml file according to the given structure. Note that some of the already bundled services are limited in supported operations due to the fact that they were developed with free API keys. API keys associated with paid subscriptions are allowed to make additional calls not open to the public and may not be restricted by a given quota. Yet, malsub can process multiple input arguments and pause between requests as a workaround for cooldown periods.

furl - 🌐 URL parsing and manipulation made easy.

  •    Python

Python's standard urllib and urlparse modules provide a number of URL related functions, but using these functions to perform common URL operations proves tedious. Furl makes parsing and manipulating URLs easy. Furl is well tested, Unlicensed in the public domain, and supports Python 2, Python 3, PyPy2, and PyPy3.

Gatsby - Blazing fast static site generator for React

  •    Javascript

Gatsby is a blazing fast static site generator for React. Gatsby.js is a static PWA (Progressive Web App) generator. You get code and data splitting out-of-the-box. Gatsby loads only the critical HTML, CSS, data, and JavaScript so your site loads as fast as possible. Once loaded, Gatsby prefetches resources for other pages so clicking around the site feels incredibly fast.

Kutt - Free Modern URL Shortener

  •    Javascript

Kutt is a modern URL shortener with support for custom domains. Shorten URLs, manage your links and view the click rate statistics.

SharePoint Site Data Reports

  •    

I have created a user control in SharePoint that will show Site Data based on the URL chosen in a drop down list. Currently, it dynamically links to the site collection, so that it can populate the drop down list with the Site Name and Url. So far I have it showing the relat...

YOURLS - 🔗 Your Own URL Shortener

  •    PHP

YOURLS is a set of PHP scripts that will allow you to run Your Own URL Shortener. You'll have full control over your data, detailed stats, analytics, plugins, and more. It's free. To get started, check yourls.org! Learn more tweaks in the Wiki documentation.

check-links - Robustly checks an array of URLs for liveness. Extremely fast ⚡

  •    Javascript

Robustly checks an array of URLs for liveness. For each URL, it first attempts an HTTP HEAD request, and if that fails it will attempt an HTTP GET request, retrying several times by default with exponential falloff.

Synchronizing SharePoint Lists Between Internal and External Sites

  •    

This is a sample program to push SharePoint List data from Site to Site. I had to do it to transform the data from intranet site to the extranet site.

Cactus - Static site generator for designers. Uses Python and Django templates.

  •    Python

We're happy to announce Cactus 3. It brings a set of great new features like asset fingerprinting, an asset pipeline, pretty urls, native Mac filesystem events, automatic nameserver configuration, support for multiple deployment backends (Google Sites) and more. Large parts of the code have been rewritten, accompanied by an extensive suite of unit tests. Many thanks to Thomas Orozco and other contributors. Cactus is a simple but powerful static website generator using Python and the Django template system. Cactus also makes it easy to develop locally and deploy your site to S3 directly. It works great for company, portfolio, personal, support websites and blogs.

OSQA - Stackoverflow like QA system in Python

  •    Python

OSQA is the open source Q&A system. It is more than just an FAQ page, it is a full-featured Q&A community. Users earn points and badges for useful participation, and everyone in the community wins. OSQA is built and maintained by a team of developers who share an interest in making a great, free, open source Q&A system available to everyone. The OSQA project is hosted and financially supported by DZone, Inc.

laravel-url-signer - Create and validate signed URLs with a limited lifetime

  •    PHP

This package can create URLs with a limited lifetime. This is done by adding an expiration date and a signature to the URL. The URL can be validated with the validate-function.






We have large collection of open source products. Follow the tags from Tag Cloud >>


Open source products are scattered around the web. Please provide information about the open source projects you own / you use. Add Projects.