Displaying 1 to 11 from 11 results

toapi - Every web site provides APIs.

  •    Python

Toapi give you the ability to make every web site provides APIs. Version v2.0.0, Completely rewrote.

ghcrawler - Crawl GitHub APIs and store the discovered orgs, repos, commits, ...

  •    Javascript

A robust GitHub API crawler that walks a queue of GitHub entities retrieving and storing their contents.

utsusemi - A tool to generate a static website by crawling the original site.

  •    Javascript

A tool to generate a static website by crawling the original site. Set environment variables.




wishlist - Read an Amazon wishlist programmatically with Python

  •    HTML

Python library to grab your Amazon wishlist so you can mess with it programmatically. Wishlist depends on Brow in order to login from the command line (including 2 factor if enabled), and Brow depends on Selenium and Firefox to be installed, so you'll need to read Brow's README if you need help installing those on Linux.

node-google-play - Get details and download apps from https://play

  •    Javascript

Call Google Play APIs from Node. You might want to check out the CLI package as well. By default behaves like a Nexus device with SDK 23 for app downloads.

ghcrawler-cli - A simple command line app for controlling a GitHub crawler

  •    Javascript

GHCrawler is utility for walking GitHub APIs and tracking GitHub events. This command line app allows you to control various aspects of a crawler's behavior. There is some overlap in function with the Crawler Dashboard. This project also has a simple Node client library for talking to a crawler. -s <url> -- Control the crawler service running at the given url. Defaults to http://localhost:3000. You can also set the CRAWLER_SERVICE_URL environment variable.


dcard-spider - A spider on Dcard. Strong and speedy.

  •    Python

快如閃電的爬蟲,迅速獲得 Dcard 上所有看板的文章! Spider needs for speed. This will be a library project for dcard continously crawling spider. And also provides end-user friendly features.

HLTV - The unofficial HLTV Node.js API

  •    TypeScript

You can create an instance of HLTV with a custom config.

robots

  •    Java

Distributed robots.txt parser and rule checker through API access. If you are working on a distributed web crawler and you want to be polite in your action, then you will find this project very useful. Also, this project can be used to integrate into any SEO tool to check if the content is being indexed correctly by robots.