Laravel-Crawler-Detect - A Laravel wrapper for CrawlerDetect - the web crawler detection library

  •        27

Run composer require jaybizzle/laravel-crawler-detect 1.* or add "jaybizzle/laravel-crawler-detect": "1.*" to your composer.json file. The last version compatible with Laravel 4 was v1.0.2 so if you need that, you will have to fix your composer.json to that specific version.

https://github.com/JayBizzle/Laravel-Crawler-Detect

Tags
Implementation
License
Platform

   




Related Projects

Crawler-Detect - 🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

  •    PHP

CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent and http_from header. Currently able to detect 1,000's of bots/spiders/crawlers. Run composer require jaybizzle/crawler-detect 1.* or add "jaybizzle/crawler-detect" :"1.*" to your composer.json.

agent - 👮 A PHP desktop/mobile user agent parser with support for Laravel, based on Mobiledetect

  •    PHP

A PHP desktop/mobile user agent parser with support for Laravel, based on Mobile Detect with desktop support and additional functionality. Check for a certain property in the user agent.

laravel-image-optimizer - Optimize images in your Laravel app

  •    PHP

This package is the Laravel 5.4 and up specific integration of spatie/image-optimizer. It can optimize PNGs, JPGs, SVGs and GIFs by running them through a chain of various image optimization tools. The package will automatically detect which optimization binaries are installed on your system and use them. The package also contains a middleware to automatically optimize all images in an request.

Norconex HTTP Collector - A Web Crawler in Java

  •    Java

Norconex HTTP Collector is a web spider, or crawler that aims to make Enterprise Search integrators and developers's life easier. It is Portable, Extensible, reusable, Robots.txt support, Obtain and manipulate document metadata, Resumable upon failure and lot more.


telegram-bot-sdk - 🤖 Telegram Bot API PHP SDK

  •    PHP

The (Unofficial) Telegram Bot API PHP SDK. Lets you develop Telegram Bots easily! Supports Laravel out of the box. Telegram Bot API is an HTTP-based interface created for developers keen on building bots for Telegram.

node-crawler - Web Crawler/Spider for NodeJS + server-side jQuery ;-)

  •    Javascript

Web Crawler/Spider for NodeJS + server-side jQuery ;-)

Norconex HTTP Collector - Enterprise Web Crawler

  •    Java

Norconex HTTP Collector is a full-featured web crawler (or spider) that can manipulate and store collected data into a repositoriy of your choice (e.g. a search engine). It very flexible, powerful, easy to extend, and portable.

go_spider - [爬虫框架 (golang)] An awesome Go concurrent Crawler(spider) framework

  •    Go

A crawler of vertical communities achieved by GOLANG. Latest stable Release: Version 1.2 (Sep 23, 2014).

colly - Elegant Scraper and Crawler Framework for Golang

  •    Go

Colly provides a clean interface to write any kind of crawler/scraper/spider. With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.

crawler - A high performance web crawler in Elixir.

  •    Elixir

A high performance web crawler in Elixir, with worker pooling and rate limiting via OPQ. Below is a very high level architecture diagram demonstrating how Crawler works.

RED_HAWK - All in one tool for Information Gathering, Vulnerability Scanning and Crawling

  •    PHP

RED HAWK's CMS Detector currently is able to detect the following CMSs (Content Management Systems) in case the website is using some other CMS, Detector will return could not detect. Want to contribute to RED HAWK or point out something wrong? Just create a new issue here: https://github.com/Tuhinshubhra/RED_HAWK/issues/new I'd love to hear from you.

firewall - Firewall package for Laravel applications

  •    PHP

This a "soft-firewall" package. Its purpose is to help people prevent unauthorized access to routes by IP address. It is able to keep track of IPs, countries and hosts (dynamic ip), and redirect non-authorized users to, for instance, a "Coming Soon" page, while letting whitelisted IPs to have access to the entire site. It is now also able to detect and block attacks (too many requests) from single IPs or whole countries. This package can prevent some headaches and help you block some access to your apps, but cannot replace firewalls and appliances, for attacks at the network level, you'll still need a real firewall.

Squzer - Distributed Web Crawler

  •    Python

Squzer is the Declum's open-source, extensible, scale, multithreaded and quality web crawler project entirely written in the Python language.

grab-site - The archivist's web crawler: WARC output, dashboard for all crawls, dynamic ignore patterns

  •    Python

grab-site is an easy preconfigured web crawler designed for backing up websites. Give grab-site a URL and it will recursively crawl the site and write WARC files. Internally, grab-site uses wpull for crawling. a dashboard with all of your crawls, showing which URLs are being grabbed, how many URLs are left in the queue, and more.

NWebCrawler

  •    

This is a web crawler program written in C#.

Open Search Server

  •    C++

Open Search Server is both a modern crawler and search engine and a suite of high-powered full text search algorithms. Built using the best open source technologies like lucene, zkoss, tomcat, poi, tagsoup. Open Search Server is a stable, high-performance piece of software.

Pavuk

  •    C

Pavuk is a UNIX program used to mirror the contents of WWW documents or files. It transfers documents from HTTP, FTP, Gopher and optionally from HTTPS (HTTP over SSL) servers. Pavuk has an optional GUI based on the GTK2 widget set.

colly - Fast and Elegant Scraping Framework for Gophers

  •    Go

Colly provides a clean interface to write any kind of crawler/scraper/spider.With Colly you can easily extract structured data from websites, which can be used for a wide range of applications, like data mining, data processing or archiving.