Displaying 1 to 12 from 12 results

crawler - An easy to use, powerful crawler implemented in PHP. Can execute Javascript.

  •    PHP

This package provides a class to crawl links on a website. Under the hood Guzzle promises are used to crawl multiple urls concurrently. Because the crawler can execute JavaScript, it can crawl JavaScript rendered sites. Under the hood Chrome and Puppeteer are used to power this feature.

RED_HAWK - All in one tool for Information Gathering, Vulnerability Scanning and Crawling

  •    PHP

RED HAWK's CMS Detector currently is able to detect the following CMSs (Content Management Systems) in case the website is using some other CMS, Detector will return could not detect. Want to contribute to RED HAWK or point out something wrong? Just create a new issue here: https://github.com/Tuhinshubhra/RED_HAWK/issues/new I'd love to hear from you.

Yioop - Open Source Search Engine Software

  •    PHP

Yioop is an open source, PHP search engine capable of crawling, index, and providing search results for hundred of millions of pages on relatively low end hardware. It can index a variety of text formats HTML, RSS, PDF, RTF, DOC and images GIF, JPEG, PNG, etc. It can import data from ARC, WARC, Media-Wiki, Open Directory RDF. It is easily localized to many languages. It has built-in support for new feeds, discussion groups, blogs, and wikis. It also supports mixing indexes to create mash ups.




Crawler-Detect - 🕷 CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent

  •    PHP

CrawlerDetect is a PHP class for detecting bots/crawlers/spiders via the user agent and http_from header. Currently able to detect 1,000's of bots/spiders/crawlers. Run composer require jaybizzle/crawler-detect 1.* or add "jaybizzle/crawler-detect" :"1.*" to your composer.json.

Laravel-Crawler-Detect - A Laravel wrapper for CrawlerDetect - the web crawler detection library

  •    PHP

Run composer require jaybizzle/laravel-crawler-detect 1.* or add "jaybizzle/laravel-crawler-detect": "1.*" to your composer.json file. The last version compatible with Laravel 4 was v1.0.2 so if you need that, you will have to fix your composer.json to that specific version.

Marsvin - Structural Crawler framework written in PHP

  •    PHP

If yes, you must know that is always a trivial task, but we have always to think how structure our code to do such a thing... So... to solve that Marvins was created, Marvins provide a simple API and structure to be followed to you create your parsers or crawler. The main focus is to facilitate the task of parser data from external resources, to extract data from websites or import data from XML, CSV files etc...


robots-txt - Determine if a page may be crawled from robots.txt, robots meta tags and robot headers

  •    PHP

Determine if a page may be crawled from robots.txt, robots meta tags and robot headers. Please see CHANGELOG for more information what has changed recently.

crawler - Libraries and scripts for crawling the TYPO3 page tree

  •    PHP

Libraries and scripts for crawling the TYPO3 page tree. Used for re-caching, re-indexing, publishing applications etc. Please see the Wiki Pages for Release notes and Known issues.

php-google - Google search results crawler, get google search results that you need - php

  •    PHP

This is an easy Google Searching crawler that you can get anything you want in the page by using it. If you have installed php-google in your project, you can get google search results that you need.