fluent-plugin-cloudwatch-ingest - Alternative to ryotarai/fluent-plugin-cloudwatch-logs for ingesting AWS Cloudwatch logs via fluentd

  •        7

I am no longer actively using this plugin and am looking for maintainers. In my experience Cloudwatch Log Subscriptions are vastly superior to external API consumers, which are subject to limiting and state synchronization issues. The plugin will assume an IAM instance role. Without either of the sts_* options that role will be used for authentication. With those set the plugin will attempt to sts:AssumeRole the sts_arn. This is useful for fetching logs from many accounts where the fluentd infrastructure lives in one single account.

https://github.com/sampointer/fluent-plugin-cloudwatch-ingest

Tags
Implementation
License
Platform

   




Related Projects

watchtower - Python CloudWatch Logging: Log Analytics and Application Intelligence

  •    Python

Watchtower is a log handler for Amazon Web Services CloudWatch Logs. CloudWatch Logs is a log management service built into AWS. It is conceptually similar to services like Splunk and Loggly, but is more lightweight, cheaper, and tightly integrated with the rest of AWS.

cloudwatch-logs-subscription-consumer - A specialized Amazon Kinesis stream reader (based on the Amazon Kinesis Connector Library) that can help you deliver data from Amazon CloudWatch Logs to any other system in near real-time using a CloudWatch Logs Subscription Filter

  •    Java

The CloudWatch Logs Subscription Consumer helps Java developers consume a real-time feed of CloudWatch Logs data for custom processing, analysis, or loading to other systems.

awslogs - AWS CloudWatch logs for Humans™

  •    Python

awslogs is a simple command line tool for querying groups, streams and events from Amazon CloudWatch logs. Running: awslogs get /var/logs/syslog ALL -s1d will return you events from any stream in the /var/logs/syslog group generated in the last day.


Convox Rack - Open-source PaaS, Built entirely on AWS cloud services for maximum privacy and minimum upkeep

  •    Go

Convox Rack is open source PaaS built on top of expert infrastructure automation and devops best practices. Rack gives you a simple developer-focused API that lets you build, deploy, scale and manage apps on private infrastructure with ease.

node-monitor - Amazon EC2 monitoring with CloudWatch and Node.js

  •    Javascript

Amazon EC2 monitoring with CloudWatch and Node.js

AWS Monitor

  •    ASPNET

A web app utilizing the Amazon Web Services API with a focus on browsing through and analyzing data graphically, mostly from CloudWatch - the API providing metrics about the usage of all the Amazon Web Services such as EC2 (Cloud Computing) and ELB (Elastic Load Balancing).

kappa - What precedes Lambda

  •    Python

Kappa is a command line tool that (hopefully) makes it easier to deploy, update, and test functions for AWS Lambda. Kappa tries to help you with some of this. It creates all IAM policies for you based on the resources you have told it you need to access. It creates the IAM execution role for you and associates the policy with it. Kappa will zip up the function and any dependencies and upload them to AWS Lambda. It also sends test data to the uploaded function and finds the related CloudWatch log stream and displays the log events. Finally, it will add the event source to turn your function on.

ecs-refarch-service-discovery - An EC2 Container Service Reference Architecture for providing Service Discovery to containers using CloudWatch Events, Lambda and Route 53 private hosted zones

  •    Go

Service discovery is a key component of most distributed systems and service-oriented architectures. With service discovery, services are automatically discovered as they get created and terminated on a given infrastructure. This reference architecture illustrates how service discovery can be built on AWS.Many AWS customers build service-oriented, distributed applications using services such as Amazon EC2 Container Service (Amazon ECS) or Amazon EC2. The distributed nature of this type of architecture requires a fair amount of integration and synchronization, and the answer to that problem is not trivial. Quite often, our customers build such a functionality themselves and this can be time-consuming. Or they use a third-party solution and this often comes with a financial cost.

fluent-logger-python - A structured logger for Fluentd (Python)

  •    Python

Many web/mobile applications generate huge amount of event logs (c,f. login, logout, purchase, follow, etc). To analyze these event logs could be really valuable for improving the service. However, the challenge is collecting these logs easily and reliably. Fluentd solves that problem by having: easy installation, small footprint, plugins, reliable buffering, log forwarding, etc.

sof-elk - Configuration files for the SOF-ELK VM, used in SANS FOR572

  •    Python

This repository contains the configuration and support files for the SANS FOR572 SOF-ELK® VM Appliance. SOF-ELK® is a “big data analytics” platform focused on the typical needs of computer forensic investigators/analysts and information security operations personnel. The platform is a customized build of the open source ELK stack, consisting of the Elasticsearch storage and search engine, Logstash ingest and enrichment system, and the Kibana dashboard frontend. With a significant amount of customization and ongoing development, SOF-ELK® users can avoid the typically long and involved setup process the ELK stack requires. Instead, they can simply download the pre-built and ready-to-use SOF-ELK® virtual appliance that consumes various source data types (numerous log types as well as NetFlow), parsing out the most critical data and visualizing it on several stock dashboards. Advanced users can build visualizations the suit their own investigative or operational requirements, optionally contributing those back to the primary code repository.

fluent-bit - Fast and Lightweight Log processor and forwarder for Linux, BSD and OSX

  •    C

Fluent Bit is a Data Forwarder for Linux, Embedded Linux, OSX and BSD family operating systems. It's part of the Fluentd Ecosystem. Fluent Bit allows collection of information from different sources, buffering and dispatching them to different outputs such as Fluentd, Elasticsearch, Nats or any HTTP end-point within others. It's fully supported on x86_64, x86 and ARM architectures. In order to contribute to the project please refer to the CONTRIBUTING guidelines.

streamalert - StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define

  •    Python

StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.

Fluentd - Data collector, Log Everything in JSON

  •    Ruby

Fluentd is an event collector system. It is a generalized version of syslogd, which handles JSON objects for its log messages. It collects logs from various data sources and writes them to files, database or other types of storages.

aws-mobile-react-native-starter - AWS Mobile React Native Starter App https://aws

  •    Javascript

Bootstrap a React Native application on AWS. This starter automatically provisions a Serverless infrastructure with authentication, authorization, image storage, API access and database operations. It also includes user registration and MFA support. The sample use case is a "Pet Tracker" where after a user registers and logs in they can upload pictures of their pet to the system along with information like the birthday or breed.A companion blog post for this repository can be found in the AWS Mobile Blog: Announcing: React Native Starter Project with One-Click AWS Deployment and Serverless Infrastructure.

log-pilot - Collect logs for docker containers

  •    Go

log-pilot is an awesome docker log tool. With log-pilot you can collect logs from docker hosts and send them to your centralized log system such as elasticsearch, graylog2, awsog and etc. log-pilot can collect not only docker stdout but also log file that inside docker containers. Then access kibana under the tips. You will find that tomcat's has been collected and sended to kibana.