StreamAlert is a serverless, realtime data analysis framework which empowers you to ingest, analyze, and alert on data from any environment, using datasources and alerting logic you define.
security kinesis serverless terraform lambda aws rules analysis dataA curated list of awesome AWS libraries, open source repos, guides, blogs, and other resources.Inspired by the awesome list.
aws cloud aws-sdk aws-cli cloudformation cloudwatch dynamodb elasticsearch kinesis lambda serverless machine-learning redshift s3 ec2 iam rds route53 ecs cloud-managementAn implementation of Amazon's Kinesis, focussed on correctness and performance, and built on LevelDB (well, @rvagg's awesome LevelUP to be precise).The Kinesis equivalent of dynalite.
kinesis mock kinesilite kinesis-mock test event stream streaming processing aws logs logging amazon-kinesisA small utility to sign vanilla node.js http(s) request options using Amazon's AWS Signature Version 4.Can also be used in the browser.
amazon aws signature s3 ec2 autoscaling cloudformation elasticloadbalancing elb elasticbeanstalk cloudsearch dynamodb kinesis lambda glacier sqs sns iam sts ses swf storagegateway datapipeline directconnect redshift opsworks rds monitoring cloudtrail cloudfront codedeploy elasticache elasticmapreduce elastictranscoder emr cloudwatch mobileanalytics cognitoidentity cognitosync cognito containerservice ecs appstream keymanagementservice kms config cloudhsm route53 route53domains logsInstrumentation for logs, metrics, and exceptions
metrics instrumentation exception logging sentry datadog kinesisExample project for a personal serverless Google Analytics clone to track website visitors.After deploying the service you will have an HTTP endpoint using Amazon API Gateway that accepts requests and puts them into a Kinesis Stream. A Lambda function processes the stream and writes basic metrics about how many visitors you have per absolute URL to DynamoDB.
serverless sls aws analytics kinesis lambda dynamodb exampleKine makes reading from an aws kinesis stream easy.See API.md for complete reference.
kinesis awsA KPL-like batch producer for Amazon Kinesis built on top of the official Go AWS SDK and using the same aggregation format that KPL use.
kpl amazon-kinesis kpl-aggregation kinesis producer kinesis-producerRegistry is a versioned entity framework that allows to build various registry services such as Schema Registry, ML Model Registry etc..
schema-registry kafka kinesis flink spark-streaming metadata schemas stormAmazon Kinesis Firehose simplifies delivery of streaming data to Amazon S3 and Amazon Redshift with a simple, automatically scaled, and zero operations requirement. Where customers have existing systems built on streaming interfaces, the addition of Firehose can enable simple archive, or be used to facilitate long term analysis of data from Amazon Redshift. Integration can be accomplished by using the Kinesis Agent to automatically publish file data to Amazon Kinesis Streams and/or Amazon Kinesis Firehose delivery streams. This project includes an AWS Lambda function that enables customers who are already using Amazon Kinesis Streams for real time processing to take advantage of Amazon Kinesis Firehose. Furthermore, if you are using Amazon DynamoDB and would like to store a history of changes made to the table, this function can push events to Amazon Kinesis Firehose.In order to effectively use this function, you should already have configured an Amazon Kinesis Stream or an Amazon DynamoDB Table with Update Streams, as well as an Amazon Kinesis Firehose Delivery Stream of the correct name. For Amazon Kinesis Streams, please ensure that producer applications can write to the Stream, and that the Amazon Kinesis Firehose Delivery Stream is able to deliver data to Amazon S3 or Amazon Redshift. This function makes no changes to Stream or Firehose configurations. You must also have the AWS Command Line Interface (https://aws.amazon.com/cli) installed to take advantage of the Stream Tagging utility supplied.
amazon aws kinesis firehose lambdaThis function answers a need I have had multiple times, where I want to replicate data from an Amazon Kinesis Stream to another account or another region for processing, or to another environment such as development.This AWS Lambda function can be used to propagate incoming messages from Amazon Kinesis Streams or Amazon DynamoDB Streams to other services (Amazon SNS, Amazon SQS, Amazon Elasticsearch Service, Amazon Kinesis Streams, Amazon Kinesis Firehose, AWS IoT, AWS Lambda, Amazon ElastiCache for Memcached and Redis), regions or accounts. This function generates metrics that will be published to Amazon CloudWatch Metrics, in the Custom/FanOut namespace.
amazon kinesis dynamodb sqs sns elasticache elasticsearch iot memcached redis firehose lambdaThis package provides an interface to the Amazon Kinesis Client Library (KCL) MultiLangDaemon for the Node.js framework.Developers can use the KCL to build distributed applications that process streaming data reliably at scale. The KCL takes care of many of the complex tasks associated with distributed computing, such as load-balancing across multiple instances, responding to instance failures, checkpointing processed records, and reacting to changes in stream volume.
api amazon aws big-data kinesis kinesis-client-library kcl node.jsNode.js stream interface for reading records from AWS Kinesis.
kinesis streamBased on the AWS Kinesis Client Library for Java, reimplemented in Node.js. Install with npm install kinesis-client-library --save.
aws kinesisAWS Lambda function for insert Kinesis event to Google BigQuery example. Wait role is created.
kinesis bigqueryKinesis consumer applications written in Go. This library is intended to be a lightweight wrapper around the Kinesis API to read records, save checkpoints (with swappable backends), and gracefully recover from service timeouts/errors. Kinesis to Firehose can be used to archive data directly to S3, Redshift, or Elasticsearch without running a consumer application.
kinesis-consumer kinesis stream golang-kinesis-connectorData-migrator (version 0.6.3.dev2) is a simple data-migration package for python lovers. Data-migrator is a declarative DSL for table driven data transformations, set up as an open and extensive system. Use this to create data transformations for changing databases as a result of changing code, initial loads to datalakes (it contains a Kinesis provider)and more.
data-structures migration python-library development-tools development-utils database gdpr anonymization data-policy kinesisAmazon Kinesis Analytics can fan-out your Kinesis Streams and avoid read throttling. Each Kinesis Streams shard can support a maximum total data read rate of 2 MBps (max 5 transactions), and a maximum total data write rate of 1 MBps (max 1,000 records). Even if you provision enough write capacity, you are not free to connect as many consumers as you'd like, especially with AWS Lambda, because you'll easily reach the read capacity.
aws aws-lambda kinesis-stream kinesis kinesis-analytics serverless serverless-framework amazon-kinesis kinesis-streams serverless-pluginType definitions and helpers for AWS Lambda event sources. AWS Lambda lets you run code without provisioning or managing servers. With eawsy/aws-lambda-go-shim, you can author your Lambda function code in Go. This project provides type definitions and helpers to deal with AWS Lambda event source mapping.
aws aws-lambda lambda event-sources kinesis dynamodb s3 sns ses cognito cloudwatch apigateway serverless firehose events
We have large collection of open source products. Follow the tags from
Tag Cloud >>
Open source products are scattered around the web. Please provide information
about the open source projects you own / you use.
Add Projects.