Displaying 1 to 5 from 5 results

terraform-aws-efs-backup - Terraform module designed to easily backup EFS filesystems to S3 using DataPipeline

  •    HCL

Terraform module designed to easily backup EFS filesystems to S3 using DataPipeline. This project is part of our comprehensive "SweetOps" approach towards DevOps.

scala-datapipeline-dsl - Domain-specific language to help build and maintain AWS Data Pipelines

  •    Scala

A Scala domain-specific language and toolkit to help you build and maintain AWS DataPipeline definitions. This tool aims to ease the burden of maintaining a large suite of AWS DataPipelines. At Shazam, we use this tool to define our data pipelines in Scala code and avoid the boilerplate and maintenance headache of managing 10s or 100s of JSON pipeline configuration files.




sparkplug - Spark package to "plug" holes in data using SQL based rules ⚡️ 🔌

  •    Scala

Spark package to "plug" holes in data using SQL based rules. At Indix, we work with a lot of data. Our data pipelines run a wide variety of ML models against our data. There are cases where we have to "plug" or override certain values or predictions in our data. This maybe due to bugs or deficiencies in our current models or just the inherent quality in the source/raw data.