Displaying 1 to 2 from 2 results

aws-dbs-refarch-datalake - Reference Architectures for Datalakes on AWS

  •    HTML

A datalake is a data repository that stores data in its raw format until it is used for analytics. It is designed to store massive amount of data at scale. A schema to the dataset in data lake is given as part of transformation while reading it. Below is a pictorial representation of a typical datalake on AWS cloud. Keeping track of all of the raw assets that are loaded into your datalake, and then tracking all of the new data assets and versions that are created by data transformation, data processing, and analytics can be a major challenge. An essential component of an Amazon S3 based data lake is a Data Catalog. A data catalog is designed to provide a single source of truth about the contents of the data lake, and rather than end users reasoning about storage buckets and prefixes, a data catalog lets them interact with more familiar structures of databases, tables, and partitions.

data-federation-project - A project focused on tools and best practices to supported federated data collection efforts

  •    

Federated data refers to data that are aggregated across a number of organizations, departments or agencies at various scales. Managing the volume, quality and completeness of data coming in from multiple sources is a common challenge for government agencies charged with collecting and maintaining domain-specific information. This project aims to address this challenge by providing ways to ingest data more easily, and provide real-time user feedback so that data contributors can make corrections before it is submitted to a central repository. Federated data efforts are increasingly seen as an engine for transparency, economic growth, and accountability, yet collecting this kind of data remains a challenge. Despite the fact that efforts of this sort are increasing in frequency, each new effort is still improvising solutions in terms of processes, tooling, and compliance infrastructure.