DATA LAKE
Data Lakes offer the scalability, speed, and cost effectiveness to help you manage large volumes and multiple types of data across your various analytic initiatives – AI, machine learning, streaming analytics, BI, and more.
- AWS Cloud
- Azure Cloud
- Google Cloud
- Cloudera
- Databricks
- Snowflake
- Delta Lake
- Apache Iceberg
AWS Data Lake
Amazon Web Service (AWS) offers multiple services for building secure, flexible and cost-effective data lakes.
Core services provided by AWS based lakes are
- Amazon Simple Storage (S3)
- Amazon Elastic MapReduce (EMR)
Azure Data Lake
Azure Data Lake provides scalable storage, processing and analytics across several platforms and programming languages.
key elements
- Azure Data Lake Storage (ADLS)
- Azure HDInsight
- Azure Data Lake Analytics
Cloudera Data Platform (CDP)
With the public, private, hybrid, and multi-cloud Cloudera Data Platform (CDP), you can manage your infrastructure, data, and analytical workloads in whichever environment your company employs.
Key elements
- Data Hub
- Shared Data Experience (SDX)
- Self-service analytics services for data warehouse and machine
learning use cases
Google Data Lake
To help you in safely ingesting, storing, and analysing enormous volumes of varied data, Google Cloud Platform (GCP) provides its own data lake. The GCP services are well integrated.
Key elements
- Google Cloud Storage (GCS)
- Google Dataproc
Databricks Unified
This was initially focused on modernising data lakes, now portrays itself as a data lakehouse, an open, unified platform created to store and manage all of your data for all of your business's analytical requirements.
Key elements
- Delta Lake
- Delta Engine
Snowflake Cloud Data Platform
The distinction between a data lake and a data warehouse has become increasingly hazy thanks to Snowflake, better known as a cloud data warehouse. based on an adaptable platform, Snowflake offers the security, governance, and performance of a warehouse coupled with the scalability, elasticity, and inexpensive storage of a lake.
Key elements
- To load a diverse array of data in its native format, without having to transform it.
- To share data with partner tools like Apache
Spark, using ODBC and JDBC connectors for real-time large-scale data processing.
Delta Lake
Delta Lake is an open-source storage framework that allows you to design a Lakehouse architecture using compute engines such as Spark, PrestoDB, Flink, Trino, and Hive, as well as APIs for Scala, Java, Rust, Ruby, and Python.
Key elements
- ACID Transactions: ACID (Atomicity, Consistency, Isolation, Durability) transactions
- Schema Enforcement and Evolution
- Data Compaction and Optimization
Apache Iceberg
Apache Iceberg is a data table format for large-scale data processing that is open source. It is intended to offer the advantages of both classic data warehouses and new data lakes.
Key elements
- Table Metadata Management
- Integration with Ecosystem Tools
- Schema Evolution
