Databricks on Azure

About this Course

This 10-day course offers a comprehensive, hands-on introduction to Azure's data platform and key services, with a focus on Azure Databricks, Delta Lake, and data pipeline creation. Participants will gain a deep understanding of Azure's core services, including storage, data management, and redundancy, followed by a dive into Azure SQL, NoSQL databases like Cosmos DB, and advanced analytics services such as Azure Synapse Analytics and Azure Data Factory. The course covers the essentials of Spark and Databricks, from architecture to execution plans, alongside advanced topics like Lakehouse and Medallion architecture, and the benefits of Delta Lake for data management and transformation. Labs throughout the course offer practical experience in accessing data from Azure Data Lake, building data pipelines, and working with Databricks workflows and Unity Catalog. By the end of the course, participants will be proficient in leveraging Azure's powerful data services for real-time analytics, ETL processes, and building scalable, reliable data solutions.

Key Concepts of Azure Databricks
Spark Overview
Deep Dive into Spark Architecture:
Understanding Databricks Architecture:
Workspace, Creating Notebooks, Cluster, Job Cluster– Lab
Reading data from ADLS using Azure AD service principal from Databricks Spark cluster – Lab
Azure Key Vault: Secret management, Secret scope – Lab
Deep Dive into Spark Execution Plan– Lab

Data Lakehouse: Understanding the whitepaper, Data Lake, Delta Lake
Delta Lake components: _delta logs (transaction logs), Checkpoint, Parquet, Time Travel, Delta Engine (Photon) – Lab
Lakehouse Medallion Architecture: Bronze, Silver, and Gold Layers
Implementing Medallion Architecture using Delta Lake– Lab
Read/Write – Lab:

Delta Lake Fundamentals: Introduction, architecture, and benefits of Delta Lake
Ingestion Techniques: Using CREATE TABLE AS SELECT (CTAS), COPY INTO, and Autoloader for streaming data
Medallion Architecture: Bronze, Silver, and Gold layers for data transformation
Data Management in Delta Lake: Delta Lake DDL, versioning, schema enforcement, time travel, and compaction
Data Quality and Reliability: Applying constraints, managing schema evolution, and data cleaning

Delta Live Tables Overview: Introduction to DLT, role in ETL pipelines, and UI walkthrough
Creating and Scheduling DLT Pipelines: Incremental processing and ingestion with Auto Loader
Change Data Capture (CDC) with APPLY CHANGES INTO syntax
Pipeline Monitoring and Troubleshooting: Reviewing event logs, data artifacts, and addressing syntax errors