Essential Sql: Azure Data Factory And Data Engineering
Published 11/2023
MP4 | Video: h264, 1920x1080 | Audio: AAC, 44.1 KHz
Language: English | Size: 3.93 GB | Duration: 8h 36m
Data Engineering using ADF | Azure Data Engineer | Data Mapping Flows | SQL Stored Procedure | Data Engineering
What you'll learn
Learn how to construct ELT solutions using Azure Data Factory and SQL Server.
Implement various ELT solutions using Mapping Data Flows, Copy Activities, and Stored Procedures.
Develop the skills to build and troubleshoot Azure Data Factory pipelines with confidence.
Enhance pipeline reliability by mastering the use of parameters to optimize your data workflows.
Learn efficient methods for storing and structuring data using Azure Storage services.
Understand the distinctive roles played by Azure Data Factory, Data Storage, SQL Server, and Data Bricks in the creation of an ELT solution.
Requirements
Familiarity with SQL and basic programming skills can be beneficial.
Proficiency in programming will be valuable when working with scripting and data transformation tasks
Experience in troubleshooting and debugging technical issues will aid in building and troubleshooting Azure Data Factory pipelines.
Description
Use Azure data factory to automate data engineering tasks. This is a great course to introduce yourself to Azure Data Factory's capabilities. We'll look at the data factory from a data engineer perspective.Through examples we'll work side by side to create an Extract Transform and Load process to ingest movie rating data.We are going to explore three different ways to transform your data.The first is by using Azure Data Mapping flows. These are great no-code ways to do ETL.We'll then look at how you can do the same transformations using Python. This way, if you love Python, you know a solid way to use Azure data factory to work with your data.Lastly, we'll look at how you can create pipelines to use your knowledge of SQL and stored procedures.What you'll like about this course is that once you learn one way to transform the data, you can use that knowledge to learn about the other methods. So, if you're a SQL expert but soft on python, you can learn the SQL way first, then go back and try it using Python. In the end you come out learning and appreciating alternative methods to ingesting, transforming, and storing data.
Overview
Section 1: Introduction
Lecture 1 Welcome to the Course
Lecture 2 Who Should Take Azure Data Factory?
Lecture 3 Important! What to Expect from the Course.
Lecture 4 How to Take the Course
Section 2: Set Up Azure
Lecture 5 Setup Azure Account
Lecture 6 Setup Azure Resource Group
Lecture 7 Setup Azure Storage Account
Lecture 8 Create Microsoft Data Factory
Lecture 9 Test Data Factory to Storage Link
Lecture 10 Setup Azure Database
Lecture 11 Set up Azure Data Studio
Lecture 12 Test Azure Database
Lecture 13 Setup Microsoft DevOps
Section 3: Case Study - Using Data Mapping Flows to Refine Data
Lecture 14 Medallion Architecture and Bronze Layer Introduction
Lecture 15 Bronze Layer - Setup Data Lake Folders to support the Medallion Architecture
Lecture 16 Bronze Layer - Prep data using Data Mapping Flows.
Lecture 17 Bronze Layer - Execute Mapping Data Flow using ADF Pipeline.
Lecture 18 Silver Layer - Introduction
Lecture 19 Silver Layer - Building Mapping Flow Part 1 - Data Types
Lecture 20 Silver Layer - Building Mapping Flow Part 2 - Filter and Deduplication
Lecture 21 Silver Layer - Building Mapping Flow Part 3 - Parsing Addresses
Lecture 22 Silver Layer - Building Mapping Flow Part 4 - Touchups and Final Check
Lecture 23 Gold Layer - Introduction
Lecture 24 Gold Layer - Build Dimensions and Fact table using Mapping Flow
Lecture 25 Gold Layer - Automate Dimensions and Fact table Builds using ADF
Lecture 26 Gold Layer - Refactor ADF Pipeline and then See Results in PowerBI
Section 4: Case Study - SQL and the Azure Data Factory Copy Activity to Refine Data
Lecture 27 Medallion Architecture and Bronze Layer Introduction
Lecture 28 Bronze Layer - Using ADF Copy Activity
Lecture 29 Silver Layer - Introduction
Lecture 30 Silver Layer - Land Data into SQL
Lecture 31 Silver Layer - Create Address Split Query
Lecture 32 Silver Layer - Create Stored Procedure Transform Loaded Data
Lecture 33 Silver Layer - Execute Stored Proc from Data Factory Pipeline
Lecture 34 Gold Layer - Introduction
Lecture 35 Gold Layer - Dimension Building
Lecture 36 Gold Layer - Build the Sales Order Fact
Lecture 37 Gold Layer - Move Table to Gold (Production)
Lecture 38 Gold Layer - Review Start Schema in PowerBI
Section 5: ADF Pipeline and Activities Reference
Lecture 39 Azure Data Factory - Introduction
Lecture 40 Azure Data Factory - Variables and Parameters
Lecture 41 Azure Data Factory - Parameters and Return Values
Lecture 42 Azure Data Factory - If Activity
Lecture 43 Azure Data Factory - Switch Activity
Lecture 44 Azure Data Factory - For Each Activity
Lecture 45 Azure Data Factory - Get Meta Data and IF
Lecture 46 Azure Data Factory - Filter Activity
Section 6: Resources
Lecture 47 Product Database
