Data Build Tool (DBT) – Build Your First Project

从头开始学习如何使用 DBT 测试、部署、记录和可视化您的数据转换过程

发表于 04/2022
MP4 | 视频：h264, 1280×720 | 音频：AAC，44.1 KHz，2 Ch
类型：eLearning | 语言：英语 + srt | 时长：22 节课（2h 7m）| 大小：937.1 MB

您将学到什么
了解如何使用 DBT 命令行工具
测试数据质量、集成和代码性能。
手动运行脚本，然后运行自动化测试并在通过所述测试后部署更改。
使用 Dbt 内置包管理器发布公共和私有存储库。
在 git 中合并更新的代码后部署一个 dbt 项目。
自动创建数据如何在整个组织中流动的可视化表示。
通过架构文件轻松创建文档。

要求
处理数据的知识

描述
在将数据加载到集中式数据仓库之前，必须对其进行清理、保持一致并根据需要进行组合。换句话说，必须转换数据——ETL（提取、转换、加载）和 ELT 中的“T”。这允许组织通过分析和报告开发有价值的、值得信赖的见解。

DBT 使数据分析师和数据工程师能够自动化数据转换过程的测试和部署。这特别有用，因为许多公司的报告数据背后的业务逻辑越来越复杂。DBT 工具会记录对底层逻辑所做的所有更改，并通过版本控制轻松跟踪数据并更新或修复管道。

在本课程中，您将学习

安装 DBT，初始化一个新项目，然后将您的项目发布到 GitHub 存储库。

了解如何将源添加到您的 DBT 项目并使用 Jijna 模板将它们添加到您的模型脚本中。

了解 DBT 如何将 SQL 和 Jijna 函数的组合转换为可用于调试模型逻辑的纯 SQL 脚本。

更改默认 DBT 设置，以便您可以根据 DBTproject 文件夹部署到自定义模型。

测试如何运行以及如何使用它们来调试任何错误以确保数据质量的基础。

物化是在仓库中持久化 DBT 模型的策略。了解如何在您的项目中调整此设置。

包管理和 DBT Hub。

使用内置的 DBT docs 命令将您的文件作为网站提供。您可以在本地查看该站点，也可以将其托管在单独的 Web 服务器上以供其他人查看。

使用 DBT Seeds 处理静态 CSV 数据

了解创建自定义模式测试、覆盖默认测试以及最后在哪里可以找到一些预打包模式测试的过程，这些测试可以节省您编写自己的测试的时间。

学习使用 DBT 允许您定义变量的两种方式——在 dbt_project.yml 文件中和在命令行中。

使用挂钩来简化重复的 dbt 运行活动

使用 docs 块简化 dbt 文档，因为有时您的文档会变得有点复杂，以至于最好在 YML 文件之外编写它。或者您可能会发现您正在多次重写相同的定义。

了解新鲜度块的语法以及它如何在后台工作以定义最近记录与现在之间的可接受时间量，以便将表视为“新鲜”。

了解如何将查询标签添加到 DBT 项目，然后在 Snowflake 中查看结果。

了解 manifest.json 工件并逐步查看如何使用 state 方法仅运行修改后的模型。

多回购方法——如何构建 DBT 项目

使用 {{ invocation_id }} 审核您的 DBT 运行

在windows上将DBT安装到虚拟环境的过程，看看如何直接从DBT core（GitHub）而不是使用pip安装DBT。

清理 DBT 项目文件，分组运行/测试模型，并使用 EPHEMERAL DBT 模型

本课程中使用的所有项目文件都提供给学生在学习过程中使用。

本课程适用于
数据分析师、
数据工程师、
数据库管理员、
学生

Learn from scratch how to use DBT to test, deploy, document and visualize your data transformation process

Published 04/2022
MP4 | Video: h264, 1280×720 | Audio: AAC, 44.1 KHz, 2 Ch
Genre: eLearning | Language: English + srt | Duration: 22 lectures (2h 7m) | Size: 937.1 MB

What you’ll learn
Learn how to use the DBT command line tool
Test data quality, integration, and code performance.
Manually run scripts that will then run automated tests and deploy changes after passing said tests.
Publish both public and private repositories using Dbt built-in package manager.
Deploy a dbt project after merging updated code in git.
Automatically create a visual representation of how data flows throughout your organization.
Easily create documentation through schema files.

Requirements
Knowledge of working with data

Description
Before loading data into a centralized data warehouse, it must be cleaned up, made consistent, and combined as necessary. In other words, data must be transformed – the “T” in ETL (extract, transform, load) and ELT. This allows an organization to develop valuable, trust-worthy insights through analytics and reporting.

DBT enables data analysts and data engineers to automate the testing and deployment of the data transformation process. This is especially useful because many companies have increasingly complex business logic behind their reporting data. The DBT tool keeps a record of all changes made to the underlying logic and makes it easy to trace data and update or fix the pipeline through version control.

In this course you will learn

Install DBT, initialize a new project and then publish your project to a GitHub repository.

Learn how to add sources to your DBT project and add them to your model scripts using the Jijna templates.

Learn how DBT translates the combination of SQL and Jijna functions into a pure SQL script that you can use to debug your model logic.

Allter the default DBT settings so that you can deploy to custom models based on your DBTproject folders.

Foundation for how tests are run and how you can use them to debug any errors to ensure data quality.

Materializations are strategies for persisting DBT models in a warehouse. Learn how to adjust this setting in your project.

Package management and DBT Hub.

Using the built-in DBT docs command to serve your files up as a website. You can either view the site locally or host it on a separate web server for others to see.

Use DBT Seeds to work with static CSV data

Learn the process of creating a custom schema test, overriding the default tests and finally where to find some pre-packaged schema tests that may save you the time of writing your own.

Learn to use the two ways that DBT allows you to define variables – In the dbt_project.yml file and on the command line.

Use hooks to simplify repetitive dbt run activities

Simplify dbt documentation with docs blocks as sometimes your documentation becomes a bit complex to the point where it would be better to write it outside of your YML file. Or perhaps you find you are re-writing the same definition multiple times.

Learn the syntax of the freshness block and how it works under the hood to define the acceptable amount of time between the most recent record, and now, for a table to be considered “fresh”.

Learn how to add query tags to a DBT project and then review the results in Snowflake.

Learn the manifest.json artifact and review step-by-step how to use the state method to only run modified models.

The multi-repo approach – how to structure DBT projects

Audit your DBT runs by using {{ invocation_id }}

The process of installing DBT into a virtual environment on windows, and look at how to install DBT directly from DBT core (GitHub) rather than by using pip.

Clean DBT project files, run/test models in group, and use EPHEMERAL DBT model

All project files used in this course are provided for students to use along the way.

Who this course is for
Data Analysts
Data Engineers
Database Administrators
Students