Snowflake cloud database with ETL(Airflow+Python+Talend)

在我们之前的课程Snowflake Masterclass [实时演示+最佳实践+实验室]中，我们深入了解了snowflake的基础知识，解决了大量作业，并了解了加载数据和卸载数据的最佳实践。

此外，我们仔细评估了大部分雪花特征，以了解它们在幕后的工作方式。通过这些讨论，您了解了如何有效地使用 Snowflake。

有一个缺失的部分，如何在 Snowflake 上构建和编排 ETL 工作流。本课程仅此而已。

在本课程中，我们将学习，

在 Airflow 中构建工作流。

我们将利用 Talend 功能构建通用代码以在雪花中摄取数据和处理数据。

我们将构建审计表并记录我们在雪花上发出的每个命令。我们将记录每个任务消耗的时间并捕获雪花积分。

构建框架后，我们将构建一个工作流来处理和转换 250 + GB 的纽约市交通数据。

最后，我们将用 python 连接雪花并编写代码来捕获我们加载到雪花的数据的统计信息。

您还可以访问预配置的 Jupyter notebook，以在 Snowflake 上运行您的 Python 代码。

如果您以前没有使用过 Talend、Airflow 和 Python，请不要担心它们是非常简单的工具，我将提供必要的介绍。

Genre: eLearning | MP4 | Video: h264, 1280×720 | Audio: AAC, 48.0 KHz
Language: English | Size: 3.62 GB | Duration: 4h 22m
Building ETL workflows for snowflake.

What you’ll learn
Build workflows for snowflake.
Talend basics.
Airflow basics.
Building airflow Dags.
Connect snowflake with python.
Auditing snowflake commands.
Capturing cost and performance metrics.
Virtual machine with preconfigured airflow.

Description
In our previous course Snowflake Masterclass [Real time demos+Best practices+Labs] we deep-dived and understood the fundamentals of snowflake, solved lot of assignments, and understood best practices to load data and unload data.

Also, we closely evaluated most of the snowflake features understanding how they work under the hood. Through these discussions, you realized how to use Snowflake efficiently.

There was one missing piece, How to build and orchestrate ETL workflows on Snowflake. This course is just about that.

In this course, we are going to learn,

Build workflows in Airflow.

We will leverage Talend capabilities to build generic code to ingest data and process data in snowflake.

We will build audit tables and record every command we fire on the snowflake. We will record the time consumed for each task and capture snowflake credits.

Once we build the framework we will build a workflow to process and transform 250 + GB volume of NYC traffic data.

At last, we will connect the Snowflake with python and write code to capture stats of data we loaded to the snowflake.

you will also get access to preconfigured Jupyter notebook to run your python code on the Snowflake.

If you have previously not worked with Talend, Airflow and Python don’t worry they are very simple tools I will provide the necessary introduction.

I am sure you will learn a lot from this journey. See you in the course!!