Spark SQL and PySpark 3 using Python 3 (Formerly CCA175)

使用 Python 3 使用 PySpark 2 和 3 的 Spark SQL 和数据帧 API 综合课程。

您将学到什么

• 与验证 HDFS 中的文件和文件夹相关的所有 HDFS 命令。
• 快速回顾与学习 Spark 相关的 Python
• 能够使用 Spark SQL 使用 SQL 样式语法解决问题。
• Pyspark Dataframe API 使用 Dataframe 样式 API 解决问题。
• Spark Metastore 将数据帧转换为临时视图的相关性，以便可以使用 Spark SQL 处理数据帧中的数据。
• Apache Spark 应用程序开发生命周期
• Apache Spark 应用程序执行生命周期和 Spark UI
• 设置 SSH 代理以访问 Spark 应用程序日志 Spark 应用程序的部署模式（集群和客户端）
• 在运行 Spark 应用程序时传递应用程序属性文件和外部依赖项

要求

• 使用任何编程语言的基本编程技能
• 自助实验室（提供说明）或 ITVersity 实验室，需要额外付费才能获得适当的环境。
• 所需的最小内存取决于您使用 64 位操作系统的环境
• 4 GB RAM 可访问适当的集群或 16 GB RAM 与 Cloudera QuickStart VM 等虚拟机

作为本课程的一部分，您将学习使用 Spark SQL 和使用 Python 作为编程语言的 Spark Data Frame API 构建数据工程管道的所有关键技能。本课程曾经是 CCA 175 Spark 和 Hadoop 开发人员课程，用于准备认证考试。自 2021 年 10 月 31 日起，考试结束，我们已使用 Python 3 将其重命名为 Apache Spark 2 和 3，因为它涵盖了认证范围之外的行业相关主题。

关于数据工程

数据工程只不过是根据我们的下游需求处理数据。作为数据工程的一部分，我们需要构建不同的管道，例如批处理管道、流管道等。与数据处理相关的所有角色都合并在数据工程下。通常，它们被称为 ETL 开发、数据仓库开发等。Apache Spark 已发展成为大规模处理数据工程的领先技术。

我为任何想使用 Pyspark (Python + Spark) 转换为数据工程师角色的人准备了这门课程。我本人是一位久经考验的数据工程解决方案架构师，在使用 Apache Spark 设计解决方案方面拥有成熟的经验。

让我们详细了解您将在本课程中学习的内容。请记住，该课程是通过大量实践任务创建的，这将使您能够使用正确的工具进行足够的练习。此外，还有大量的任务和练习来评估自己。

单节点大数据集群搭建

你们中的许多人希望从大型机、Oracle PL/SQL 等传统技术过渡到大数据，但您可能无法访问大数据集群。以正确的方式设置环境对您来说非常重要。如果您手边没有集群，请不要担心，我们将通过 Udemy Q&A 为您提供支持。

使用正确的配置设置基于 Ubuntu 的 AWS Cloud9 实例

确保 Docker 已设置

设置 Jupyter Lab 和其他关键组件

设置和验证 Hadoop、Hive、YARN 和 Spark

Python 的快速回顾

本课程需要对 Python 有一定的了解。为了确保您从数据工程的角度理解 Spark，我们添加了一个模块来快速使用 Python 进行热身。如果您不熟悉 Python，那么我们建议您学习我们的其他课程数据工程基础 – Python、SQL 和 Spark。

使用 Spark SQL 进行数据工程

让我们深入研究 Spark SQL，以了解如何使用它来构建数据工程管道。Spark with SQL 将使我们能够利用 Spark 的分布式计算功能以及易于使用的开发人员友好的 SQL 样式语法。

Spark SQL 入门

使用 Spark SQL 的基本转换

管理 Spark Metastore 表 – 基本 DDL 和 DML

管理 Spark Metastore 表 – DML 和分区

Spark SQL 函数概述

使用 Spark SQL 的窗口函数

使用 Spark Data Frame API 进行数据工程

Spark Data Frame API 是利用 Spark 的分布式计算能力大规模构建数据工程应用程序的另一种方法。具有应用程序开发背景的数据工程师可能更喜欢 Data Frame API 而不是 Spark SQL 来构建数据工程应用程序。

使用 PySpark 数据框 API 的数据处理概述

使用 PySpark 数据框 API 处理列数据

使用 PySpark 数据框 API 的基本转换 – 过滤、聚合和排序

使用 PySpark 数据框 API 连接数据集

使用 PySpark 数据框 API 的窗口函数 – 聚合、排名和分析函数

Spark Metastore 数据库和表

Apache Spark 应用程序开发和部署生命周期

作为基于 Apache Spark 的数据工程师，我们应该熟悉应用程序开发和部署生命周期。作为本节的一部分，您将了解开发和部署生命周期的完整生命周期。它包括但不限于生产代码、外部化属性、查看 Spark 作业的详细信息等等。

使用 Python 作为编程语言的 Apache Spark 应用程序开发生命周期

Spark 应用程序执行生命周期和 Spark UI

设置 SSH 代理以访问 Spark 应用程序日志

Spark应用部署方式

传递应用程序属性文件和外部依赖项

MP4 | Video: h264, 1280×720 | Audio: AAC, 44.1 KHz
Language: English | Size: 9.34 GB | Duration: 28h 36m

A comprehensive course on Spark SQL as well as Data Frame APIs using PySpark 2 and 3 leveraging Python 3.

What you’ll learn
All the HDFS Commands that are relevant to validate files and folders in HDFS.
Quick recap of Python which is relevant to learn Spark
Ability to use Spark SQL to solve the problems using SQL style syntax.
Pyspark Dataframe APIs to solve the problems using Dataframe style APIs.
Relevance of Spark Metastore to convert Dataframs into Temporary Views so that one can process data in Dataframes using Spark SQL.
Apache Spark Application Development Life Cycle
Apache Spark Application Execution Life Cycle and Spark UI
Setup SSH Proxy to access Spark Application logs
Deployment Modes of Spark Applications (Cluster and Client)
Passing Application Properties Files and External Dependencies while running Spark Applications

Requirements
Basic programming skills using any programming language
Self support lab (Instructions provided) or ITVersity lab at additional cost for appropriate environment.
Minimum memory required based on the environment you are using with 64 bit operating system
4 GB RAM with access to proper clusters or 16 GB RAM with virtual machines such as Cloudera QuickStart VM
Description
As part of this course, you will learn all the key skills to build Data Engineering Pipelines using Spark SQL and Spark Data Frame APIs using Python as a Programming language. This course used to be a CCA 175 Spark and Hadoop Developer course for the preparation of the Certification Exam. As of 10/31/2021, the exam is sunset and we have renamed it to Apache Spark 2 and 3 using Python 3 as it covers industry-relevant topics beyond the scope of certification.

About Data Engineering

Data Engineering is nothing but processing the data depending upon our downstream needs. We need to build different pipelines such as Batch Pipelines, Streaming Pipelines, etc as part of Data Engineering. All roles related to Data Processing are consolidated under Data Engineering. Conventionally, they are known as ETL Development, Data Warehouse Development, etc. Apache Spark is evolved as a leading technology to take care of Data Engineering at scale.

I have prepared this course for anyone who would like to transition into a Data Engineer role using Pyspark (Python + Spark). I myself am a proven Data Engineering Solution Architect with proven experience in designing solutions using Apache Spark.

Let us go through the details about what you will be learning in this course. Keep in mind that the course is created with a lot of hands-on tasks which will give you enough practice using the right tools. Also, there are tons of tasks and exercises to evaluate yourself.

Setup of Single Node Big Data Cluster

Many of you would like to transition to Big Data from Conventional Technologies such as Mainframes, Oracle PL/SQL, etc and you might not have access to Big Data Clusters. It is very important for you set up the environment in the right manner. Don’t worry if you do not have the cluster handy, we will guide you through with support via Udemy Q&A.

Setup Ubuntu based AWS Cloud9 Instance with the right configuration

Ensure Docker is setup

Setup Jupyter Lab and other key components

Setup and Validate Hadoop, Hive, YARN, and Spark

A quick recap of Python

This course requires a decent knowledge of Python. To make sure you understand Spark from a Data Engineering perspective, we added a module to quickly warm up with Python. If you are not familiar with Python, then we suggest you go through our other course Data Engineering Essentials – Python, SQL, and Spark.

Data Engineering using Spark SQL

Let us, deep-dive into Spark SQL to understand how it can be used to build Data Engineering Pipelines. Spark with SQL will provide us the ability to leverage distributed computing capabilities of Spark coupled with easy-to-use developer-friendly SQL-style syntax.

Getting Started with Spark SQL

Basic Transformations using Spark SQL

Managing Spark Metastore Tables – Basic DDL and DML

Managing Spark Metastore Tables – DML and Partitioning

Overview of Spark SQL Functions

Windowing Functions using Spark SQL

Data Engineering using Spark Data Frame APIs

Spark Data Frame APIs are an alternative way of building Data Engineering applications at scale leveraging distributed computing capabilities of Spark. Data Engineers from application development backgrounds might prefer Data Frame APIs over Spark SQL to build Data Engineering applications.

Data Processing Overview using PySpark Data Frame APIs

Processing Column Data using PySpark Data Frame APIs

Basic Transformations using PySpark Data Frame APIs – Filtering, Aggregations, and Sorting

Joining Data Sets using PySpark Data Frame APIs

Windowing Functions using PySpark Data Frame APIs – Aggregations, Ranking, and Analytic Functions

Spark Metastore Databases and Tables

Apache Spark Application Development and Deployment Life Cycle

As Apache Spark-based Data engineers we should be familiar with Application Development and Deployment Lifecycle. As part of this section, you will learn the complete life cycle of the Development and Deployment Life cycle. It includes but is not limited to productionizing the code, externalizing the properties, reviewing the details of Spark Jobs, and many more.

Apache Spark Application Development Lifecycle using Python as Programming Language

Spark Application Execution Life Cycle and Spark UI

Setup SSH Proxy to access Spark Application logs

Deployment Modes of Spark Applications

Passing Application Properties Files and External Dependencies

All the demos are given on our state-of-the-art Big Data cluster. You can avail of one-month complimentary lab access by reaching out to support@itversity.com with a Udemy receipt.

Who this course is for
Any IT aspirant/professional willing to learn Data Engineering using Apache Spark
Python Developers who want to learn Spark to add the key skill to be a Data Engineer