Spark, Ray, and Python for Scalable Data Science

课程说明

第 1 课：Python 分布式计算简介

第 1 课首先介绍数据科学过程和工作流。然后转向一些历史，说明为什么需要像 Spark 和 Ray 这样的框架。接下来是分布式系统理论的简短入门。接下来是基于 Python 的分布式计算框架。最后，Jonathan 开始解释 Spark 生态系统以及 Spark 与 Ray 的比较。

第 2 课：使用 Spark 扩展数据处理

第 2 课从使用 Spark 编程的“Hello World”示例开始详细介绍 Spark 框架。然后 Jonathan 转向 Spark API。您对 Spark 的主要数据结构之一——弹性分布式数据集 (RDD) 有一定的经验。接下来是键值对以及 Spark 如何对它们进行类似于 MapReduce 的操作。本课以一些 Spark 内部结构和整个 Spark 应用程序生命周期结束。

第 3 课：使用 PySpark 进行探索性数据分析

在第 3 课中，Jonathan 继续使用 Spark，但现在是在以自然语言处理 (NLP) 为中心的更大数据科学工作流的上下文中。他首先介绍了探索性数据分析 (EDA)，然后快速浏览了 Jupyter 笔记本。接下来，他讨论了如何使用 Spark 大规模执行 EDA，然后向您展示如何创建统计数据和数据可视化来汇总数据集。最后，他处理了 NLP 示例，向您展示了如何将大型文本语料库转换为适合机器学习的数字表示。

第 4 课：使用 Ray 进行并行计算

第 4 课介绍了 Ray 编程 API，Jonathan 比较了 Ray 和 Spark API 之间的异同。您将学习如何使用 Ray 分发函数，以及如何使用 Ray actor 使用分布式类或对象执行操作。最后，Jonathan 完成了大规模模拟以突出 Ray 框架的优势。

第 5 课：使用 Ray 扩展 AI 应用程序

第 5 课讨论 Ray 如何使您能够使用 Python 扩展机器学习和人工智能应用程序。本课从 Python 中的一般模型训练和评估过程开始。然后转向 Ray 如何使您能够扩展我们模型的评估和调整。您会看到 Ray 如何使非常有效的超参数调整成为可能。您还将看到，一旦您拥有一个经过训练的模型，Ray 就可以从您的机器学习模型中进行预测。最后，本课程最后介绍 Ray 如何使您能够将机器学习模型部署到生产中并在它们出现后对其进行监控。

MP4 | Video: AVC 1280 x 720 | Audio: AAC 48 Khz 2ch | Duration: 07:01:37 | 8.57 GB
Genre: eLearning | Language: English

Conceptual overviews and code-along sessions get you scaling up your data science projects using Spark, Ray, and Python.

Overview

Machine learning is moving from futuristic AI projects to data analysis on your desk. You need to go beyond following along in discussions to coding machine learning tasks. Spark, Ray, and Python for Scalable Data Science LiveLessons show you how to scale machine learning and artificial intelligence projects using Python, Spark, and Ray.

About the Instructor

Jonathan Dinu is the founder of Zipfian Academy—an advanced immersive training program for data scientists and data engineers in San Francisco—and served as its CAO/CTO before it was acquired by Galvanize, where he now is the VP of Academic Excellence. He first discovered his love of all things data while studying Computer Science and Physics at UC Berkeley, and in a former life he worked for Alpine Data Labs developing distributed machine learning algorithms for predictive analytics on Hadoop.

Jonathan is a dedicated educator, author, and speaker with a passion for sharing the things he has learned in the most creative ways he can. He has run data science workshops at Strata and PyData (among others), built a Data Visualization course with Udacity, and served on the UC Berkeley Extension Data Science Advisory Board. Currently he is writing a book on practical Data Science applications using Python. When he is not working with students, you can find him blogging about data, visualization, and education at

Skill Level
Beginner to Intermediate
Learn How To
Integrate Python and distributed computing
Scale data processing with Spark
Conduct exploratory data analysis with PySpark
Utilize parallel computing with Ray
Scale machine learning and artificial intelligence applications with Ray
Who Should Take This Course
This course is a good fit for anyone who needs to improve their fundamental understanding of scalable data processing integrated with Python for use in machine learning or artificial intelligence applications.
Course Requirements
A basic understanding of programming in Python (variables, basic control flow, simple scripts).
Familiarity with the vocabulary of data processing at scale, machine learning (dataset, training set, test set, model), and AI.
Lesson Descriptions

Lesson 1: Introduction to Distributed Computing in Python

Lesson 1 starts with an introduction to the data science process and workflow. It then turns to a bit of history on why frameworks like Spark and Ray are necessary. Next comes a short primer on distributed systems theory. Python-based distributed computing frameworks come up next. Finally, Jonathan begins to explain the Spark ecosystem as well as how Spark compares to Ray.

Lesson 2: Scaling Data Processing with Spark

Lesson 2 goes into detail on the Spark framework beginning with a “Hello World” example of programming with Spark. Then Jonathan turns to the Spark APIs. You get some experience with one of Spark’s primary data structures, the resilient distributed dataset (RDD). Next is key-value pairs and how Spark does operations on them similar to MapReduce. The lesson finishes up with a bit of Spark internals and the overall Spark application lifecycle.

Lesson 3: Exploratory Data Analysis with PySpark

In Lesson 3, Jonathan continues using Spark but now in the context of a larger data science workflow centered around natural language processing (NLP). He starts off with a general introduction to exploratory data analysis (EDA), followed by a quick tour of Jupyter notebooks. Next he discusses how to do EDA with Spark at scale, and then he shows you how to create statistics and data visualizations to summarize data sets. Finally, he tackles the NLP example, showing you how to transform a large corpus of text into numerical representation suitable for machine learning.

Lesson 4: Parallel Computing with Ray

Lesson 4 introduces the Ray programming API, with Jonathan comparing the similarities and differences between the Ray and Spark APIs. You learn how you can distribute functions with Ray, as well as how you can perform operations with distributed classes or objects with Ray actors. Finally, Jonathan finishes up with a large scale simulation to highlight the strengths of the Ray framework.

Lesson 5: Scaling AI Applications with Ray

Lesson 5 discusses how Ray enables you to scale up machine learning and artificial intelligence applications with Python. The lesson starts with the general model training and evaluation process in Python. Then it turns to how Ray enables you to scale both the evaluation and tuning of our models. You see how Ray makes possible very efficient hyperparameter tuning. You also see how, once you have a trained model, Ray can serve predictions from your machine learning model. Finally, the lesson finishes with an introduction to how Ray can enable you to both deploy machine learning models to production and monitor them once they are there.