Data Engineering – SSIS/ETL/Pipelines/Python/Web Scraping

数据工程师是创建大数据 ETL 管道的人，可以获取大量数据并将其转化为洞察力。他们专注于数据的生产准备以及格式、弹性、扩展性和安全性等方面。

SQL Server Integration Services 是 Microsoft SQL Server 数据库软件的一个组件，可用于执行范围广泛的数据迁移任务。SSIS 是一个用于数据集成和工作流应用程序的平台。它具有用于数据提取、转换和加载的数据仓库工具。

ETL 代表提取、转换和加载，是一种数据集成过程，它将来自多个数据源的数据组合到一个单一的、一致的数据存储中，然后再加载到数据仓库或其他目标系统中。

ETL 管道是用于将数据从一个源或多个源移动到数据库（如数据仓库或目标数据库）的一组流程。

SQL Server 集成服务 (SSIS) 提供了一种方便且统一的方式来从不同来源读取数据（提取）、执行聚合和转换（转换），然后集成数据（加载）以用于数据仓库和分析目的。当您需要处理大量数据（GB 或 TB）时，SSIS 成为此类工作负载的理想方法。

Web 抓取、Web 收集或 Web 数据提取是用于从网站中提取数据的数据抓取。网络抓取软件可以使用超文本传输​​协议或网络浏览器直接访问万维网。虽然网络抓取可以由软件用户手动完成，但该术语通常是指使用机器人或网络爬虫实现的自动化过程。它是一种复制形式，其中从网络收集和复制特定数据，通常复制到中央本地数据库或电子表格中，以供以后检索或分析。

Published 06/2022
Genre: eLearning | MP4 | Video: h264, 1280×720 | Audio: AAC, 44.1 KHz
Language: English | Size: 1.96 GB | Duration: 79 lectures • 6h 30m

Creates ETL pipelines and extract data from the web.

What you’ll learn
Creates ETL pipelines
Extract data from the web with Python
Create SSIS Package
Execute SSIS Package
Build Web Scraping Script
Prototype web scraping script
Configure data source and data destination
Clean and Transform Data
Perform Data migration from SQL Server to Oracle

Requirements
Basic knowledge of Python advised
Basic knowledge of database concepts advised
Description
A data engineer is someone who creates big data ETL pipelines, and makes it possible to take huge amounts of data and translate it into insights. They are focused on the production readiness of data and things like formats, resilience, scaling, and security.

SQL Server Integration Services is a component of the Microsoft SQL Server database software that can be used to perform a broad range of data migration tasks. SSIS is a platform for data integration and workflow applications. It features a data warehousing tool used for data extraction, transformation, and loading .

ETL, which stands for extract, transform and load, is a data integration process that combines data from multiple data sources into a single, consistent data store that is loaded into a data warehouse or other target system.

An ETL pipeline is the set of processes used to move data from a source or multiple sources into a database such as a data warehouse or target databases.

SQL Server Integration Service (SSIS) provides an convenient and unified way to read data from different sources (extract), perform aggregations and transformation (transform), and then integrate data (load) for data warehousing and analytics purpose. When you need to process large amount of data (GBs or TBs), SSIS becomes the ideal approach for such workload.

Web scraping, web harvesting, or web data extraction is data scraping used for extracting data from websites. The web scraping software may directly access the World Wide Web using the Hypertext Transfer Protocol or a web browser. While web scraping can be done manually by a software user, the term typically refers to automated processes implemented using a bot or web crawler. It is a form of copying in which specific data is gathered and copied from the web, typically into a central local database or spreadsheet, for later retrieval or analysis.

Who this course is for
Beginners to Data Engineering