APACHE SPARK educational workshops

Apache Spark is a framework used for processing large amounts of data and it can be used to process all kinds of data. Apache Spark is a very popular system widely used for advanced analytics, data science, modern BigData architecture, as well as for complex batch (ETL) processing and real-time data processing. Spark contains several key components, such as Spark SQL for data structuring, Spark Streaming for processing big amounts of live data, Spark MLlib for machine learning, Spark GraphX for graph processing and SparkR for statistical data processing using R language. Spark can be run individually, on a YARN (Hadoop) cluster, or on Mesos – basically, it can be run in any environment. Spark is a polyglot framework, which means that it abstracts its usage to the maximum, and it imposes using a programming language (Python, Java, Scala, R) to a development environment which is the best fit for the organization or business type. All the examples in this education will be processed primarily in Python, but other programming languages, e.g. Scala, will also be used. Participants will work in independent and cluster environments, depending on the assignment.

Target audience

This education is aimed at IT architects, development engineers and business analysts.

Workshop Modules

Introduction to Spark is a workshop intended for everyone who wants to learn basic programming in the Spark framework. The workshop is modular, meaning that participants can choose modules depending on their points of interests. For instance, the participant can choose modules 1, 2 and 5, or 1, 2, 3 or 1, 2, 4, as well as combine them however they see fit.

Data Science is a topic suitable for BI experts, business analysts and predictive analysts.

Data Engineering is a topic convenient for system administrators, development engineers and system architects.

Workshop description according to modules

WORKSHOP PARTICIPATION PREREQUISITES

In order to partake in this training, you need to have knowledge of OO programming, as well as basic knowledge of SQL and basic knowledge of Python and/or Scala.

Introduction to Spark environments

This module covers an introduction to Spark and a basic explanation of how it works.  All aspects of this technology will be explained by means of interactive examples. System architecture, Apache Hadoop, the basics of MapReduce framework and basic Spark APIs RDD, DataFrame, Spark SQL will all be covered in detail.

This module is intended for IT architects, development engineers and business analysts.

Spark analytics – Spark in practice

The second module deals with developing a Spark application using previously acquired knowledge. The application will deal with advanced analytics and will process targeted data – ranging from loading big sets of data and cleaning to the final visualization.

Advanced usage – Spark in practice

The third module focuses on deploying Spark applications to production. How to set up a development environment, how to optimize Spark apps and how to run them in Hadoop environment will be explained in detail.

This module is aimed at IT architects and development engineers.

Advanced usage – Streaming and real-time data processing

The fourth module is aimed at participants that want to acquire advanced skills in Spark, such as Spark Streaming. The participants will be taught how to set up a streaming process for real-time data processing and will then upgrade it with streaming elements.

This module is intended for IT architects and development engineers.

Advanced usage – Data Science

The star of the fifth module is MLlib library for machine learning. The participants will build a model for machine learning on which they will show the process of training the model. By using Spark GraphX for processing graphs, several examples will show how to successfully use them in practice.

This module is aimed at business analysts and data science engineers.

Detailed descriptions of the workshop by days

Day 1 – Introduction to Apache Spark

What is Apache Spark?

Using Spark (independently, cluster, shell)

What is RDD and how to use it?

• Transformations
• Actions
• “Lazy evaluation”
• Data persistence – caching
• Functions (Python, Scala, Java)

Key/Value structures

• Types of data – creating and maintaining
• Transformation (aggregation, sorting, join)

DataFrame

• What is DataFrame?
• Transformations and actions on DataFrame
• Advanced usage

Loading and saving data

• Working with different file formats (TXT, JSON, AVRO, Parquet, Seq File)
• Working with different data repositories and file systems (Local, Amazon S3, HDFS)
• Databases (RDBMS, Cassandra, HBase)
• Spark SQL
• Connecting Spark SQL
• SQL in applications (initialization, DataFrames, Caching)
• Functions

Day 2 – Spark Analytics – Spark in practice

Collecting data for the full-day task in the context of advanced analytics

• Getting to know the data set
• Job requirements and task goals

Data preparation and cleaning

• Profiling data and existing structures
• Cleaning data by removing anomalies and potential errors
  • Aggregating data
• Methods of aggregating data
• Finding the optimal case
• Process optimization

Application debugging

Data visualization

Day 3 – Spark in production

Development environment:

• Workspace (PyCharm, Anaconda, Zeppelin and Jupyter notebook)
• Application build
• Application deployment
• Debugging
• Testing (Unit test, integration test)

Running Spark applications on cluster environments (Hadoop YARN)

• Executor and worker optimization
• HW Sizing
• Monitoring

Spark Memory Management

• Persistence – Caching
• Tungsten
• Garbage collection

24/7 Streaming operations

• Streaming process control mechanisms – Checkpointing
• Streaming fault-tolerance (driver, worker, receiver, process guarantees)
• Data supervision
• Process supervision
• UI Streaming

Performance management

• System resizing (batch and window size)
• Level parallelism

Day 4 – Spark Streaming – real-time data processing in practice

Introduction to streaming processes

Data sets for a streaming process

• Correcting existing data sets
• Preparation for streaming

Spark Streaming

• Introduction to Spark Streaming
• Basic concepts and operations
• Streaming operations (Transformations, window operations)

Spark Structured Streaming

• Structure Streaming API concept
• Continuous application
• Operations on streaming
• Stream management and queries
• Stream recovery and checkpointing

Operations for controlling the stream

• Output operations (SaveDStream)
• Data stream sources (basic sources, socket, Kafka, files)
• Managing multiple sources
• Resizing clusters for streaming operations

Day 5 – Data Science

Using SparkMLlib for machine learning

The basics of machine learning

Types of data – vectors

Using most frequent algorithms with concrete Spark examples

• Classifications and regression
• “Feature Extraction”
• Clustering
• Dimension reduction

Tricks and best practices

Model evaluation

Spark GraphX – processing a graph

The basics of graphs and why we use them

How to prepare a data set (GraphFrames)

Graph algorithms used on graphs using Spark GraphX

• ShortestPaths
• PageRank
• ConnectedComponents

For more information, feel free to contact us!