General Design Principles for Data Processing
Overview
Today, successful data architectures and data processing platforms must undertake continuous modernization to maintain data platform sustainability. Various technologies (ETL, Streaming, MLOps) and concepts have emerged over the past decade and sustainable ETL process design that isn’t directly dependent on a platform or concept is becoming increasingly challenging. This education is intended for engineers and architects who wish to develop their own Framework or standard for data processing, whether it’s for populating a DWH, DataLake, LakeHouse, Data Fabric, or Data Mesh to ensure sustainability on the given platform.
The primary goal of this education is to teach participants how to efficiently use key design patterns depending on the business case and architectural principles. This course covers all the important architectural patterns for data processing, ranging from ETL, ELT, and near real-time, to advanced transformations and support for streaming processing. The education will provide answers on how to approach the analysis of existing processing requirements and define standards for each individual process.
Target audience
The course is intended for business and technical teams that need an introduction and a structured overview of Modern Data Architecture. Its intended roles are architects, data engineers, and business and data analysts.
Prerequisites
The prerequisite for this course is that the participants are familiar with the concepts and architectures of data management.
Content
Day 1:
- Introduction and Overview
- Review of the agenda
- Introduction to the topic: Overview of data architectures and data processing
- Overview of relevant technologies and main requirements for Batch, Streaming, MLOps
- Introduction to Concepts
- Data Architecture: DWH, DataLake, LakeHouse, Data Fabric, Data Mesh
- Definition and importance of Data Ingestion processes
- Definition and importance of traditional ETL processes
- Definition and importance of self-service data processing
- Workshop: Analysis requirements for building individual Framework and Standards
- A practical workshop where participants analyse examples of their Frameworks and standards
- Discussion and Q&A
- Discussion of the material covered during the day
- Answers to questions
Day 2:
- Application of Key Design Patterns
- Learning about key design patterns (CDC, push, pulls, API)
- Review of business cases and architectural principles
- Data Processing and Architectural Patterns
- Overview of all important architectural patterns for data processing
- A detailed discussion of ETL, ELT, near real-time, advanced transformations, and support for streaming processing
- A detailed discussion of automation of the whole process
- Workshop: Build and Publish Standards
- Defining and publishing standards for each individual process (design, build, control)
- Conclusion and Q&A
- Review of what was covered during the two days
- Opportunity for additional questions and discussion
For all inquiries regarding education, please contact us at learn@croz.net.
More information:
For all inquiries regarding education, please contact us at learn@croz.net or apply online.
