Table of Contents

👋 I am Aurimas. I have now written and sent out almost 40 SwirlAI Newsletter episodes.

The value of most of the content in this Newsletter is meant to be timeless - the content can be read long after the episodes are released. However, until now it was complicated to find topics of interest as they are scattered throughout newsletter issues with no clear ordering or grouping. Hence, I built a Table of Contents that I will be updating and curating.

The amount of content really surprised me. There are more than 90 topics covered supplemented by a infographic drawn in my style!

The SwirlAI Data Engineering Project Master Template.

• Introduction

Data Engineering

Fundamentals

• Row Based vs. Column Based File Format

• Splittable vs. Non-Splittable Files

• Data Contracts

• Data Contracts in the Data Pipeline

• Data Contracts and Outbox Tables

• ACID Properties in DBMS

• The CAP Theorem

• Lambda vs. Kappa Architecture

• Architecture of Database Management System (DBMS)

• SQL Query order of Execution

• Encoding in Parquet

• Change Data Capture (CDC)

• What is a Vector Database?

Spark

• Architecture

• Shuffle

• Distributed Joins

• Parallelism

• Caching

• Executor Memory Structure

• Partitioning vs. Bucketing (Part 1)

Kafka

• Use Cases

• Writing Data

• Reading Data

• At-Least/Most Once Processing

• Data Retention Strategies

• Data Replication

• Reliability

Stream Processing

• Event vs. Processing Time

• Introduction to Watermarks in Stream Processing Systems

• Apache Flink - Architecture

General

• Airflow - Architecture

• Event Latency in Data Systems

• Mounting a Feature Store on top of Curated Data

MLOps

Deployment

• Machine Learning Deployment Types

• Batch Model Deployment

• Stream Processing Model Deployment

• Implementing ML Inference in Streaming Applications

• Real Time Model Deployment

• The 4 types of ML Model Deployment

• Decomposing Real Time Machine Learning Service Latency

• Model Deployment - Scaling

• Model Deployment - AutoScaling

• Patterns for implementing Business Logic in Machine Learning Services

• Real Time Model Release Strategies

• Machine Learning model compression

Infrastructure

• Feature Store

• Central Role of The Model Registry

• Experiment/Model Tracking

• Experimentation Environments

• The Great Divide of MLOps Tooling Landscape

• Feature Platforms

• Building efficient Experimentation Environments for ML Projects

Observability

• Real Time Model Observability

• Training/Serving Skew

• Types of Training/Serving Skew

• Model Observability - Analytical Logs

• Feature vs. Concept Drift

• Data Quality in Machine Learning Systems

Processes

• Machine Learning Pipeline

• Static vs. Dynamic Features

• Data Latency in Real Time ML Systems

• Continuous Training (CT) in Machine Learning Systems

• Feedback Loops in Machine Learning System

• CI/CD for Machine Learning

• Latency of Feedback Loops in Machine Learning applications

Maturity Levels

• MLOps Maturity Levels - Level 0

• MLOps Maturity Levels - Level 1

• MLOps Maturity Levels - Level 2

System Design

• Search/Recommender System Design

• Website Activity Tracking System Design

• Decomposing the Data System

• Deconstructing a Feature Store

• Twitter's Recommender System

• LLM based Chatbots to query your Private Knowledge Base

Systems Thinking

• ML Systems in the context of end-to-end Product Delivery

• Why you should learn Data Engineering and Machine Learning Engineering Pipelines

Building Organisations

• Placing Data Roles on the Data Value Chain

• The Data Value Chain

• Organisational structure for effective MLOps

• Organisational Structure for Effective MLOps Implementation (Notes)

Containers and Kubernetes

• What's in Kubernetes for MLOps?

Career

• Moving to MLOps Engineering from other roles

• Data Engineer's Learning Path

If you like the content, consider supporting my work by Subscribing and Sharing the article.