ML Observability with eBpf and OTel Pt. 1: Basics | ML Engineering and MLOps

 


Hi All 

In this first part of a series on eBPF and Open Telemetry. We detail a couple of useful eBPF scripts tailored for an MLE with no background of the project(s). They focus on observability, performance monitoring and data collection - key areas where eBPF shines. I tried to make these as practical as possible, let's get to it.

1. Tracing Python Function Calls in ML Pipelines

Monitor which Python functions are being called in an ML training script (eg. PyTorch/Tensorflow) and their execution time. This helps identity bottlenecks in data loading, preprocessing or model training.

Tools:

*    bpftrace (for high level scripting)
*    libbpf (for custom C-based eBPF programs)
*    Run the below script while your ML script is executing.


sudo bpftrace -e 'uprobe:python3:PyEval_CallObject { printf("Function called: %s\n", str(arg1)); }'

*    Trace all torch.Tensor method calls in a PyTorch script
*    The output will show which Python functions are being called (eg. torch.Tensor.forward)
*    Useful in identifying slow functions in ML pipelines (eg. data augmentation, model inference)
*    Correlate function calls with performance (eg. CPU/GPU usage).

2. Monitoring GPU Memory and Kernel Launches (CUDA) 

Track GPU memory usage and CUDA kernel launches in ML workloads (eg. PyTorch/CUDA). This helps debug OOM errors or inefficient GPU memory usage.

Tools:

*    bpftrace with CUDA probes
*    nvidia-smi (for comparison)


sudo bpftrace -e 'tracepoint:cuda:kernel_start { printf("Kernel launched: %s\n", str(args->name)); }'

*    Run this while your CUDA-accelerated ML script is running
*    The output will show kernel names (eg. aten::matmul)
*    Detect Memory leaks in GPU accelerated ML models
*    Correlate kernel launches with GPU utilization (eg. using $nvidia-smi)

3. Measuring TCP Latency For Distributed Training

Monitor network latency between nodes in a distributed ML training setup (eg. PyTorch Distributed, Horovod). High latency can bottleneck gradient synchronization.

Tools:
*    bpftrace with TCP probe
*    ss or netstat (for comparison) 


sudo bpftrace -e 'kprobe:tcp_retransmit_skb { @[comm] = count(); }'

*    Run this on all nodes in your distributed training cluster.
*    The output will show which processes are experiencing retransmissions.
*    Diagnose network issues affecting distributed training performance.
*    Correlate latency spikes with model convergence issues.

4. Profiling Disk I/O for Data Loading

Monitor disk I/O patterns during data loading (eg. PyTorch DataLoader). Slow I/O can bottleneck training loops.
 

Tools:
*    bpftrace with block I/O probes
*    iostat (for comparison)  


sudo bpftrace -e 'tracepoint:block:block_rq_issue { printf("PID %d: %s %d bytes\n", pid, args->rwbs, args->bytes); }'

*    Run this while your ML script is loading data
*    The output will show which files are being read and their sizes.
*    Identify slow data loading bottlenecks.
*    Optimize data pipelines (eg, caching, prefetching)

5. Tracing System Calls for ML Containers

Monitor system calls made by ML containers (eg. Docker) to debug permission issues or unexpected behavior.

Tools:
*    bpftrace with syscall probes
*    strace (for comparison) 


sudo bpftrace -e 'tracepoint:syscalls:sys_enter_open { printf("PID %d opened %s\n", pid, str(args->filename)); }'

Trace all open syscalls in a container

*    Run this on the host while the container is running
*    The output will show which files the container is accessing.
*    Debug files access issues in ML containers
*    Monitor unexpected syscalls (eg. due to misconfigured environments)

These experiments are all very practical and can be run on Linux, and provide insights that are valuable for ML workflows (eg. debugging latency, monitoring GPU usage or tracing system calls). Find the full source for this example here, that's it for now folks. Stay tuned for part 2 of this observability series where we'll look at integrating eBPF with our observability stack (Prometheus, Grafana and OTel), for device level observability at scale.

 Later

-    Ed 

Comments

Post a Comment

Popular posts from this blog

Fine-Tuning Mistral 7B using QLoRA with PyTorch pt. 1: The Model | ML Engineering

Image
     Hi All Today we're working with a popular and slightly bigger model than our previous example. Mistral 7B is capable of chat and light coding tasks, for older hardware it's a winner for sure.  Here's a complete, runnable example of fine-tuning Mistral 7B using QLoRA with the peft , transformers , and bitsandbytes libraries. This example assumes you're working with a single GPU (eg. an A100 or similar). First install the required packages: pip install -q bitsandbytes datasets accelerate peft transformers trl View full script below, also available here :   Full breakdown of the script above, block-by-block. 1.      Dataset Loading dataset = load_dataset("timdettmers/openassistant-guanaco", split="train") *      Loads a preprocessed instruction-following dataset (Guanco, derived from OpenAssistant). *      split="train" selects the training portion *      The dataset is in a conversational ...
Read more

Deploying LoRA Optimised BERT as a FastApi service on GKE | ML Engineering & MLOps

Image
  Hi All  Alright today on "Bored MLE" we're doing some inference, using a LoRA optimized BERT model on GKE . Now there's a lot more to MLOps than just this (like cluster level observability for example), but for this example I'm keeping it brief.  The full code is available on GitHub with added minikube deployment instructions (I only cover the Cloud deployment here).    I assume familiarity with Python, PyTorch, K8s, FastApi, Minikube and GKE. Let's get on with it.  View the full FastApi service source below, also available here : Let's breakdown the above code, block by block. 0. Install Dependencies :  pip install fastapi uvicorn torch transformers peft accelerate  1. Imports from fastapi import FastAPI, HTTPException from pydantic import BaseModel from typing import List import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer from peft import PeftModel, PeftConfig  *      fastapi : F...
Read more

Advanced Rust ML: Custom Modules with Tch-rs | ML Engineering

Image
  Hi All Another day, another Rust ML backend application. Today we're looking at defining custom modules in Rust in the ML backend. We're going to be creating a custom linear layer in Rust using Tch-rs ( Rust PyTorch bindings ).   View full source below, also available here :   Let's break the above code down. Block by block. 1. Imports use tch::{nn, nn::Module, Tensor}; What it does:  *        tch::nn - Neural network module containing layer definitions. *      nn:Module - The trait that all neural network modules must implement. *      Tensor - The Tensor type used throughout tch-rs   2. Struct Definition   struct CustomLayer { weight: Tensor, bias: Tensor, } What it does: *      weight - A tensor holding a layer's weights (matrix) *      bias -  A tensor holding the layer's bias (vector) This is essentially a linear layer (fully connected laye...
Read more