ML Observability with eBpf and OTel Pt. 1: Basics | ML Engineering and MLOps
Hi All In this first part of a series on eBPF and Open Telemetry . We detail a couple of useful eBPF scripts tailored for an MLE with no background of the project(s). They focus on observability, performance monitoring and data collection - key areas where eBPF shines. I tried to make these as practical as possible, let's get to it. 1. Tracing Python Function Calls in ML Pipelines Monitor which Python functions are being called in an ML training script (eg. PyTorch/Tensorflow) and their execution time. This helps identity bottlenecks in data loading, preprocessing or model training. Tools: * bpftrace (for high level scripting) * libbpf (for custom C-based eBPF programs) * Run the below script while your ML script is executing. sudo bpftrace -e 'uprobe:python3:PyEval_CallObject { printf("Function called: %s\n", str(arg1)); }' * Trace all torch.Tensor method calls in a PyTorch script * The output will show ...