Bored MLE

  Hi All Continuing from Part 1, this post details the Kubernetes and Observability configs of the project. The full source is available here . Let's break the code down shall we.  1. K3's Server Config ( infra/server-config.yaml )  write-kubeconfig-mode: "0644"  *      Sets file permissions for kubeconfig file (readable by all users in the group) *      0644 means owner can read/write, group and others can only read disable: - traefik - servicelb - local-storage - metrics-server *      Disables default k3s components that we'll replace with better alternatives. Components Disabled: *      _traefik: Replaced with ingress-nginx for better control *      _servicelb: Replaced with MetalLB or cloud load balancer *      _local-storage: Replaced with Longhon for dynamic provisioning *      _metrics-server: Replaced with Prometheus for better monit...

  Hi All  In this first part of a series on eBPF and Open Telemetry . We detail a couple of useful eBPF scripts tailored for an MLE with no background of the project(s). They focus on observability, performance monitoring and data collection - key areas where eBPF shines. I tried to make these as practical as possible, let's get to it. 1. Tracing Python Function Calls in ML Pipelines Monitor which Python functions are being called in an ML training script (eg. PyTorch/Tensorflow) and their execution time. This helps identity bottlenecks in data loading, preprocessing or model training. Tools: *    bpftrace (for high level scripting) *    libbpf (for custom C-based eBPF programs) *    Run the below script while your ML script is executing. sudo bpftrace -e 'uprobe:python3:PyEval_CallObject { printf("Function called: %s\n", str(arg1)); }' *    Trace all torch.Tensor method calls in a PyTorch script *    The output will show ...

    Hi All I'm aiming to solve some tough issues in the ML backend, and heterogeneous backend support is up there as one of the most pressing . Today we'll be training BERT on a Rocm (AMD) backend and running the binary (inference) on a Nvidia backend.  Training a complex model like BERT (250M+ params) on a ROCm backend and then running the trained model on NVIDIA backend is not impossible, but it comes with some challenges and considerations. Here's a breakdown of the key aspects:  Steps to achieve Cross-Backend Training/Inference Option 1: Train on ROCm, Inference on NVIDIA (Same Framework) 1.     Train the model on AMD GPU's using PyTorch/Tensorflow with ROCm. # Example PyTorch training script (ROCm) import torch device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # ... rest of the training code *      Note: ROCm uses cuda device string for compatibility, but maps it to AMD hardware. 2.      Save t...