Bored MLE

  Hi All  Alright today on "Bored MLE" we're doing some inference, using a LoRA optimized BERT model on GKE . Now there's a lot more to MLOps than just this (like cluster level observability for example), but for this example I'm keeping it brief.  The full code is available on GitHub with added minikube deployment instructions (I only cover the Cloud deployment here).    I assume familiarity with Python, PyTorch, K8s, FastApi, Minikube and GKE. Let's get on with it.  View the full FastApi service source below, also available here : Let's breakdown the above code, block by block. 0. Install Dependencies :  pip install fastapi uvicorn torch transformers peft accelerate  1. Imports from fastapi import FastAPI, HTTPException from pydantic import BaseModel from typing import List import torch from transformers import AutoModelForSequenceClassification, AutoTokenizer from peft import PeftModel, PeftConfig  *      fastapi : F...

    Hi All I'm aiming to solve some tough issues in the ML backend, and heterogeneous backend support is up there as one of the most pressing . Today we'll be training BERT on a Rocm (AMD) backend and running the binary (inference) on a Nvidia backend.  Training a complex model like BERT (250M+ params) on a ROCm backend and then running the trained model on NVIDIA backend is not impossible, but it comes with some challenges and considerations. Here's a breakdown of the key aspects:  Steps to achieve Cross-Backend Training/Inference Option 1: Train on ROCm, Inference on NVIDIA (Same Framework) 1.     Train the model on AMD GPU's using PyTorch/Tensorflow with ROCm. # Example PyTorch training script (ROCm) import torch device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # ... rest of the training code *      Note: ROCm uses cuda device string for compatibility, but maps it to AMD hardware. 2.      Save t...