Bored MLE

     Hi All Today we're working with a popular and slightly bigger model than our previous example. Mistral 7B is capable of chat and light coding tasks, for older hardware it's a winner for sure.  Here's a complete, runnable example of fine-tuning Mistral 7B using QLoRA with the peft , transformers , and bitsandbytes libraries. This example assumes you're working with a single GPU (eg. an A100 or similar). First install the required packages: pip install -q bitsandbytes datasets accelerate peft transformers trl View full script below, also available here :   Full breakdown of the script above, block-by-block. 1.      Dataset Loading dataset = load_dataset("timdettmers/openassistant-guanaco", split="train") *      Loads a preprocessed instruction-following dataset (Guanco, derived from OpenAssistant). *      split="train" selects the training portion *      The dataset is in a conversational ...

    Hi All I'm aiming to solve some tough issues in the ML backend, and heterogeneous backend support is up there as one of the most pressing . Today we'll be training BERT on a Rocm (AMD) backend and running the binary (inference) on a Nvidia backend.  Training a complex model like BERT (250M+ params) on a ROCm backend and then running the trained model on NVIDIA backend is not impossible, but it comes with some challenges and considerations. Here's a breakdown of the key aspects:  Steps to achieve Cross-Backend Training/Inference Option 1: Train on ROCm, Inference on NVIDIA (Same Framework) 1.     Train the model on AMD GPU's using PyTorch/Tensorflow with ROCm. # Example PyTorch training script (ROCm) import torch device = torch.device("cuda" if torch.cuda.is_available() else "cpu") # ... rest of the training code *      Note: ROCm uses cuda device string for compatibility, but maps it to AMD hardware. 2.      Save t...