Rust experiments: Wrapping unsafe (malloc/free) in safe abstractions | GPU engineering

 


 

I'll have a full example soon on how to wrap unsafe code (malloc/free like in a C++ Cuda kernel), in safe abstractions using Rust. See snippet below:

 

 

Comments

Post a Comment

Popular posts from this blog

Advanced Rust ML: Loading Pre-trained models with Tch-rs | ML Engineering

Image
  Hi All Today we're looking at some Rust ML with  Tch-rs . Now Tch-rs is a  Rust bindings library for the C++ api of PyTorch. "The goal of the tch crate is to provide some thin wrappers around the C++ PyTorch api (a.k.a. libtorch)" - github . Let's get to it then. Let's demonstrate tch-rs's core capability: creating tensors, moving them to devices, performing operations and getting results - all in Rust while using PyTorch's backend. See introductory example below.  use tch::{Device, Tensor, Kind}; fn main() { let device = Device::cuda_if_available(); let x = Tensor::of_slice(&[1.0, 2.0, 3.0]).to_device(device); let y = Tensor::of_slice(&[4.0, 5.0, 6.0]).to_device(device); let z = x + y; println!("{:?}", z); } Breakdown: *      Imports : Device , Tensor and Kind   from Tch-rs, the core types for device management and memory operations. *        Device Selection : Device::cuda_if_available(); a...
Read more

PyTorch: Training TinyLlama 1.1B in Google Colab | ML Engineering

Image
Hi All Today we'll be loading a pre-trained model and train it on Colab. Here's a minimal example using Huggingface for transformers and accelerate for memory efficiency. View source below. Let's break down the above code. Block by block. I'm assuming you're using a notebook for this example. However in the full example in my GitHub I've included the Ops files too (Docker and k8s for GKE or AKS). I don't go into those here, I'll have a series of articles tacking MLOps in depth, this is about fine-tuning a pre-trained small LLM in Colab.  Prep: Setup Environment   !pip install -q transformers accelerate datasets peft bitsandbytes   *     transformers provides pre-trained models and training utilities *      peft enables LoRA (low-rank Adaptation), reducing memory usage *        bitsandbytes allows 8-bit precision to shrink model size *        accelerate helps manage GPU memory and multi-GPU setups ...
Read more