mlir-aie

An MLIR-based toolchain for AMD AI Engine-enabled devices.

Stars: 354

Visit

This repository contains an MLIR-based toolchain for AI Engine-enabled devices, such as AMD Ryzen™ AI and Versal™. This repository can be used to generate low-level configurations for the AI Engine portion of these devices. AI Engines are organized as a spatial array of tiles, where each tile contains AI Engine cores and/or memories. The spatial array is connected by stream switches that can be configured to route data between AI Engine tiles scheduled by their programmable Data Movement Accelerators (DMAs). This repository contains MLIR representations, with multiple levels of abstraction, to target AI Engine devices. This enables compilers and developers to program AI Engine cores, as well as describe data movements and array connectivity. A Python API is made available as a convenient interface for generating MLIR design descriptions. Backend code generation is also included, targeting the aie-rt library. This toolchain uses the AI Engine compiler tool which is part of the AMD Vitis™ software installation: these tools require a free license for use from the Product Licensing Site.

README:

MLIR-based AI Engine toolchain

This project is primarily intended to support the open-source community, particularly tool builders, with low-level access to AIE devices and enable the development of a wide variety of programming models from higher level abstractions. We provide an example programming flow: Interface Representation for hands-ON (IRON) close-to-metal programming of the AIE-array. IRON is an open access toolkit enabling performance engineers to build fast and efficient, often specialized designs through a set of Python language bindings around the mlir-aie dialect. As such, it contains some examples, however this project is not intended to represent an end-to-end compilation flow for all application designs. If you're looking for an out-of-the-box experience for highly efficient machine learning, check out the AMD Ryzen™ AI Software Platform.

Getting Started for AMD Ryzen™ AI on Linux

These instructions will guide you through everything required for building and executing a program on the Ryzen™ AI NPU, starting from a fresh bare-bones Ubuntu 24.04 or Ubuntu 24.10 install.

Initial Setup

Be sure you have the latest BIOS on your laptop or mini-PC that enables the NPU. See here.

If starting from Ubuntu 24.04 you may need to update the Linux kernel to 6.11+ by installing the Hardware Enablement (HWE) stack:

sudo apt update 
sudo apt install --install-recommends linux-generic-hwe-24.04
sudo reboot

Prerequisites

BIOS Settings:

Turn off SecureBoot (Allows for unsigned drivers to be installed): BIOS → Security → Secure boot → Disable

Build and install the XDNA™ Driver and XRT

Execute the scripted build process:

This script will install package dependencies, build the xdna-driver and xrt packages, and install them. These steps require sudo access.
```
bash ./utils/build_drivers.sh
```
Reboot as directed after the script exits.
```
sudo reboot
```

Check that the NPU is working if the device appears with xrt-smi:

source /opt/xilinx/xrt/setup.sh
xrt-smi examine

At the bottom of the output you should see:

Devices present
BDF             :  Name             
------------------------------------
[0000:66:00.1]  :  RyzenAI-npu1

Install IRON and MLIR-AIE Prerequisites

Install the following packages needed for MLIR-AIE:

sudo apt install \
build-essential clang clang-14 lld lld-14 cmake python3-venv python3-pip

Install opencv which is needed for some programming examples:
```
sudo apt install libopencv-dev python3-opencv
```

Install IRON for AMD Ryzen™ AI AIE Application Development

Clone the mlir-aie repository:

git clone https://github.com/Xilinx/mlir-aie.git
cd mlir-aie

Install IRON library, mlir-aie and llvm-aie compilers from whls:
```
source utils/quick_setup.sh
```

Build an IRON Design for AIEs in the AMD Ryzen™ AI NPU

For your design of interest, for instance from programming_examples, 2 steps are needed: (i) build the AIE design and then (ii) build the host code.

Build Device AIE Part

Goto the design of interest and run:
```
make
```
Build host code and execute the design:
```
make run
```

Learn more about NPU programming with IRON

Continue to the IRON AIE Application Programming Guide
Additional MLIR-AIE documentation is available on the website
AIE API header library documentation for single-core AIE programming in C++ is avaiable here

Optional: Install AIETools

You may skip the Vitis™ installation step if you intend to only target AMD XDNA™/AIE-ML (AIE2) and AMD XDNA™ 2 (AIE2P) using our open-source single-core compiler Peano. Compiling with xchesscc is not supported without installing AMD Vitis™ AIE Essentials.

Install Vitis™ AIE Essentials from Ryzen AI Software 1.3 Early Accesss. We will assume you use the installation directory, /tools/ryzen_ai-1.3.0/vitis_aie_essentials.

This is an early access lounge, you must register and be granted access at this time.
1. Download VAIML Installer for Linux based compilation: ryzen_ai-1.3.0ea1.tgz
2. Extract the required tools:
```
   tar -xzvf ryzen_ai-1.3.0ea1.tgz
   cd ryzen_ai-1.3.0
   mkdir vitis_aie_essentials
   mv vitis_aie_essentials*.whl vitis_aie_essentials
   cd vitis_aie_essentials
   unzip vitis_aie_essentials*.whl
```
Set up an AI Engine license.
1. Get a local license for AI Engine tools from https://www.xilinx.com/getlicense.
2. Copy your license file (Xilinx.lic) to your preferred location, e.g. /opt/Xilinx.lic:

Setup your environment using the following script for Vitis™ for AIETools:

#!/bin/bash
 #################################################################################
 # Setup Vitis AIE Essentials
 #################################################################################
 export AIETOOLS_ROOT=/tools/ryzen_ai-1.3.0/vitis_aie_essentials
 export PATH=$PATH:${AIETOOLS_ROOT}/bin
 export LM_LICENSE_FILE=/opt/Xilinx.lic

Troubleshooting:

Update BIOS:

Be sure you have the latest BIOS for your laptop or mini PC, this will ensure the NPU (sometimes referred to as IPU) is enabled in the system. You may need to manually enable the NPU: Advanced → CPU Configuration → IPU

NOTE: Some manufacturers only provide Windows executables to update the BIOS, please do this before installing Ubuntu.

Detailed Getting Started Guides and Documentation:

Getting Started on a Versal™ board

Running on a Versal™ board

Getting Started and Running on Windows Ryzen™ AI

Getting Started and Running on Linux Ryzen™ AI

IRON AIE Application Programming Guide

MLIR Dialect and Compiler Documentation

Interested in contributing MLIR-AIE? Information for developers

For Tasks:

Click tags to check more tools for each tasks

generate low-level configurations program ai engine cores describe data movements describe array connectivity generate mlir design descriptions

For Jobs:

ai engineer mlir developer compiler developer data scientist machine learning engineer

Alternative AI tools for mlir-aie

Similar Open Source Tools

mlir-aie

github

: 354

flock

Flock is a workflow-based low-code platform that enables rapid development of chatbots, RAG applications, and coordination of multi-agent teams. It offers a flexible, low-code solution for orchestrating collaborative agents, supporting various node types for specific tasks, such as input processing, text generation, knowledge retrieval, tool execution, intent recognition, answer generation, and more. Flock integrates LangChain and LangGraph to provide offline operation capabilities and supports future nodes like Conditional Branch, File Upload, and Parameter Extraction for creating complex workflows. Inspired by StreetLamb, Lobe-chat, Dify, and fastgpt projects, Flock introduces new features and directions while leveraging open-source models and multi-tenancy support.

github

: 170

supavec

Supavec is an open-source tool that serves as an alternative to Carbon.ai. It allows users to build powerful RAG applications using any data source and at any scale. The tool is designed to provide a simple API endpoint for easy integration and usage. Supavec is built with Next.js, Supabase, Tailwind CSS, Bun, and Upstash, offering a robust and flexible solution for application development. Users can refer to the API documentation for detailed information on how to utilize the tool effectively.

github

: 557

JetStream

JetStream is a throughput and memory optimized engine for LLM inference on XLA devices, starting with TPUs (and GPUs in future -- PRs welcome). It is designed to provide high performance and scalability for large language models, enabling efficient inference on cloud-based TPUs. JetStream leverages XLA to optimize the execution of LLM models, resulting in faster and more efficient inference. Additionally, JetStream supports quantization techniques to further enhance performance and reduce memory consumption. By utilizing JetStream, developers can deploy and run LLM models on TPUs with ease, achieving optimal performance and cost-effectiveness.

github

: 174

morphic

Morphic is an AI-powered answer engine with a generative UI. It utilizes a stack of Next.js, Vercel AI SDK, OpenAI, Tavily AI, shadcn/ui, Radix UI, and Tailwind CSS. To get started, fork and clone the repo, install dependencies, fill out secrets in the .env.local file, and run the app locally using 'bun dev'. You can also deploy your own live version of Morphic with Vercel. Verified models that can be specified to writers include Groq, LLaMA3 8b, and LLaMA3 70b.

github

: 7.2k

OpenAI-Whisper-GUI

OpenAI Whisper GUI is a modern GUI application designed to transcribe and translate audio/video files using OpenAI Whisper. It features a modern UI with light/dark mode, the ability to export transcribed text, add subtitles to videos, and more. The latest version includes updates to widgets, layouts, and themes, as well as new features such as a config handler, GPU info retrieval, a new app logo, settings interface, and bug fixes like code refactoring and fixing Cuda not found warning message. Users can easily install the tool by cloning the GitHub repository and running setup.py and main.py scripts. For more information, users can visit the OpenAI Whisper GitHub repository.

github

: 93

ClaraVerse

ClaraVerse is a privacy-first AI assistant and agent builder that allows users to chat with AI, create intelligent agents, and turn them into fully functional apps. It operates entirely on open-source models running on the user's device, ensuring data privacy and security. With features like AI assistant, image generation, intelligent agent builder, and image gallery, ClaraVerse offers a versatile platform for AI interaction and app development. Users can install ClaraVerse through Docker, native desktop apps, or the web version, with detailed instructions provided for each option. The tool is designed to empower users with control over their AI stack and leverage community-driven innovations for AI development.

github

: 444

JetStream

JetStream is a throughput and memory optimized engine for Large Language Model (LLM) inference on XLA devices, specifically TPUs. It provides reference engine implementations for Jax and Pytorch models, along with documentation for online inference, serving Gemma using TPUs on GKE, benchmarking, observability, profiling, and standalone local setup. Users can easily set up a local server, run tests, and test core modules. JetStream aims to enhance the performance of LLM inference on XLA devices.

github

: 290

ai-collection

github

: 7.8k

Bavarder

Bavarder is an AI-powered chit-chat tool designed for informal conversations about unimportant matters. Users can engage in light-hearted discussions with the AI, simulating casual chit-chat scenarios. The tool provides a platform for users to interact with AI in a fun and entertaining way, offering a unique experience of engaging with artificial intelligence in a conversational manner.

github

: 243

java-sdk

The MCP Java SDK is a set of projects that provide Java SDK integration for the Model Context Protocol. It enables Java applications to interact with AI models and tools through a standardized interface, supporting both synchronous and asynchronous communication patterns.

github

: 805

OmniSteward

OmniSteward is an AI-powered steward system based on large language models that can interact with users through voice or text to help control smart home devices and computer programs. It supports multi-turn dialogue, tool calling for complex tasks, multiple LLM models, voice recognition, smart home control, computer program management, online information retrieval, command line operations, and file management. The system is highly extensible, allowing users to customize and share their own tools.

github

: 66

Apex-Aim-Panel-v2.31.23.4

Apex-Aim-Panel-v2.31.23.4 is a cheat tool designed for Apex Legends that provides features like Aimbot, ESP, and Misc functionalities. Users can safely ignore antivirus triggers and follow the provided instructions to run the software, enabling them to enhance their gameplay experience in Apex Legends.

github

: 98

aituber-server

AITuberKit server-side is a tool that allows users to receive messages via WebSocket and obtain responses from Open Interpreter. Users can also send files to the server for storage and issue commands to Open Interpreter. The tool is designed for WebSocket operation and provides a default connection URL of `ws://127.0.0.1:8000/ws`. It supports debugging in VSCode with DEBUG_MODE=1. The tool is licensed under KillianLucas/open-interpreter and includes a guide on how to use Open Interpreter.

github

: 148

NotHotDog

NotHotDog is an open-source platform for testing, evaluating, and simulating AI agents. It offers a robust framework for generating test cases, running conversational scenarios, and analyzing agent performance.

github

: 55

Easy-Voice-Toolkit

Easy Voice Toolkit is a toolkit based on open source voice projects, providing automated audio tools including speech model training. Users can seamlessly integrate functions like audio processing, voice recognition, voice transcription, dataset creation, model training, and voice conversion to transform raw audio files into ideal speech models. The toolkit supports multiple languages and is currently only compatible with Windows systems. It acknowledges the contributions of various projects and offers local deployment options for both users and developers. Additionally, cloud deployment on Google Colab is available. The toolkit has been tested on Windows OS devices and includes a FAQ section and terms of use for academic exchange purposes.

github

: 641

For similar tasks

mlir-aie

github

: 354

For similar jobs

sweep

Sweep is an AI junior developer that turns bugs and feature requests into code changes. It automatically handles developer experience improvements like adding type hints and improving test coverage.

github

: 7.1k

teams-ai

The Teams AI Library is a software development kit (SDK) that helps developers create bots that can interact with Teams and Microsoft 365 applications. It is built on top of the Bot Framework SDK and simplifies the process of developing bots that interact with Teams' artificial intelligence capabilities. The SDK is available for JavaScript/TypeScript, .NET, and Python.

github

: 502

ai-guide

This guide is dedicated to Large Language Models (LLMs) that you can run on your home computer. It assumes your PC is a lower-end, non-gaming setup.

github

: 159

classifai

Supercharge WordPress Content Workflows and Engagement with Artificial Intelligence. Tap into leading cloud-based services like OpenAI, Microsoft Azure AI, Google Gemini and IBM Watson to augment your WordPress-powered websites. Publish content faster while improving SEO performance and increasing audience engagement. ClassifAI integrates Artificial Intelligence and Machine Learning technologies to lighten your workload and eliminate tedious tasks, giving you more time to create original content that matters.

github

: 620

chatbot-ui

Chatbot UI is an open-source AI chat app that allows users to create and deploy their own AI chatbots. It is easy to use and can be customized to fit any need. Chatbot UI is perfect for businesses, developers, and anyone who wants to create a chatbot.

github

: 27.7k

BricksLLM

BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM: * Set LLM usage limits for users on different pricing tiers * Track LLM usage on a per user and per organization basis * Block or redact requests containing PIIs * Improve LLM reliability with failovers, retries and caching * Distribute API keys with rate limits and cost limits for internal development/production use cases * Distribute API keys with rate limits and cost limits for students

github

: 953

uAgents

uAgents is a Python library developed by Fetch.ai that allows for the creation of autonomous AI agents. These agents can perform various tasks on a schedule or take action on various events. uAgents are easy to create and manage, and they are connected to a fast-growing network of other uAgents. They are also secure, with cryptographically secured messages and wallets.

github

: 1.3k

griptape

Griptape is a modular Python framework for building AI-powered applications that securely connect to your enterprise data and APIs. It offers developers the ability to maintain control and flexibility at every step. Griptape's core components include Structures (Agents, Pipelines, and Workflows), Tasks, Tools, Memory (Conversation Memory, Task Memory, and Meta Memory), Drivers (Prompt and Embedding Drivers, Vector Store Drivers, Image Generation Drivers, Image Query Drivers, SQL Drivers, Web Scraper Drivers, and Conversation Memory Drivers), Engines (Query Engines, Extraction Engines, Summary Engines, Image Generation Engines, and Image Query Engines), and additional components (Rulesets, Loaders, Artifacts, Chunkers, and Tokenizers). Griptape enables developers to create AI-powered applications with ease and efficiency.

github

: 2.2k