AI-Toolbox

A C++ framework for MDPs and POMDPs with Python bindings

Stars: 657

Visit

AI-Toolbox is a C++ library aimed at representing and solving common AI problems, with a focus on MDPs, POMDPs, and related algorithms. It provides an easy-to-use interface that is extensible to many problems while maintaining readable code. The toolbox includes tutorials for beginners in reinforcement learning and offers Python bindings for seamless integration. It features utilities for combinatorics, polytopes, linear programming, sampling, distributions, statistics, belief updating, data structures, logging, seeding, and more. Additionally, it supports bandit/normal games, single agent MDP/stochastic games, single agent POMDP, and factored/joint multi-agent scenarios.

README:

AI-Toolbox

This C++ toolbox is aimed at representing and solving common AI problems, implementing an easy-to-use interface which should be hopefully extensible to many problems, while keeping code readable.

Current development includes MDPs, POMDPs and related algorithms. This toolbox was originally developed taking inspiration from the Matlab MDPToolbox, which you can find here, and from the pomdp-solve software written by A. R. Cassandra, which you can find here.

If you are new to the field of reinforcement learning, we have a few simple tutorials that can help you get started. An excellent, more in depth introduction to the basics of reinforcement learning can be found freely online in this book.

If you use this toolbox for research, please consider citing our JMLR article:

@article{JMLR:v21:18-402,
  author  = {Eugenio Bargiacchi and Diederik M. Roijers and Ann Now\'{e}},
  title   = {AI-Toolbox: A C++ library for Reinforcement Learning and Planning (with Python Bindings)},
  journal = {Journal of Machine Learning Research},
  year    = {2020},
  volume  = {21},
  number  = {102},
  pages   = {1-12},
  url     = {http://jmlr.org/papers/v21/18-402.html}
}

Example

// The model can be any custom class that respects a 10-method interface.
auto model = makeTigerProblem();
unsigned horizon = 10; // The horizon of the solution.

// The 0.0 is the convergence parameter. It gives a way to stop the
// computation if the policy has converged before the horizon.
AIToolbox::POMDP::IncrementalPruning solver(horizon, 0.0);

// Solve the model and obtain the optimal value function.
auto [bound, valueFunction] = solver(model);

// We create a policy from the solution to compute the agent's actions.
// The parameters are the size of the model (SxAxO), and the value function.
AIToolbox::POMDP::Policy policy(2, 3, 2, valueFunction);

// We begin a simulation with a uniform belief. We sample from the belief
// in order to get a "real" state for the world, since this code has to
// both emulate the environment and control the agent.
AIToolbox::POMDP::Belief b(2); b << 0.5, 0.5;
auto s = AIToolbox::sampleProbability(b.size(), b, rand);

// We sample the first action. The id is to follow the policy tree later.
auto [a, id] = policy.sampleAction(b, horizon);

double totalReward = 0.0;// As an example, we store the overall reward.
for (int t = horizon - 1; t >= 0; --t) {
    // We advance the world one step.
    auto [s1, o, r] = model.sampleSOR(s, a);
    totalReward += r;

    // We select our next action from the observation we got.
    std::tie(a, id) = policy.sampleAction(id, o, t);

    s = s1; // Finally we update the world for the next timestep.
}

Documentation

The latest documentation is available here. We have a few tutorials that can help you get started with the toolbox. The tutorials are in C++, but the examples folder contains equivalent Python code which you can follow along just as well.

For Python docs you can find them by typing help(AIToolbox) from the interpreter. It should show the exported API for each class, along with any differences in input/output.

Features

Cassandra POMDP Format Parsing

Cassandra's POMDP format is a type of text file that contains a definition of an MDP or POMDP model. You can find some examples here. While it is absolutely not necessary to use this format, and you can define models via code, we do parse a reasonable subset of Cassandra's POMDP format, which allows to reuse already defined problems with this library. Here's the docs on that.

Python 2 and 3 Bindings!

The user interface of the library is pretty much the same with Python than what you would get by using simply C++. See the examples folder to see just how much Python and C++ code resemble each other. Since Python does not allow templates, the classes are binded with as many instantiations as possible.

Additionally, the library allows the usage of native Python generative models (where you don't need to specify the transition and reward functions, you only sample next state and reward). This allows for example to directly use OpenAI gym environments with minimal code writing.

That said, if you need to customize a specific implementation to make it perform better on your specific use-cases, or if you want to try something completely new, you will have to use C++.

Utilities

The library has an extensive set of utilities which would be too long to enumerate here. In particular, we have utilities for combinatorics, polytopes, linear programming, sampling and distributions, automated statistics, belief updating, many data structures, logging, seeding and much more.

Bandit/Normal Games:

	Models
Basic Model
	Policies
Exploring Selfish Reinforcement Learning (ESRL)	Q-Greedy Policy	Softmax Policy
Linear Reward Penalty	Thompson Sampling (Student-t distribution)	Random Policy
Top-Two Thompson Sampling (Student-t distribution)	Successive Rejects	T3C (Normal distribution)

Single Agent MDP/Stochastic Games:

	Models
Basic Model	Sparse Model	Maximum Likelihood Model
Sparse Maximum Likelihood Model	Thompson Model (Dirichlet + Student-t distributions)
	Algorithms
Dyna-Q	Dyna2	Expected SARSA
Hysteretic Q-Learning	Importance Sampling	Linear Programming
Monte Carlo Tree Search (MCTS)	Policy Evaluation	Policy Iteration
Prioritized Sweeping	Q-Learning	Double Q-Learning
Q(λ)	R-Learning	SARSA(λ)
SARSA	Retrace(λ)	Tree Backup(λ)
Value Iteration
	Policies
Basic Policy	Epsilon-Greedy Policy	Softmax Policy
Q-Greedy Policy	PGA-APP	Win or Learn Fast Policy Iteration (WoLF)

Single Agent POMDP:

	Models
Basic Model	Sparse Model
	Algorithms
Augmented MDP (AMDP)	Blind Strategies	Fast Informed Bound
GapMin	Incremental Pruning	Linear Support
PERSEUS	POMCP with UCB1	Point Based Value Iteration (PBVI)
QMDP	Real-Time Belief State Search (RTBSS)	SARSOP
Witness	rPOMCP
	Policies
Basic Policy

Factored/Joint Multi-Agent:

Bandits:

Not in Python yet.

	Models
Basic Model	Flattened Model
	Algorithms
Max-Plus	Multi-Objective Variable Elimination (MOVE)	Upper Confidence Variable Elimination (UCVE)
Variable Elimination	Local Search	Reusing Iterative Local Search
	Policies
Q-Greedy Policy	Random Policy	Learning with Linear Rewards (LLR)
Multi-Agent Upper Confidence Exploration (MAUCE)	Multi-Agent Thompson-Sampling (Student-t distribution)	Multi-Agent RMax (MARMax)
Single-Action Policy

MDP:

Not in Python yet.

	Models
Cooperative Basic Model	Cooperative Maximum Likelihood Model	Cooperative Thompson Model (Dirichlet + Student-t distributions)
	Algorithms
FactoredLP	Multi Agent Linear Programming	Joint Action Learners
Sparse Cooperative Q-Learning	Cooperative Prioritized Sweeping
	Policies
All Bandit Policies	Epsilon-Greedy Policy	Q-Greedy Policy

Build Instructions

Dependencies

To build the library you need:

cmake >= 3.12
the boost library >= 1.67
the Eigen 3.4 library.
the lp_solve library (a shared library must be available to compile the Python wrapper).

In addition, C++20 support is now required (this means at least g++-10)

On a Ubuntu system, you can install these dependencies with the following command:

sudo apt install g++-10 cmake libboost1.71-all-dev liblpsolve55-dev lp-solve libeigen3-dev

Building

Once you have all required dependencies, you can simply execute the following commands from the project's main folder:

mkdir build
cd build/
cmake ..
make

cmake can be called with a series of flags in order to customize the output, if building everything is not desirable. The following flags are available:

CMAKE_BUILD_TYPE   # Defines the build type
MAKE_ALL           # Builds all there is to build in the project, but Python.
MAKE_LIB           # Builds the whole core C++ libraries (MDP, POMDP, etc..)
MAKE_MDP           # Builds only the core C++ MDP library
MAKE_FMDP          # Builds only the core C++ Factored/Multi-Agent and MDP libraries
MAKE_POMDP         # Builds only the core C++ POMDP and MDP libraries
MAKE_TESTS         # Builds the library's tests for the compiled core libraries
MAKE_EXAMPLES      # Builds the library's examples using the compiled core libraries
MAKE_PYTHON        # Builds Python bindings for the compiled core libraries
AI_PYTHON_VERSION  # Selects the Python version you want (2 or 3). If not
                   #   specified, we try to guess based on your default interpreter.
AI_LOGGING_ENABLED # Whether the library logging code is enabled at runtime.

These flags can be combined as needed. For example:

# Will build MDP and MDP Python 3 bindings
cmake -DCMAKE_BUILD_TYPE=Debug -DMAKE_MDP=1 -DMAKE_PYTHON=1 -DAI_PYTHON_VERSION=3 ..

The default flags when nothing is specified are MAKE_ALL and CMAKE_BUILD_TYPE=Release.

Note that by default MAKE_ALL does not build the Python bindings, as they have a minor performance hit on the C++ static libraries. You can easily enable them by using the flag MAKE_PYTHON.

The static library files will be available directly in the build directory. Three separate libraries are built: AIToolboxMDP, AIToolboxPOMDP and AIToolboxFMDP. In case you want to link against either the POMDP library or the Factored MDP library, you will also need to link against the MDP one, since both of them use MDP functionality.

A number of small tests are included which you can find in the test/ folder. You can execute them after building the project using the following command directly from the build directory, just after you finish make:

ctest

The tests also offer a brief introduction for the framework, waiting for a more complete descriptive write-up. Only the tests for the parts of the library that you compiled are going to be built.

To compile the library's documentation you need Doxygen. To use it it is sufficient to execute the following command from the project's root folder:

doxygen

After that the documentation will be generated into an html folder in the main directory.

Compiling a Program

For an extensive pre-made setup of a C++/CMake project using AI-Toolbox on Linux, please do checkout this repository. It contains the setup I personally use when working with AI-Toolbox. It also comes with many additional tools you might need, which are nevertheless all optional.

Alternatively, to compile a program that uses this library, simply link it against the compiled libraries you need, and possibly to the lp_solve libraries (if using POMDP or FMDP).

Please note that since both POMDP and FMDP libraries rely on the MDP code, you MUST specify those libraries before the MDP library when linking, otherwise it may result in undefined reference errors. The POMDP and Factored MDP libraries are not currently dependent on each other so their order does not matter.

For Python, you just need to import the AIToolbox.so module, and you'll be able to use the classes as exported to Python. All classes are documented, and you can run in the Python CLI

help(AIToolbox.MDP)
help(AIToolbox.POMDP)

to see the documentation for each specific class.

For Tasks:

Click tags to check more tools for each tasks

solve mdps simulate agent actions parse cassandra's pomdp format build python generative models customize ai implementations

For Jobs:

ai researcher machine learning engineer data scientist software developer research scientist

Alternative AI tools for AI-Toolbox

Similar Open Source Tools

AI-Toolbox

github

: 657

TableLLM

TableLLM is a large language model designed for efficient tabular data manipulation tasks in real office scenarios. It can generate code solutions or direct text answers for tasks like insert, delete, update, query, merge, and chart operations on tables embedded in spreadsheets or documents. The model has been fine-tuned based on CodeLlama-7B and 13B, offering two scales: TableLLM-7B and TableLLM-13B. Evaluation results show its performance on benchmarks like WikiSQL, Spider, and self-created table operation benchmark. Users can use TableLLM for code and text generation tasks on tabular data.

github

: 77

cambrian

Cambrian-1 is a fully open project focused on exploring multimodal Large Language Models (LLMs) with a vision-centric approach. It offers competitive performance across various benchmarks with models at different parameter levels. The project includes training configurations, model weights, instruction tuning data, and evaluation details. Users can interact with Cambrian-1 through a Gradio web interface for inference. The project is inspired by LLaVA and incorporates contributions from Vicuna, LLaMA, and Yi. Cambrian-1 is licensed under Apache 2.0 and utilizes datasets and checkpoints subject to their respective original licenses.

github

: 1.4k

last_layer

last_layer is a security library designed to protect LLM applications from prompt injection attacks, jailbreaks, and exploits. It acts as a robust filtering layer to scrutinize prompts before they are processed by LLMs, ensuring that only safe and appropriate content is allowed through. The tool offers ultra-fast scanning with low latency, privacy-focused operation without tracking or network calls, compatibility with serverless platforms, advanced threat detection mechanisms, and regular updates to adapt to evolving security challenges. It significantly reduces the risk of prompt-based attacks and exploits but cannot guarantee complete protection against all possible threats.

github

: 79

llm-python

A set of instructional materials, code samples and Python scripts featuring LLMs (GPT etc) through interfaces like llamaindex, langchain, Chroma (Chromadb), Pinecone etc. Mainly used to store reference code for my LangChain tutorials on YouTube.

github

: 673

floneum

Floneum is a graph editor that makes it easy to develop your own AI workflows. It uses large language models (LLMs) to run AI models locally, without any external dependencies or even a GPU. This makes it easy to use LLMs with your own data, without worrying about privacy. Floneum also has a plugin system that allows you to improve the performance of LLMs and make them work better for your specific use case. Plugins can be used in any language that supports web assembly, and they can control the output of LLMs with a process similar to JSONformer or guidance.

github

: 1.8k

flowgen

FlowGen is a tool built for AutoGen, a great agent framework from Microsoft and a lot of contributors. It provides intuitive visual tools that streamline the construction and oversight of complex agent-based workflows, simplifying the process for creators and developers. Users can create Autoflows, chat with agents, and share flow templates. The tool is fully dockerized and supports deployment on Railway.app. Contributions to the project are welcome, and the platform uses semantic-release for versioning and releases.

github

: 123

holmesgpt

HolmesGPT is an open-source DevOps assistant powered by OpenAI or any tool-calling LLM of your choice. It helps in troubleshooting Kubernetes, incident response, ticket management, automated investigation, and runbook automation in plain English. The tool connects to existing observability data, is compliance-friendly, provides transparent results, supports extensible data sources, runbook automation, and integrates with existing workflows. Users can install HolmesGPT using Brew, prebuilt Docker container, Python Poetry, or Docker. The tool requires an API key for functioning and supports OpenAI, Azure AI, and self-hosted LLMs.

github

: 732

llm-graph-builder

Knowledge Graph Builder App is a tool designed to convert PDF documents into a structured knowledge graph stored in Neo4j. It utilizes OpenAI's GPT/Diffbot LLM to extract nodes, relationships, and properties from PDF text content. Users can upload files from local machine or S3 bucket, choose LLM model, and create a knowledge graph. The app integrates with Neo4j for easy visualization and querying of extracted information.

github

: 3.2k

OmAgent

OmAgent is an open-source agent framework designed to streamline the development of on-device multimodal agents. It enables agents to empower various hardware devices, integrates speed-optimized SOTA multimodal models, provides SOTA multimodal agent algorithms, and focuses on optimizing the end-to-end computing pipeline for real-time user interaction experience. Key features include easy connection to diverse devices, scalability, flexibility, and workflow orchestration. The architecture emphasizes graph-based workflow orchestration, native multimodality, and device-centricity, allowing developers to create bespoke intelligent agent programs.

github

: 1.3k

all-rag-techniques

This repository provides a hands-on approach to Retrieval-Augmented Generation (RAG) techniques, simplifying advanced concepts into understandable implementations using Python libraries like openai, numpy, and matplotlib. It offers a collection of Jupyter Notebooks with concise explanations, step-by-step implementations, code examples, evaluations, and visualizations for various RAG techniques. The goal is to make RAG more accessible and demystify its workings for educational purposes.

github

: 504

factorio-learning-environment

Factorio Learning Environment is an open source framework designed for developing and evaluating LLM agents in the game of Factorio. It provides two settings: Lab-play with structured tasks and Open-play for building large factories. Results show limitations in spatial reasoning and automation strategies. Agents interact with the environment through code synthesis, observation, action, and feedback. Tools are provided for game actions and state representation. Agents operate in episodes with observation, planning, and action execution. Tasks specify agent goals and are implemented in JSON files. The project structure includes directories for agents, environment, cluster, data, docs, eval, and more. A database is used for checkpointing agent steps. Benchmarks show performance metrics for different configurations.

github

: 525

FFAIVideo

FFAIVideo is a lightweight node.js project that utilizes popular AI LLM to intelligently generate short videos. It supports multiple AI LLM models such as OpenAI, Moonshot, Azure, g4f, Google Gemini, etc. Users can input text to automatically synthesize exciting video content with subtitles, background music, and customizable settings. The project integrates Microsoft Edge's online text-to-speech service for voice options and uses Pexels website for video resources. Installation of FFmpeg is essential for smooth operation. Inspired by MoneyPrinterTurbo, MoneyPrinter, and MsEdgeTTS, FFAIVideo is designed for front-end developers with minimal dependencies and simple usage.

github

: 55

PlanExe

PlanExe is a planning AI tool that helps users generate detailed plans based on vague descriptions. It offers a Gradio-based web interface for easy input and output. Users can choose between running models in the cloud or locally on a high-end computer. The tool aims to provide a straightforward path to planning various tasks efficiently.

github

: 95

AIOS

AIOS, a Large Language Model (LLM) Agent operating system, embeds large language model into Operating Systems (OS) as the brain of the OS, enabling an operating system "with soul" -- an important step towards AGI. AIOS is designed to optimize resource allocation, facilitate context switch across agents, enable concurrent execution of agents, provide tool service for agents, maintain access control for agents, and provide a rich set of toolkits for LLM Agent developers.

github

: 3.9k

supallm

Supallm is a Python library for super resolution of images using deep learning techniques. It provides pre-trained models for enhancing image quality by increasing resolution. The library is easy to use and allows users to upscale images with high fidelity and detail. Supallm is suitable for tasks such as enhancing image quality, improving visual appearance, and increasing the resolution of low-quality images. It is a valuable tool for researchers, photographers, graphic designers, and anyone looking to enhance image quality using AI technology.

github

: 110

For similar tasks

AI-Toolbox

github

: 657

For similar jobs

weave

Weave is a toolkit for developing Generative AI applications, built by Weights & Biases. With Weave, you can log and debug language model inputs, outputs, and traces; build rigorous, apples-to-apples evaluations for language model use cases; and organize all the information generated across the LLM workflow, from experimentation to evaluations to production. Weave aims to bring rigor, best-practices, and composability to the inherently experimental process of developing Generative AI software, without introducing cognitive overhead.

github

: 847

LLMStack

LLMStack is a no-code platform for building generative AI agents, workflows, and chatbots. It allows users to connect their own data, internal tools, and GPT-powered models without any coding experience. LLMStack can be deployed to the cloud or on-premise and can be accessed via HTTP API or triggered from Slack or Discord.

github

: 1.5k

VisionCraft

The VisionCraft API is a free API for using over 100 different AI models. From images to sound.

github

: 94

kaito

Kaito is an operator that automates the AI/ML inference model deployment in a Kubernetes cluster. It manages large model files using container images, avoids tuning deployment parameters to fit GPU hardware by providing preset configurations, auto-provisions GPU nodes based on model requirements, and hosts large model images in the public Microsoft Container Registry (MCR) if the license allows. Using Kaito, the workflow of onboarding large AI inference models in Kubernetes is largely simplified.

github

: 405

PyRIT

PyRIT is an open access automation framework designed to empower security professionals and ML engineers to red team foundation models and their applications. It automates AI Red Teaming tasks to allow operators to focus on more complicated and time-consuming tasks and can also identify security harms such as misuse (e.g., malware generation, jailbreaking), and privacy harms (e.g., identity theft). The goal is to allow researchers to have a baseline of how well their model and entire inference pipeline is doing against different harm categories and to be able to compare that baseline to future iterations of their model. This allows them to have empirical data on how well their model is doing today, and detect any degradation of performance based on future improvements.

github

: 2.3k

tabby

Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features: * Self-contained, with no need for a DBMS or cloud service. * OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE). * Supports consumer-grade GPUs.

github

: 30.5k

spear

SPEAR (Simulator for Photorealistic Embodied AI Research) is a powerful tool for training embodied agents. It features 300 unique virtual indoor environments with 2,566 unique rooms and 17,234 unique objects that can be manipulated individually. Each environment is designed by a professional artist and features detailed geometry, photorealistic materials, and a unique floor plan and object layout. SPEAR is implemented as Unreal Engine assets and provides an OpenAI Gym interface for interacting with the environments via Python.

github

: 224

Magick

Magick is a groundbreaking visual AIDE (Artificial Intelligence Development Environment) for no-code data pipelines and multimodal agents. Magick can connect to other services and comes with nodes and templates well-suited for intelligent agents, chatbots, complex reasoning systems and realistic characters.

github

: 675