rai

rai

RAI is a multi-vendor agent framework for robotics, utilizing Langchain and ROS 2 tools to perform complex actions, defined scenarios, free interface execution, log summaries, voice interaction and more.

Stars: 117

Visit
 screenshot

RAI is a framework designed to bring general multi-agent system capabilities to robots, enhancing human interactivity, flexibility in problem-solving, and out-of-the-box AI features. It supports multi-modalities, incorporates an advanced database for agent memory, provides ROS 2-oriented tooling, and offers a comprehensive task/mission orchestrator. The framework includes features such as voice interaction, customizable robot identity, camera sensor access, reasoning through ROS logs, and integration with LangChain for AI tools. RAI aims to support various AI vendors, improve human-robot interaction, provide an SDK for developers, and offer a user interface for configuration.

README:

RAI

[!IMPORTANT]
RAI is in beta phase now, expect friction. Early contributors are the most welcome!
RAI is developing fast towards a glorious release in time for ROSCon 2024.

RAI is a flexible AI agent framework to develop and deploy Gen AI features for your robots.


rai-image


License GitHub Release Contributors

Static Badge Static Badge Static Badge Static Badge Static Badge Static Badge


Overview

The RAI framework aims to:

  • Supply a general multi-agent system, bringing Gen AI features to your robots.
  • Add human interactivity, flexibility in problem-solving, and out-of-box AI features to existing robot stacks.
  • Provide first-class support for multi-modalities, enabling interaction with various data types.
  • Incorporate an advanced database for persistent agent memory.
  • Include ROS 2-oriented tooling for agents.
  • Support a comprehensive task/mission orchestrator.

Table of Contents

Features

  • [x] Voice interaction (both ways).
  • [x] Customizable robot identity, including constitution (ethical code) and documentation (understanding own capabilities).
  • [x] Accessing camera ("What do you see?") sensor, utilizing VLMs.
  • [x] Reasoning about its own state through ROS logs.
  • [x] ROS 2 action calling and other interfaces. The Agent can dynamically list interfaces, check their message type, and publish.
  • [x] Integration with LangChain to abstract vendors and access convenient AI tools.
  • [x] Tasks in natural language to nav2 goals.
  • [x] NoMaD integration.
  • [x] Tracing.
  • [ ] Grounded SAM 2 integration.
  • [ ] Improved Human-Robot Interaction with voice and text.
  • [ ] SDK for RAI developers.
  • [ ] Support for at least 3 different AI vendors.
  • [ ] Additional tooling such as GroundingDino.
  • [ ] UI for configuration to select features and tools relevant for your deployment.

Setup

1. Setting up the workspace:

1.1 Install poetry

RAI uses Poetry for python packaging and dependency management. Install poetry (1.8+) with the following line:

curl -sSL https://install.python-poetry.org | python3 -

Alternatively, you can opt to do so by following the official docs.

1.2 Clone the repository:

git clone https://github.com/RobotecAI/rai.git
cd rai

1.3 Create poetry virtual environment and install dependencies:

poetry install
rosdep install --from-paths src --ignore-src -r -y

2. Build the project:

2.1 Build RAI workspace

colcon build --symlink-install

2.2 Activate a virtual environment:

source ./setup_shell.sh

3. Setting up vendors

RAI is vendor-agnostic. Use the configuration in config.toml to set up your vendor of choice for RAI modules. Vendor choices for RAI and our recommendations are summarized in Vendors Overview.

We strongly recommend you to use of best-performing AI models to get the most out of RAI!

Pick your local solution or service provider and follow one of these guides:

Running RAI

You are now ready to run RAI!

rosbot-xl-example

You can start by running the following examples:

  1. Hello RAI: Interact directly with your ROS 2 environment through an intuitive Streamlit chat interface.
  2. O3DE Husarion ROSbot XL demo" give tasks to a simulated robot using natural language.

Hello RAI

Chat seamlessly with your ROS 2 environment, retrieve images from cameras, adjust parameters, and get information about your ROS interfaces.

streamlit run src/rai_hmi/rai_hmi/text_hmi.py

Remember to run this command in a sourced shell.

O3DE Rosbot XL Demo

This demo provides a practical way to interact with and control a virtual Husarion ROSbot XL within a simulated environment. Using natural language commands, you can assign tasks to the robot, allowing it to perform a variety of actions.

Given that this is a beta release, consider this demo as an opportunity to explore the framework's capabilities, provide feedback, and contribute. Try different commands, see how the robot responds, and use this experience to understand the potential and limitations of the system.

Follow this guide: husarion-rosbot-xl-demo

What's next?

Once you know your way around RAI, try the following challenges, with the aid the developer guide:

  • Run RAI on your own robot and talk to it, asking questions about what is in its documentation (and others!).
  • Implement additional tools and use them in your interaction.
  • Try a complex, multi-step task for your robot, such as going to several points to perform observations!

Soon you will have an opportunity to work with new RAI demos across several domains.

Planned demos

Application Robot Description Link
Mission and obstacle reasoning in orchards Autonomous tractor In a beautiful scene of a virtual orchard, RAI goes beyond obstacle detection to analyze best course of action for a given unexpected situation. 🌾 demo
Manipulation tasks with natural language Robot Arm (Franka Panda) Complete flexible manipulation tasks thanks to RAI and Grounded SAM 2 🦾 demo
Quadruped inspection demo A robot dog (ANYbotics ANYmal) Perform inspection in a warehouse environment, find and report anomalies link TBD

Community

RAI Q&A

Please take a look at Q&A.

Developer Resources

See our Developer Guide for a deeper dive into RAI, including instructions on creating a configuration specifically for your robot.

Contributing

You are welcome to contribute to RAI! Please see our Contribution Guide.

RAI release and talk

RAI will be released on October 15th, right before ROSCon 2024. If you are going to the conference, come join us at RAI talk on October 23rd.

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for rai

Similar Open Source Tools

For similar tasks

For similar jobs