ai-sdk-cpp

The AI Toolkit for Modern C++. From the engineers at ClickHouse, ai-sdk-cpp is a free, open‑source library for building AI‑powered applications and agents.

Stars: 94

Visit

The AI SDK CPP is a modern C++ toolkit that provides a unified, easy-to-use API for building AI-powered applications with popular model providers like OpenAI and Anthropic. It bridges the gap for C++ developers by offering a clean, expressive codebase with minimal dependencies. The toolkit supports text generation, streaming content, multi-turn conversations, error handling, tool calling, async tool execution, and configurable retries. Future updates will include additional providers, text embeddings, and image generation models. The project also includes a patched version of nlohmann/json for improved thread safety and consistent behavior in multi-threaded environments.

README:

AI SDK CPP

The AI SDK CPP is a modern C++ toolkit designed to help you build AI-powered applications with popular model providers like OpenAI and Anthropic. It provides a unified, easy-to-use API that abstracts away the complexity of different provider implementations.

Motivation

C++ developers have long lacked a first-class, convenient way to interact with modern AI services like OpenAI, Anthropic, and others. AI SDK CPP bridges this gap by providing:

Unified API: Work with multiple AI providers through a single, consistent interface
Modern C++: Built with C++20 features for clean, expressive code
Minimal Dependencies: Minimal external dependencies for easy integration

Installation

You will need a C++20 compatible compiler and CMake 3.16+ installed on your development machine.

Usage

Core API

The AI SDK CPP Core module provides a unified API to interact with model providers like OpenAI and Anthropic.

OpenAI Integration

#include <ai/openai.h>
#include <ai/generate.h>
#include <iostream>

int main() {
    // Ensure OPENAI_API_KEY environment variable is set
    auto client = ai::openai::create_client();
    
    auto result = client.generate_text({
        .model = ai::openai::models::kGpt4o, // this can also be a string like "gpt-4o"
        .system = "You are a friendly assistant!",
        .prompt = "Why is the sky blue?"
    });
    
    if (result) {
        std::cout << result->text << std::endl;
    }
    
    return 0;
}

Anthropic Integration

#include <ai/anthropic.h>
#include <ai/generate.h>
#include <iostream>

int main() {
    // Ensure ANTHROPIC_API_KEY environment variable is set
    auto client = ai::anthropic::create_client();
    auto result = client.generate_text({
        .model = ai::anthropic::models::kClaude35Sonnet,
        .system = "You are a helpful assistant.",
        .prompt = "Explain quantum computing in simple terms."
    });
    
    if (result) {
        std::cout << result->text << std::endl;
    }
    
    return 0;
}

Streaming Responses

#include <ai/openai.h>
#include <ai/stream.h>
#include <iostream>

int main() {
    auto client = ai::openai::create_client();
    
    auto stream = client.stream_text({
        .model = ai::openai::models::kGpt4o, // this can also be a string like "gpt-4o"
        .system = "You are a helpful assistant.",
        .prompt = "Write a short story about a robot."
    });
    
    for (const auto& chunk : stream) {
        if (chunk.text) {
            std::cout << chunk.text.value() << std::flush;
        }
    }
    
    return 0;
}

Multi-turn Conversations

#include <ai/openai.h>
#include <ai/generate.h>
#include <iostream>

int main() {
    auto client = ai::openai::create_client();
    
    ai::Messages messages = {
        {"system", "You are a helpful math tutor."},
        {"user", "What is 2 + 2?"},
        {"assistant", "2 + 2 equals 4."},
        {"user", "Now what is 4 + 4?"}
    };
    
    auto result = client.generate_text({
        .model = ai::openai::models::kGpt4o, // this can also be a string like "gpt-4o"
        .messages = messages
    });
    
    if (result) {
        std::cout << result->text << std::endl;
    }
    
    return 0;
}

Tool Calling

The AI SDK CPP supports function calling, allowing models to interact with external systems and APIs.

#include <ai/openai.h>
#include <ai/generate.h>
#include <ai/tools.h>
#include <iostream>

// Define a tool function
ai::JsonValue get_weather(const ai::JsonValue& args, const ai::ToolExecutionContext& context) {
    std::string location = args["location"].get<std::string>();
    
    // Your weather API logic here
    return ai::JsonValue{
        {"location", location},
        {"temperature", 72},
        {"condition", "Sunny"}
    };
}

int main() {
    auto client = ai::openai::create_client();
    
    // Create tools
    ai::ToolSet tools = {
        {"weather", ai::create_simple_tool(
            "weather",
            "Get current weather for a location", 
            {{"location", "string"}},
            get_weather
        )}
    };
    
    auto result = client.generate_text({
        .model = ai::openai::models::kGpt4o,
        .prompt = "What's the weather like in San Francisco?",
        .tools = tools,
        .max_steps = 3  // Enable multi-step tool calling
    });
    
    if (result) {
        std::cout << result->text << std::endl;
        
        // Inspect tool calls and results
        for (const auto& call : result->tool_calls) {
            std::cout << "Tool: " << call.tool_name 
                      << ", Args: " << call.arguments.dump() << std::endl;
        }
    }
    
    return 0;
}

Async Tool Calling

For long-running operations, you can define asynchronous tools:

#include <future>
#include <thread>
#include <chrono>

// Async tool that returns a future
std::future<ai::JsonValue> fetch_data_async(const ai::JsonValue& args, const ai::ToolExecutionContext& context) {
    return std::async(std::launch::async, [args]() {
        // Simulate async operation
        std::this_thread::sleep_for(std::chrono::seconds(1));
        
        return ai::JsonValue{
            {"data", "Fetched from API"},
            {"timestamp", std::time(nullptr)}
        };
    });
}

int main() {
    auto client = ai::openai::create_client();
    
    ai::ToolSet tools = {
        {"fetch_data", ai::create_simple_async_tool(
            "fetch_data",
            "Fetch data from external API",
            {{"endpoint", "string"}},
            fetch_data_async
        )}
    };
    
    // Multiple async tools will execute in parallel
    auto result = client.generate_text({
        .model = ai::openai::models::kGpt4o,
        .prompt = "Fetch data from the user and product APIs",
        .tools = tools
    });
    
    return 0;
}

Custom Retry Configuration

Configure retry behavior for handling transient failures:

#include <ai/openai.h>
#include <ai/retry/retry_policy.h>

int main() {
    // Configure custom retry behavior
    ai::retry::RetryConfig retry_config;
    retry_config.max_retries = 5;        // More retries for unreliable networks
    retry_config.initial_delay = std::chrono::milliseconds(1000);
    retry_config.backoff_factor = 1.5;   // Gentler backoff
    
    // Create client with custom retry configuration
    auto client = ai::openai::create_client(
        "your-api-key",
        "https://api.openai.com",
        retry_config
    );
    
    // The client will automatically retry on transient failures:
    // - Network errors
    // - HTTP 408, 409, 429 (rate limits), and 5xx errors
    auto result = client.generate_text({
        .model = ai::openai::models::kGpt4o,
        .prompt = "Hello, world!"
    });
    
    return 0;
}

Using OpenAI-Compatible APIs (OpenRouter, etc.)

The OpenAI client can be used with any OpenAI-compatible API by specifying a custom base URL. This allows you to use alternative providers like OpenRouter, which offers access to multiple models through a unified API.

#include <ai/openai.h>
#include <ai/generate.h>
#include <iostream>
#include <cstdlib>

int main() {
    // Get API key from environment variable
    const char* api_key = std::getenv("OPENROUTER_API_KEY");
    if (!api_key) {
        std::cerr << "Please set OPENROUTER_API_KEY environment variable\n";
        return 1;
    }
    
    // Create client with OpenRouter's base URL
    auto client = ai::openai::create_client(
        api_key,
        "https://openrouter.ai/api"  // OpenRouter's OpenAI-compatible endpoint
    );
    
    // Use any model available on OpenRouter
    auto result = client.generate_text({
        .model = "anthropic/claude-3.5-sonnet",  // or "meta-llama/llama-3.1-8b-instruct", etc.
        .system = "You are a helpful assistant.",
        .prompt = "What are the benefits of using OpenRouter?"
    });
    
    if (result) {
        std::cout << result->text << std::endl;
    }
    
    return 0;
}

This approach works with any OpenAI-compatible API provider. Simply provide:

Your provider's API key
The provider's base URL endpoint
Model names as specified by your provider

See the OpenRouter example for a complete demonstration.

Features

Currently Supported

✅ Text Generation: Generate text completions with OpenAI and Anthropic models
✅ Streaming: Real-time streaming of generated content
✅ Multi-turn Conversations: Support for conversation history
✅ Error Handling: Comprehensive error handling with optional types

Recently Added

✅ Tool Calling: Function calling and tool integration with multi-step support
✅ Async Tools: Asynchronous tool execution with parallel processing
✅ Configurable Retries: Customizable retry behavior with exponential backoff

Coming Soon

🚧 Additional Providers: Google, Cohere, and other providers
🚧 Embeddings: Text embedding support
🚧 Image Generation: Support for image generation models

Examples

Check out our examples directory for more comprehensive usage examples:

Basic Chat Application
Streaming Chat
Multi-provider Comparison
Error Handling
Retry Configuration
Basic Tool Calling
Multi-Step Tool Workflows
Async Tool Execution
OpenRouter Integration - Using OpenAI-compatible APIs

Requirements

C++ Standard: C++20 or higher
CMake: 3.16 or higher

Dependencies and Modifications

nlohmann/json (Patched)

This project uses a patched version of nlohmann/json to remove the dependency on localeconv(), which is not thread-safe. The patch ensures:

Thread Safety: Eliminates calls to the non-thread-safe localeconv() function, allowing downstream users to safely use the library in multi-threaded environments without worrying about locale-related race conditions
Consistent Behavior: Always uses '.' as the decimal point separator regardless of system locale
Simplified Integration: Downstream users don't need to implement locale synchronization or worry about thread safety issues

This modification improves both safety and portability of the JSON library in concurrent applications.

Acknowledgments

Inspired by the excellent Vercel AI SDK for TypeScript/JavaScript developers.

For Tasks:

Click tags to check more tools for each tasks

generate text stream content handle errors call tools execute async tasks

For Jobs:

software developer ai engineer data scientist machine learning engineer research scientist

Alternative AI tools for ai-sdk-cpp

Similar Open Source Tools

ai-sdk-cpp

github

: 94

mcphub.nvim

MCPHub.nvim is a powerful Neovim plugin that integrates MCP (Model Context Protocol) servers into your workflow. It offers a centralized config file for managing servers and tools, with an intuitive UI for testing resources. Ideal for LLM integration, it provides programmatic API access and interactive testing through the `:MCPHub` command.

github

: 448

acte

Acte is a framework designed to build GUI-like tools for AI Agents. It aims to address the issues of cognitive load and freedom degrees when interacting with multiple APIs in complex scenarios. By providing a graphical user interface (GUI) for Agents, Acte helps reduce cognitive load and constraints interaction, similar to how humans interact with computers through GUIs. The tool offers APIs for starting new sessions, executing actions, and displaying screens, accessible via HTTP requests or the SessionManager class.

github

: 113

Remote-MCP

Remote-MCP is a type-safe, bidirectional, and simple solution for remote MCP communication, enabling remote access and centralized management of model contexts. It provides a bridge for immediate remote access to a remote MCP server from a local MCP client, without waiting for future official implementations. The repository contains client and server libraries for creating and connecting to remotely accessible MCP services. The core features include basic type-safe client/server communication, MCP command/tool/prompt support, custom headers, and ongoing work on crash-safe handling and event subscription system.

github

: 83

tambo

tambo ai is a React library that simplifies the process of building AI assistants and agents in React by handling thread management, state persistence, streaming responses, AI orchestration, and providing a compatible React UI library. It eliminates React boilerplate for AI features, allowing developers to focus on creating exceptional user experiences with clean React hooks that seamlessly integrate with their codebase.

github

: 659

polyfire-js

Polyfire is an all-in-one managed backend for AI apps that allows users to build AI apps directly from the frontend, eliminating the need for a separate backend. It simplifies the process by providing most backend services in just a few lines of code. With Polyfire, users can easily create chatbots, transcribe audio files to text, generate simple text, create a long-term memory, and generate images with Dall-E. The tool also offers starter guides and tutorials to help users get started quickly and efficiently.

github

: 128

lionagi

LionAGI is a powerful intelligent workflow automation framework that introduces advanced ML models into any existing workflows and data infrastructure. It can interact with almost any model, run interactions in parallel for most models, produce structured pydantic outputs with flexible usage, automate workflow via graph based agents, use advanced prompting techniques, and more. LionAGI aims to provide a centralized agent-managed framework for "ML-powered tools coordination" and to dramatically lower the barrier of entries for creating use-case/domain specific tools. It is designed to be asynchronous only and requires Python 3.10 or higher.

github

: 322

solana-agent-kit

Solana Agent Kit is an open-source toolkit designed for connecting AI agents to Solana protocols. It enables agents, regardless of the model used, to autonomously perform various Solana actions such as trading tokens, launching new tokens, lending assets, sending compressed airdrops, executing blinks, and more. The toolkit integrates core blockchain features like token operations, NFT management via Metaplex, DeFi integration, Solana blinks, AI integration features with LangChain, autonomous modes, and AI tools. It provides ready-to-use tools for blockchain operations, supports autonomous agent actions, and offers features like memory management, real-time feedback, and error handling. Solana Agent Kit facilitates tasks such as deploying tokens, creating NFT collections, swapping tokens, lending tokens, staking SOL, and sending SPL token airdrops via ZK compression. It also includes functionalities for fetching price data from Pyth and relies on key Solana and Metaplex libraries for its operations.

github

: 1.1k

com.openai.unity

com.openai.unity is an OpenAI package for Unity that allows users to interact with OpenAI's API through RESTful requests. It is independently developed and not an official library affiliated with OpenAI. Users can fine-tune models, create assistants, chat completions, and more. The package requires Unity 2021.3 LTS or higher and can be installed via Unity Package Manager or Git URL. Various features like authentication, Azure OpenAI integration, model management, thread creation, chat completions, audio processing, image generation, file management, fine-tuning, batch processing, embeddings, and content moderation are available.

github

: 496

UniChat

UniChat is a pipeline tool for creating online and offline chat-bots in Unity. It leverages Unity.Sentis and text vector embedding technology to enable offline mode text content search based on vector databases. The tool includes a chain toolkit for embedding LLM and Agent in games, along with middleware components for Text to Speech, Speech to Text, and Sub-classifier functionalities. UniChat also offers a tool for invoking tools based on ReActAgent workflow, allowing users to create personalized chat scenarios and character cards. The tool provides a comprehensive solution for designing flexible conversations in games while maintaining developer's ideas.

github

: 62

aioshelly

Aioshelly is an asynchronous library designed to control Shelly devices. It is currently under development and requires Python version 3.11 or higher, along with dependencies like bluetooth-data-tools, aiohttp, and orjson. The library provides examples for interacting with Gen1 devices using CoAP protocol and Gen2/Gen3 devices using RPC and WebSocket protocols. Users can easily connect to Shelly devices, retrieve status information, and perform various actions through the provided APIs. The repository also includes example scripts for quick testing and usage guidelines for contributors to maintain consistency with the Shelly API.

github

: 51

OpenAI-DotNet

OpenAI-DotNet is a simple C# .NET client library for OpenAI to use through their RESTful API. It is independently developed and not an official library affiliated with OpenAI. Users need an OpenAI API account to utilize this library. The library targets .NET 6.0 and above, working across various platforms like console apps, winforms, wpf, asp.net, etc., and on Windows, Linux, and Mac. It provides functionalities for authentication, interacting with models, assistants, threads, chat, audio, images, files, fine-tuning, embeddings, and moderations.

github

: 732

modelfusion

ModelFusion is an abstraction layer for integrating AI models into JavaScript and TypeScript applications, unifying the API for common operations such as text streaming, object generation, and tool usage. It provides features to support production environments, including observability hooks, logging, and automatic retries. You can use ModelFusion to build AI applications, chatbots, and agents. ModelFusion is a non-commercial open source project that is community-driven. You can use it with any supported provider. ModelFusion supports a wide range of models including text generation, image generation, vision, text-to-speech, speech-to-text, and embedding models. ModelFusion infers TypeScript types wherever possible and validates model responses. ModelFusion provides an observer framework and logging support. ModelFusion ensures seamless operation through automatic retries, throttling, and error handling mechanisms. ModelFusion is fully tree-shakeable, can be used in serverless environments, and only uses a minimal set of dependencies.

github

: 918

UnrealGenAISupport

The Unreal Engine Generative AI Support Plugin is a tool designed to integrate various cutting-edge LLM/GenAI models into Unreal Engine for game development. It aims to simplify the process of using AI models for game development tasks, such as controlling scene objects, generating blueprints, running Python scripts, and more. The plugin currently supports models from organizations like OpenAI, Anthropic, XAI, Google Gemini, Meta AI, Deepseek, and Baidu. It provides features like API support, model control, generative AI capabilities, UI generation, project file management, and more. The plugin is still under development but offers a promising solution for integrating AI models into game development workflows.

github

: 263

FlashLearn

FlashLearn is a tool that provides a simple interface and orchestration for incorporating Agent LLMs into workflows and ETL pipelines. It allows data transformations, classifications, summarizations, rewriting, and custom multi-step tasks using LLMs. Each step and task has a compact JSON definition, making pipelines easy to understand and maintain. FlashLearn supports LiteLLM, Ollama, OpenAI, DeepSeek, and other OpenAI-compatible clients.

github

: 574

ElevenLabs-DotNet

ElevenLabs-DotNet is a non-official Eleven Labs voice synthesis RESTful client that allows users to convert text to speech. The library targets .NET 8.0 and above, working across various platforms like console apps, winforms, wpf, and asp.net, and across Windows, Linux, and Mac. Users can authenticate using API keys directly, from a configuration file, or system environment variables. The tool provides functionalities for text to speech conversion, streaming text to speech, accessing voices, dubbing audio or video files, generating sound effects, managing history of synthesized audio clips, and accessing user information and subscription status.

github

: 53

For similar tasks

LLM-Tool-Survey

This repository contains a collection of papers related to tool learning with large language models (LLMs). The papers are organized according to the survey paper 'Tool Learning with Large Language Models: A Survey'. The survey focuses on the benefits and implementation of tool learning with LLMs, covering aspects such as task planning, tool selection, tool calling, response generation, benchmarks, evaluation, challenges, and future directions in the field. It aims to provide a comprehensive understanding of tool learning with LLMs and inspire further exploration in this emerging area.

github

: 220

tool-ahead-of-time

Tool-Ahead-of-Time (TAoT) is a Python package that enables tool calling for any model available through Langchain's ChatOpenAI library, even before official support is provided. It reformats model output into a JSON parser for tool calling. The package supports OpenAI and non-OpenAI models, following LangChain's syntax for tool calling. Users can start using the tool without waiting for official support, providing a more robust solution for tool calling.

github

: 91

mcphub.nvim

github

: 448

go-utcp

The Universal Tool Calling Protocol (UTCP) is a modern, flexible, and scalable standard for defining and interacting with tools across various communication protocols. It emphasizes scalability, interoperability, and ease of use. It provides built-in transports for HTTP, CLI, Server-Sent Events, streaming HTTP, GraphQL, MCP, and UDP. Users can use the library to construct a client and call tools using the available transports. The library also includes utilities for variable substitution, in-memory repository for storing providers and tools, and OpenAPI conversion to UTCP manuals.

github

: 59

utcp-specification

The Universal Tool Calling Protocol (UTCP) Specification repository contains the official documentation for a modern and scalable standard that enables AI systems and clients to discover and interact with tools across different communication protocols. It defines tool discovery mechanisms, call formats, provider configuration, authentication methods, and response handling.

github

: 210

ailoy

Ailoy is a lightweight library for building AI applications such as agent systems or RAG pipelines with ease. It enables AI features effortlessly, supporting AI models locally or via cloud APIs, multi-turn conversation, system message customization, reasoning-based workflows, tool calling capabilities, and built-in vector store support. It also supports running native-equivalent functionality in web browsers using WASM. The library is in early development stages and provides examples in the `examples` directory for inspiration on building applications with Agents.

github

: 107

LocalLLMClient

LocalLLMClient is a Swift package designed to interact with local Large Language Models (LLMs) on Apple platforms. It supports GGUF, MLX models, and the FoundationModels framework, providing streaming API, multimodal capabilities, and tool calling functionalities. Users can easily integrate this tool to work with various models for text generation and processing. The package also includes advanced features for low-level API control and multimodal image processing. LocalLLMClient is experimental and subject to API changes, offering support for iOS, macOS, and Linux platforms.

github

: 81

ai-sdk-cpp

github

: 94

For similar jobs

sweep

Sweep is an AI junior developer that turns bugs and feature requests into code changes. It automatically handles developer experience improvements like adding type hints and improving test coverage.

github

: 7.1k

teams-ai

The Teams AI Library is a software development kit (SDK) that helps developers create bots that can interact with Teams and Microsoft 365 applications. It is built on top of the Bot Framework SDK and simplifies the process of developing bots that interact with Teams' artificial intelligence capabilities. The SDK is available for JavaScript/TypeScript, .NET, and Python.

github

: 502

ai-guide

This guide is dedicated to Large Language Models (LLMs) that you can run on your home computer. It assumes your PC is a lower-end, non-gaming setup.

github

: 159

classifai

Supercharge WordPress Content Workflows and Engagement with Artificial Intelligence. Tap into leading cloud-based services like OpenAI, Microsoft Azure AI, Google Gemini and IBM Watson to augment your WordPress-powered websites. Publish content faster while improving SEO performance and increasing audience engagement. ClassifAI integrates Artificial Intelligence and Machine Learning technologies to lighten your workload and eliminate tedious tasks, giving you more time to create original content that matters.

github

: 669

chatbot-ui

Chatbot UI is an open-source AI chat app that allows users to create and deploy their own AI chatbots. It is easy to use and can be customized to fit any need. Chatbot UI is perfect for businesses, developers, and anyone who wants to create a chatbot.

github

: 27.7k

BricksLLM

BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM: * Set LLM usage limits for users on different pricing tiers * Track LLM usage on a per user and per organization basis * Block or redact requests containing PIIs * Improve LLM reliability with failovers, retries and caching * Distribute API keys with rate limits and cost limits for internal development/production use cases * Distribute API keys with rate limits and cost limits for students

github

: 953

uAgents

uAgents is a Python library developed by Fetch.ai that allows for the creation of autonomous AI agents. These agents can perform various tasks on a schedule or take action on various events. uAgents are easy to create and manage, and they are connected to a fast-growing network of other uAgents. They are also secure, with cryptographically secured messages and wallets.

github

: 1.3k

griptape

Griptape is a modular Python framework for building AI-powered applications that securely connect to your enterprise data and APIs. It offers developers the ability to maintain control and flexibility at every step. Griptape's core components include Structures (Agents, Pipelines, and Workflows), Tasks, Tools, Memory (Conversation Memory, Task Memory, and Meta Memory), Drivers (Prompt and Embedding Drivers, Vector Store Drivers, Image Generation Drivers, Image Query Drivers, SQL Drivers, Web Scraper Drivers, and Conversation Memory Drivers), Engines (Query Engines, Extraction Engines, Summary Engines, Image Generation Engines, and Image Query Engines), and additional components (Rulesets, Loaders, Artifacts, Chunkers, and Tokenizers). Griptape enables developers to create AI-powered applications with ease and efficiency.

github

: 2.2k