spiceai

spiceai

A portable accelerated data query and LLM-inference engine, written in Rust, for data-grounded AI apps and agents.

Stars: 2025

Visit
 screenshot

Spice is a portable runtime written in Rust that offers developers a unified SQL interface to materialize, accelerate, and query data from any database, data warehouse, or data lake. It connects, fuses, and delivers data to applications, machine-learning models, and AI-backends, functioning as an application-specific, tier-optimized Database CDN. Built with industry-leading technologies such as Apache DataFusion, Apache Arrow, Apache Arrow Flight, SQLite, and DuckDB. Spice makes it fast and easy to query data from one or more sources using SQL, co-locating a managed dataset with applications or machine learning models, and accelerating it with Arrow in-memory, SQLite/DuckDB, or attached PostgreSQL for fast, high-concurrency, low-latency queries.

README:

spice oss logo

CodeQL License: Apache-2.0 Discord Follow on X

GitHub Actions Workflow Status - build GitHub Actions Workflow Status - docker build GitHub Actions Workflow Status - unit tests GitHub Actions Workflow Status - integration tests GitHub Actions Workflow Status - integration tests (models) GitHub Actions Workflow Status - benchmark tests

Spice is a SQL query and AI compute engine, written in Rust, for data-driven apps and agents.

Spice.ai Open Source accelerated data query and LLM-inference engine

Spice provides three industry standard APIs in a lightweight, portable runtime (single ~140 MB binary):

  1. SQL Query APIs: Arrow Flight, Arrow Flight SQL, ODBC, JDBC, and ADBC.
  2. OpenAI-Compatible APIs: HTTP APIs compatible the OpenAI SDK, AI SDK with local model serving (CUDA/Metal accelerated) and gateway to hosted models.
  3. Iceberg Catalog REST APIs: A unified Iceberg Catalog API.

🎯 Goal: Developers can focus on building data apps and AI agents confidently, knowing they are grounded in data.

Spice is primarily used for:

If you want to build with DataFusion or using DuckDB, Spice provides a simple, flexible, and production-ready engine you can just use.

πŸ“£ Read the Spice.ai OSS announcement blog post.

Spice is built-on industry leading technologies including Apache DataFusion, Apache Arrow, Arrow Flight, SQLite, and DuckDB.

How Spice works.

πŸŽ₯ Watch the CMU Databases Accelerating Data and AI with Spice.ai Open-Source

Why Spice?

Spice.ai

Spice simplifies building data-driven AI applications and agents by making it fast and easy to query, federate, and accelerate data from one or more sources using SQL, while grounding AI in real-time, reliable data. Co-locate datasets with apps and AI models to power AI feedback loops, enable RAG and search, and deliver fast, low-latency data-query and AI-inference with full control over cost and performance.

How is Spice different?

  1. AI-Native Runtime: Spice combines data query and AI inference in a single engine, for data-grounded AI and accurate AI.

  2. Application-Focused: Designed to run distributed at the application and agent level, often as a 1:1 or 1:N mapping between app and Spice instance, unlike traditional data systems built for many apps on one centralized database. It’s common to spin up multiple Spice instancesβ€”even one per tenant or customer.

  3. Dual-Engine Acceleration: Supports both OLAP (Arrow/DuckDB) and OLTP (SQLite/PostgreSQL) engines at the dataset level, providing flexible performance across analytical and transactional workloads.

  4. Disaggregated Storage: Separation of compute from disaggregated storage, co-locating local, materialized working sets of data with applications, dashboards, or ML pipelines while accessing source data in its original storage.

  5. Edge to Cloud Native: Deploy as a standalone instance, Kubernetes sidecar, microservice, or clusterβ€”across edge/POP, on-prem, and public clouds. Chain multiple Spice instances for tier-optimized, distributed deployments.

How does Spice compare?

Data Query and Analytics

Feature Spice Trino / Presto Dremio ClickHouse Materialize
Primary Use-Case Data & AI apps/agents Big data analytics Interactive analytics Real-time analytics Real-time analytics
Primary deployment model Sidecar Cluster Cluster Cluster Cluster
Federated Query Support βœ… βœ… βœ… ― ―
Acceleration/Materialization βœ… (Arrow, SQLite, DuckDB, PostgreSQL) Intermediate storage Reflections (Iceberg) Materialized views βœ… (Real-time views)
Catalog Support βœ… (Iceberg, Unity Catalog) βœ… βœ… ― ―
Query Result Caching βœ… βœ… βœ… βœ… Limited
Multi-Modal Acceleration βœ… (OLAP + OLTP) ― ― ― ―
Change Data Capture (CDC) βœ… (Debezium) ― ― ― βœ… (Debezium)

AI Apps and Agents

Feature Spice LangChain LlamaIndex AgentOps.ai Ollama
Primary Use-Case Data & AI apps Agentic workflows RAG apps Agent operations LLM apps
Programming Language Any language (HTTP interface) JavaScript, Python Python Python Any language (HTTP interface)
Unified Data + AI Runtime βœ… ― ― ― ―
Federated Data Query βœ… ― ― ― ―
Accelerated Data Access βœ… ― ― ― ―
Tools/Functions βœ… βœ… βœ… Limited Limited
LLM Memory βœ… βœ… ― βœ… ―
Evaluations (Evals) βœ… Limited ― Limited ―
Search βœ… (VSS) βœ… βœ… Limited Limited
Caching βœ… (Query and results caching) Limited ― ― ―
Embeddings βœ… (Built-in & pluggable models/DBs) βœ… βœ… Limited ―

βœ… = Fully supported ❌ = Not supported Limited = Partial or restricted support

Example Use-Cases

Data-grounded Agentic AI Applications

  • OpenAI-compatible API: Connect to hosted models (OpenAI, Anthropic, xAI) or deploy locally (Llama, NVIDIA NIM). AI Gateway Recipe
  • Federated Data Access: Query using SQL and NSQL (text-to-SQL) across databases, data warehouses, and data lakes with advanced query push-down for fast retrieval across disparate data sources. Federated SQL Query Recipe
  • Search and RAG: Search and retrieve context with accelerated embeddings for retrieval-augmented generation (RAG) workflows. Vector Search over GitHub Files
  • LLM Memory and Observability: Store and retrieve history and context for AI agents while gaining deep visibility into data flows, model performance, and traces. LLM Memory Recipe | Monitoring Features Documentation

Database CDN and Query Mesh

  • Data Acceleration: Co-locate materialized datasets in Arrow, SQLite, and DuckDB with applications for sub-second query. DuckDB Data Accelerator Recipe
  • Resiliency and Local Dataset Replication: Maintain application availability with local replicas of critical datasets. Local Dataset Replication Recipe
  • Responsive Dashboards: Enable fast, real-time analytics by accelerating data for frontends and BI tools. Sales BI Dashboard Demo
  • Simplified Legacy Migration: Use a single endpoint to unify legacy systems with modern infrastructure, including federated SQL querying across multiple sources. Federated SQL Query Recipe

Retrieval-Augmented Generation (RAG)

FAQ

  • Is Spice a cache? No specifically; you can think of Spice data acceleration as an active cache, materialization, or data prefetcher. A cache would fetch data on a cache-miss while Spice prefetches and materializes filtered data on an interval, trigger, or as data changes using CDC. In addition to acceleration Spice supports results caching.

  • Is Spice a CDN for databases? Yes, a common use-case for Spice is as a CDN for different data sources. Using CDN concepts, Spice enables you to ship (load) a working set of your database (or data lake, or data warehouse) where it's most frequently accessed, like from a data-intensive application or for AI context.

Watch a 30-sec BI dashboard acceleration demo

https://github.com/spiceai/spiceai/assets/80174/7735ee94-3f4a-4983-a98e-fe766e79e03a

See more demos on YouTube.

Supported Data Connectors

Name Description Status Protocol/Format
databricks (mode: delta_lake) Databricks Stable S3/Delta Lake
delta_lake Delta Lake Stable Delta Lake
dremio Dremio Stable Arrow Flight
duckdb DuckDB Stable Embedded
file File Stable Parquet, CSV
github GitHub Stable GitHub API
postgres PostgreSQL Stable
s3 S3 Stable Parquet, CSV
mysql MySQL Stable
graphql GraphQL Release Candidate JSON
databricks (mode: spark_connect) Databricks Beta Spark Connect
flightsql FlightSQL Beta Arrow Flight SQL
iceberg Apache Iceberg Beta Parquet
mssql Microsoft SQL Server Beta Tabular Data Stream (TDS)
odbc ODBC Beta ODBC
snowflake Snowflake Beta Arrow
spark Spark Beta Spark Connect
spice.ai Spice.ai Beta Arrow Flight
abfs Azure BlobFS Alpha Parquet, CSV
clickhouse Clickhouse Alpha
debezium Debezium CDC Alpha Kafka + JSON
dynamodb Amazon DynamoDB Alpha
ftp, sftp FTP/SFTP Alpha Parquet, CSV
http, https HTTP(s) Alpha Parquet, CSV
localpod Local dataset replication Alpha
sharepoint Microsoft SharePoint Alpha Unstructured UTF-8 documents
mongodb MongoDB Coming Soon
elasticsearch ElasticSearch Roadmap

Supported Data Accelerators

Name Description Status Engine Modes
arrow In-Memory Arrow Records Stable memory
duckdb Embedded DuckDB Stable memory, file
postgres Attached PostgreSQL Release Candidate N/A
sqlite Embedded SQLite Release Candidate memory, file

Supported Model Providers

Name Description Status ML Format(s) LLM Format(s)
openai OpenAI (or compatible) LLM endpoint Release Candidate - OpenAI-compatible HTTP endpoint
file Local filesystem Beta ONNX GGUF, GGML, SafeTensor
huggingface Models hosted on HuggingFace Beta ONNX GGUF, GGML, SafeTensor
spice.ai Models hosted on the Spice.ai Cloud Platform ONNX OpenAI-compatible HTTP endpoint
azure Azure OpenAI - OpenAI-compatible HTTP endpoint
anthropic Models hosted on Anthropic Alpha - OpenAI-compatible HTTP endpoint
xai Models hosted on xAI Alpha - OpenAI-compatible HTTP endpoint

Supported Embeddings Providers

Name Description Status ML Format(s) LLM Format(s)*
openai OpenAI (or compatible) LLM endpoint Release Candidate - OpenAI-compatible HTTP endpoint
file Local filesystem Release Candidate ONNX GGUF, GGML, SafeTensor
huggingface Models hosted on HuggingFace Release Candidate ONNX GGUF, GGML, SafeTensor
azure Azure OpenAI Alpha - OpenAI-compatible HTTP endpoint

Supported Catalogs

Catalog Connectors connect to external catalog providers and make their tables available for federated SQL query in Spice. Configuring accelerations for tables in external catalogs is not supported. The schema hierarchy of the external catalog is preserved in Spice.

Name Description Status Protocol/Format
unity_catalog Unity Catalog Stable Delta Lake
databricks Databricks Beta Spark Connect, S3/Delta Lake
iceberg Apache Iceberg Beta Parquet
spice.ai Spice.ai Cloud Platform Alpha Arrow Flight
glue AWS Glue Coming Soon JSON, Parquet, Iceberg

⚑️ Quickstart (Local Machine)

https://github.com/spiceai/spiceai/assets/88671039/85cf9a69-46e7-412e-8b68-22617dcbd4e0

Step 1. Install the Spice CLI:

On macOS, Linux, and WSL:

curl https://install.spiceai.org | /bin/bash

Or using brew:

brew install spiceai/spiceai/spice

On Windows:

curl -L "https://install.spiceai.org/Install.ps1" -o Install.ps1 && PowerShell -ExecutionPolicy Bypass -File ./Install.ps1

Step 2. Initialize a new Spice app with the spice init command:

spice init spice_qs

A spicepod.yaml file is created in the spice_qs directory. Change to that directory:

cd spice_qs

Step 3. Start the Spice runtime:

spice run

Example output will be shown as follows:

2025/01/20 11:26:10 INFO Spice.ai runtime starting...
2025-01-20T19:26:10.679068Z  INFO runtime::init::dataset: No datasets were configured. If this is unexpected, check the Spicepod configuration.
2025-01-20T19:26:10.679716Z  INFO runtime::flight: Spice Runtime Flight listening on 127.0.0.1:50051
2025-01-20T19:26:10.679786Z  INFO runtime::metrics_server: Spice Runtime Metrics listening on 127.0.0.1:9090
2025-01-20T19:26:10.680140Z  INFO runtime::http: Spice Runtime HTTP listening on 127.0.0.1:8090
2025-01-20T19:26:10.682080Z  INFO runtime::opentelemetry: Spice Runtime OpenTelemetry listening on 127.0.0.1:50052
2025-01-20T19:26:10.879126Z  INFO runtime::init::results_cache: Initialized results cache; max size: 128.00 MiB, item ttl: 1s

The runtime is now started and ready for queries.

Step 4. In a new terminal window, add the spiceai/quickstart Spicepod. A Spicepod is a package of configuration defining datasets and ML models.

spice add spiceai/quickstart

The spicepod.yaml file will be updated with the spiceai/quickstart dependency.

version: v1
kind: Spicepod
name: spice_qs
dependencies:
  - spiceai/quickstart

The spiceai/quickstart Spicepod will add a taxi_trips data table to the runtime which is now available to query by SQL.

2025-01-20T19:26:30.011633Z  INFO runtime::init::dataset: Dataset taxi_trips registered (s3://spiceai-demo-datasets/taxi_trips/2024/), acceleration (arrow), results cache enabled.
2025-01-20T19:26:30.013002Z  INFO runtime::accelerated_table::refresh_task: Loading data for dataset taxi_trips
2025-01-20T19:26:40.312839Z  INFO runtime::accelerated_table::refresh_task: Loaded 2,964,624 rows (399.41 MiB) for dataset taxi_trips in 10s 299ms

Step 5. Start the Spice SQL REPL:

spice sql

The SQL REPL inferface will be shown:

Welcome to the Spice.ai SQL REPL! Type 'help' for help.

show tables; -- list available tables
sql>

Enter show tables; to display the available tables for query:

sql> show tables;
+---------------+--------------+---------------+------------+
| table_catalog | table_schema | table_name    | table_type |
+---------------+--------------+---------------+------------+
| spice         | public       | taxi_trips    | BASE TABLE |
| spice         | runtime      | query_history | BASE TABLE |
| spice         | runtime      | metrics       | BASE TABLE |
+---------------+--------------+---------------+------------+

Time: 0.022671708 seconds. 3 rows.

Enter a query to display the longest taxi trips:

SELECT trip_distance, total_amount FROM taxi_trips ORDER BY trip_distance DESC LIMIT 10;

Output:

+---------------+--------------+
| trip_distance | total_amount |
+---------------+--------------+
| 312722.3      | 22.15        |
| 97793.92      | 36.31        |
| 82015.45      | 21.56        |
| 72975.97      | 20.04        |
| 71752.26      | 49.57        |
| 59282.45      | 33.52        |
| 59076.43      | 23.17        |
| 58298.51      | 18.63        |
| 51619.36      | 24.2         |
| 44018.64      | 52.43        |
+---------------+--------------+

Time: 0.045150667 seconds. 10 rows.

βš™οΈ Runtime Container Deployment

Using the Docker image locally:

docker pull spiceai/spiceai

In a Dockerfile:

from spiceai/spiceai:latest

Using Helm:

helm repo add spiceai https://helm.spiceai.org
helm install spiceai spiceai/spiceai

🏎️ Next Steps

Explore the Spice.ai Cookbook

The Spice.ai Cookbook is a collection of recipes and examples for using Spice. Find it at https://github.com/spiceai/cookbook.

Using Spice.ai Cloud Platform

Access ready-to-use Spicepods and datasets hosted on the Spice.ai Cloud Platform using the Spice runtime. A list of public Spicepods is available on Spicerack: https://spicerack.org/.

To use public datasets, create a free account on Spice.ai:

  1. Visit spice.ai and click Try for Free. Try for Free

  2. After creating an account, create an app to generate an API key. Create App

Once set up, you can access ready-to-use Spicepods including datasets. For this demonstration, use the taxi_trips dataset from the Spice.ai Quickstart.

Step 1. Initialize a new project.

# Initialize a new Spice app
spice init spice_app

# Change to app directory
cd spice_app

Step 2. Log in and authenticate from the command line using the spice login command. A pop up browser window will prompt you to authenticate:

spice login

Step 3. Start the runtime:

# Start the runtime
spice run

Step 4. Configure the dataset:

In a new terminal window, configure a new dataset using the spice dataset configure command:

spice dataset configure

Enter a dataset name that will be used to reference the dataset in queries. This name does not need to match the name in the dataset source.

dataset name: (spice_app) taxi_trips

Enter the description of the dataset:

description: Taxi trips dataset

Enter the location of the dataset:

from: spice.ai/spiceai/quickstart/datasets/taxi_trips

Select y when prompted whether to accelerate the data:

Locally accelerate (y/n)? y

You should see the following output from your runtime terminal:

2024-12-16T05:12:45.803694Z  INFO runtime::init::dataset: Dataset taxi_trips registered (spice.ai/spiceai/quickstart/datasets/taxi_trips), acceleration (arrow, 10s refresh), results cache enabled.
2024-12-16T05:12:45.805494Z  INFO runtime::accelerated_table::refresh_task: Loading data for dataset taxi_trips
2024-12-16T05:13:24.218345Z  INFO runtime::accelerated_table::refresh_task: Loaded 2,964,624 rows (8.41 GiB) for dataset taxi_trips in 38s 412ms.

Step 5. In a new terminal window, use the Spice SQL REPL to query the dataset

spice sql
SELECT tpep_pickup_datetime, passenger_count, trip_distance from taxi_trips LIMIT 10;

The output displays the results of the query along with the query execution time:

+----------------------+-----------------+---------------+
| tpep_pickup_datetime | passenger_count | trip_distance |
+----------------------+-----------------+---------------+
| 2024-01-11T12:55:12  | 1               | 0.0           |
| 2024-01-11T12:55:12  | 1               | 0.0           |
| 2024-01-11T12:04:56  | 1               | 0.63          |
| 2024-01-11T12:18:31  | 1               | 1.38          |
| 2024-01-11T12:39:26  | 1               | 1.01          |
| 2024-01-11T12:18:58  | 1               | 5.13          |
| 2024-01-11T12:43:13  | 1               | 2.9           |
| 2024-01-11T12:05:41  | 1               | 1.36          |
| 2024-01-11T12:20:41  | 1               | 1.11          |
| 2024-01-11T12:37:25  | 1               | 2.04          |
+----------------------+-----------------+---------------+

Time: 0.00538925 seconds. 10 rows.

You can experiment with the time it takes to generate queries when using non-accelerated datasets. You can change the acceleration setting from true to false in the datasets.yaml file.

πŸ“„ Documentation

Comprehensive documentation is available at spiceai.org/docs.

Over 45 quickstarts and samples available in the Spice Cookbook.

πŸ”Œ Extensibility

Spice.ai is designed to be extensible with extension points documented at EXTENSIBILITY.md. Build custom Data Connectors, Data Accelerators, Catalog Connectors, Secret Stores, Models, or Embeddings.

πŸ”¨ Upcoming Features

πŸš€ See the Roadmap to v1.0-stable for upcoming features.

🀝 Connect with us

We greatly appreciate and value your support! You can help Spice in a number of ways:

⭐️ star this repo! Thank you for your support! πŸ™

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for spiceai

Similar Open Source Tools

For similar tasks

For similar jobs