indexify

A realtime serving engine for Data-Intensive Generative AI Applications

Stars: 1182

Visit

Indexify is an open-source engine for building fast data pipelines for unstructured data (video, audio, images, and documents) using reusable extractors for embedding, transformation, and feature extraction. LLM Applications can query transformed content friendly to LLMs by semantic search and SQL queries. Indexify keeps vector databases and structured databases (PostgreSQL) updated by automatically invoking the pipelines as new data is ingested into the system from external data sources. **Why use Indexify** * Makes Unstructured Data **Queryable** with **SQL** and **Semantic Search** * **Real-Time** Extraction Engine to keep indexes **automatically** updated as new data is ingested. * Create **Extraction Graph** to describe **data transformation** and extraction of **embedding** and **structured extraction**. * **Incremental Extraction** and **Selective Deletion** when content is deleted or updated. * **Extractor SDK** allows adding new extraction capabilities, and many readily available extractors for **PDF**, **Image**, and **Video** indexing and extraction. * Works with **any LLM Framework** including **Langchain**, **DSPy**, etc. * Runs on your laptop during **prototyping** and also scales to **1000s of machines** on the cloud. * Works with many **Blob Stores**, **Vector Stores**, and **Structured Databases** * We have even **Open Sourced Automation** to deploy to Kubernetes in production.

README:

Indexify

Compute Engine for Building Data Platforms

Indexify is a compute engine for building data platforms in Python. Create large-scale data processing workflows and agentic applications with durable execution—functions automatically retry on failure, and workflows seamlessly scale across machines. Upon deployment, each application gets a unique URL that can be called from any system.

Note: Indexify is the open-source core that powers Tensorlake Cloud—a serverless platform for document processing, media pipelines, and agentic applications.

✨ Features

Feature	Description
🐍 Python Native	Define workflows as Python functions with type hints—no DSLs, YAML, or config files
🔄 Durable Execution	Functions automatically retry on failure with persistent state across restarts
📊 Distributed Map/Reduce	Parallelize functions over sequences across machines with automatic data shuffling
⚡ Request Queuing	Automatically queue and batch invocations to maximize GPU utilization
🌐 Multi-Cloud	Run across multiple clouds, datacenters, or regions with minimal configuration
📈 Autoscaling	Server automatically redistributes work when machines come and go

🚀 What Can You Build?

Large-Scale Data Processing Workflows

Build production-grade data pipelines entirely in Python with automatic parallelization, fault tolerance, and distributed execution:

Document Processing — Extract tables, images, and text from PDFs at scale; build knowledge graphs; implement RAG pipelines
Media Pipelines — Transcribe and summarize video/audio content; detect and describe objects in images
ETL & Data Transformation — Process millions of records with distributed map/reduce operations

Agentic Applications

Build durable AI agents that reliably execute multi-step workflows:

Tool-Calling Agents — Orchestrate LLM tool calls with automatic state management and retry logic
Multi-Agent Systems — Coordinate multiple agents with durable message passing

📖 Explore the Cookbooks → for complete examples and tutorials.

📦 Installation

Using pip:

pip install indexify tensorlake

🎯 Quick Start

Define Your Application

Create applications using @application() and @function() decorators. Each function runs in its own isolated sandbox with durable execution—if a function crashes, it automatically restarts from where it left off.

from typing import List
from pydantic import BaseModel
from tensorlake.applications import application, function, Image, run_local_application

# Define container image with dependencies
embedding_image = Image(base_image="python:3.11-slim", name="embedding_image").run(
    "pip install sentence-transformers langchain-text-splitters chromadb"
)


class TextChunk(BaseModel):
    chunk: str
    page_number: int


class ChunkEmbedding(BaseModel):
    text: str
    embedding: List[float]


@function(image=embedding_image)
def chunk_text(text: str) -> List[TextChunk]:
    """Split text into chunks for embedding."""
    from langchain_text_splitters import RecursiveCharacterTextSplitter

    splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=20)
    texts = splitter.create_documents([text])
    return [
        TextChunk(chunk=chunk.page_content, page_number=i)
        for i, chunk in enumerate(texts)
    ]


@function(image=embedding_image)
def embed_chunks(chunks: List[TextChunk]) -> List[ChunkEmbedding]:
    """Embed text chunks using sentence transformers."""
    from sentence_transformers import SentenceTransformer
    
    model = SentenceTransformer("all-MiniLM-L6-v2")
    return [
        ChunkEmbedding(text=chunk.chunk, embedding=model.encode(chunk.chunk).tolist())
        for chunk in chunks
    ]


@function(image=embedding_image)
def write_to_vectordb(embeddings: List[ChunkEmbedding]) -> str:
    """Write embeddings to ChromaDB."""
    import chromadb
    import uuid
    
    client = chromadb.PersistentClient("./chromadb_data")
    collection = client.get_or_create_collection("documents")
    
    for emb in embeddings:
        collection.upsert(
            ids=[str(uuid.uuid4())],
            embeddings=[emb.embedding],
            documents=[emb.text],
        )
    return f"Indexed {len(embeddings)} chunks"


@application()
@function(description="Text embedding pipeline")
def text_embedder(text: str) -> str:
    """Main application: chunks text, embeds it, and stores in vector DB."""
    chunks = chunk_text(text)
    embeddings = embed_chunks(chunks)
    result = write_to_vectordb(embeddings)
    return result

Deploy to Tensorlake Cloud (Fastest Way to Get Started)

Tensorlake Cloud is the fastest way to test and deploy your applications—no infrastructure setup required. Get an API key and deploy in seconds:

# Set your API key
export TENSORLAKE_API_KEY="your-api-key"

# Deploy the application
tensorlake deploy workflow.py
# => Deployed! URL: https://api.tensorlake.ai/namespaces/default/applications/text_embedder

Invoke your application using the SDK or call the URL directly:

from tensorlake.applications import run_remote_application

request = run_remote_application(text_embedder, "Your document text here...")
result = request.output()
print(result)

Self-Host with Indexify

If you prefer to self-host or need on-premise deployment, you can run the Indexify server locally:

# Terminal 1: Start the server
docker run -p 8900:8900 tensorlake/indexify-server

# Terminal 2: Start an executor (repeat for more parallelism)
indexify-cli executor

Set the API URL and deploy:

export TENSORLAKE_API_URL=http://localhost:8900
tensorlake deploy workflow.py
# => Deployed! URL: http://localhost:8900/namespaces/default/applications/text_embedder

Run your application:

from tensorlake.applications import run_remote_application

request = run_remote_application(text_embedder, "Your document text here...")
result = request.output()
print(result)

Test Locally (No Server Required)

For quick iteration during development, run applications locally without any infrastructure:

if __name__ == "__main__":
    request = run_local_application(text_embedder, "Your document text here...")
    result = request.output()
    print(result)

🏗️ Production Self-Hosted Deployment

For production self-hosted deployments, see operations/k8s for Kubernetes deployment manifests and Helm charts.

☁️ Tensorlake Cloud vs Self-Hosted Indexify

Start with Tensorlake Cloud to build and test your applications without infrastructure overhead. When you're ready for self-hosting or need on-premise deployment, Indexify provides the same runtime you can run anywhere.

Feature	Tensorlake Cloud	Indexify (Self-Hosted)
Setup Time	Instant—just get an API key	Deploy server + executors
Image Building	Automatic image builds when you deploy	Build and manage container images yourself
Auto-Scaling	Dynamic container scaling with scale-to-zero	Manual executor management
Security	Secure sandboxes (gVisor, Linux containers, virtualization)	Standard container isolation
Secrets	Built-in secret management for applications	Manage secrets externally
Observability	Logging, tracing, and observability built-in	Bring your own logging/tracing
Testing	Interactive playground to invoke applications	Local development only

Get started with Tensorlake Cloud →

🤝 Contributing

We welcome contributions! See CONTRIBUTING.md for guidelines.

📄 License

Indexify is licensed under the Apache 2.0 License.

Website • Docs • Slack • Twitter

For Tasks:

Click tags to check more tools for each tasks

extract data from videos extract data from audio extract data from images extract data from documents query data with sql

For Jobs:

data engineer data scientist machine learning engineer research scientist software engineer

Alternative AI tools for indexify

Similar Open Source Tools

indexify

github

: 1.2k

ctinexus

CTINexus is a framework that leverages optimized in-context learning of large language models to automatically extract cyber threat intelligence from unstructured text and construct cybersecurity knowledge graphs. It processes threat intelligence reports to extract cybersecurity entities, identify relationships between security concepts, and construct knowledge graphs with interactive visualizations. The framework requires minimal configuration, with no extensive training data or parameter tuning needed.

github

: 70

layra

LAYRA is the world's first visual-native AI automation engine that sees documents like a human, preserves layout and graphical elements, and executes arbitrarily complex workflows with full Python control. It empowers users to build next-generation intelligent systems with no limits or compromises. Built for Enterprise-Grade deployment, LAYRA features a modern frontend, high-performance backend, decoupled service architecture, visual-native multimodal document understanding, and a powerful workflow engine.

github

: 817

MassGen

MassGen is a cutting-edge multi-agent system that leverages the power of collaborative AI to solve complex tasks. It assigns a task to multiple AI agents who work in parallel, observe each other's progress, and refine their approaches to converge on the best solution to deliver a comprehensive and high-quality result. The system operates through an architecture designed for seamless multi-agent collaboration, with key features including cross-model/agent synergy, parallel processing, intelligence sharing, consensus building, and live visualization. Users can install the system, configure API settings, and run MassGen for various tasks such as question answering, creative writing, research, development & coding tasks, and web automation & browser tasks. The roadmap includes plans for advanced agent collaboration, expanded model, tool & agent integration, improved performance & scalability, enhanced developer experience, and a web interface.

github

: 454

redamon

github

: 896

seline

Seline is a local-first AI desktop application that integrates conversational AI, visual generation tools, vector search, and multi-channel connectivity. It allows users to connect WhatsApp, Telegram, or Slack to create always-on bots with full context and background task delivery. The application supports multi-channel connectivity, deep research mode, local web browsing with Puppeteer, local knowledge and privacy features, visual and creative tools, automation and agents, developer experience enhancements, and more. Seline is actively developed with a focus on improving user experience and functionality.

github

: 136

osaurus

Osaurus is a native, Apple Silicon-only local LLM server built on Apple's MLX for maximum performance on M‑series chips. It is a SwiftUI app + SwiftNIO server with OpenAI‑compatible and Ollama‑compatible endpoints. The tool supports native MLX text generation, model management, streaming and non‑streaming chat completions, OpenAI‑compatible function calling, real-time system resource monitoring, and path normalization for API compatibility. Osaurus is designed for macOS 15.5+ and Apple Silicon (M1 or newer) with Xcode 16.4+ required for building from source.

github

: 3.6k

agentfield

AgentField is an open-source control plane designed for autonomous AI agents, providing infrastructure for agents to make decisions beyond chatbots. It offers features like scaling infrastructure, routing & discovery, async execution, durable state, observability, trust infrastructure with cryptographic identity, verifiable credentials, and policy enforcement. Users can write agents in Python, Go, TypeScript, or interact via REST APIs. The tool enables the creation of AI backends that reason autonomously within defined boundaries, offering predictability and flexibility. AgentField aims to bridge the gap between AI frameworks and production-ready infrastructure for AI agents.

github

: 757

executorch

ExecuTorch is an end-to-end solution for enabling on-device inference capabilities across mobile and edge devices including wearables, embedded devices and microcontrollers. It is part of the PyTorch Edge ecosystem and enables efficient deployment of PyTorch models to edge devices. Key value propositions of ExecuTorch are: * **Portability:** Compatibility with a wide variety of computing platforms, from high-end mobile phones to highly constrained embedded systems and microcontrollers. * **Productivity:** Enabling developers to use the same toolchains and SDK from PyTorch model authoring and conversion, to debugging and deployment to a wide variety of platforms. * **Performance:** Providing end users with a seamless and high-performance experience due to a lightweight runtime and utilizing full hardware capabilities such as CPUs, NPUs, and DSPs.

github

: 4.3k

inference-gateway

The Inference Gateway is an open-source proxy server designed to simplify access to various language model APIs. It allows users to interact with different language models through a unified interface, stream tokens in real-time, process images alongside text, and use Docker or Kubernetes for deployment. The gateway supports Model Context Protocol integration, provides metrics and observability features, and is production-ready with minimal resource consumption. It offers middleware control and bypass mechanisms, enabling users to manage capabilities like MCP and vision support. The CLI tool provides status monitoring, interactive chat, configuration management, project initialization, and tool execution functionalities. The project aims to provide a flexible solution for AI Agents, supporting self-hosted LLMs and avoiding vendor lock-in.

github

: 101

AgentNeo

AgentNeo is an advanced, open-source Agentic AI Application Observability, Monitoring, and Evaluation Framework designed to provide deep insights into AI agents, Large Language Model (LLM) calls, and tool interactions. It offers robust logging, visualization, and evaluation capabilities to help debug and optimize AI applications with ease. With features like tracing LLM calls, monitoring agents and tools, tracking interactions, detailed metrics collection, flexible data storage, simple instrumentation, interactive dashboard, project management, execution graph visualization, and evaluation tools, AgentNeo empowers users to build efficient, cost-effective, and high-quality AI-driven solutions.

github

: 5.9k

vibesdk

Cloudflare VibeSDK is an open source full-stack AI webapp generator built on Cloudflare's developer platform. It allows companies to build AI-powered platforms, enables internal development for non-technical teams, and supports SaaS platforms to extend product functionality. The platform features AI code generation, live previews, interactive chat, modern stack generation, one-click deploy, and GitHub integration. It is built on Cloudflare's platform with frontend in React + Vite, backend in Workers with Durable Objects, database in D1 (SQLite) with Drizzle ORM, AI integration via multiple LLM providers, sandboxed app previews and execution in containers, and deployment to Workers for Platforms with dispatch namespaces. The platform also offers an SDK for programmatic access to build apps programmatically using TypeScript SDK.

github

: 4.8k

transformerlab-app

Transformer Lab is an app that allows users to experiment with Large Language Models by providing features such as one-click download of popular models, finetuning across different hardware, RLHF and Preference Optimization, working with LLMs across different operating systems, chatting with models, using different inference engines, evaluating models, building datasets for training, calculating embeddings, providing a full REST API, running in the cloud, converting models across platforms, supporting plugins, embedded Monaco code editor, prompt editing, inference logs, all through a simple cross-platform GUI.

github

: 4.8k

GMTalker

GMTalker is an interactive digital human rendered by Unreal Engine, developed by the Media Intelligence Team at Bright Laboratory. The system integrates speech recognition, speech synthesis, natural language understanding, and lip-sync animation driving. It supports rapid deployment on Windows with only 2GB of VRAM required. The project showcases two 3D cartoon digital human avatars suitable for presentations, expansions, and commercial integration.

github

: 385

handit.ai

Handit.ai is an autonomous engineer tool designed to fix AI failures 24/7. It catches failures, writes fixes, tests them, and ships PRs automatically. It monitors AI applications, detects issues, generates fixes, tests them against real data, and ships them as pull requests—all automatically. Users can write JavaScript, TypeScript, Python, and more, and the tool automates what used to require manual debugging and firefighting.

github

: 180

trpc-agent-go

A powerful Go framework for building intelligent agent systems with large language models (LLMs), hierarchical planners, memory, telemetry, and a rich tool ecosystem. tRPC-Agent-Go enables the creation of autonomous or semi-autonomous agents that reason, call tools, collaborate with sub-agents, and maintain long-term state. The framework provides detailed documentation, examples, and tools for accelerating the development of AI applications.

github

: 880

For similar tasks

indexify

github

: 1.2k

unstructured

The `unstructured` library provides open-source components for ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, and many more. The use cases of `unstructured` revolve around streamlining and optimizing the data processing workflow for LLMs. `unstructured` modular functions and connectors form a cohesive system that simplifies data ingestion and pre-processing, making it adaptable to different platforms and efficient in transforming unstructured data into structured outputs.

github

: 14.0k

logfire

Pydantic Logfire is an observability platform that provides simple and powerful dashboard, Python-centric insights, SQL querying, OpenTelemetry integration, and Pydantic validation analytics. It offers unparalleled visibility into Python applications' behavior and allows querying data using standard SQL. Logfire is an opinionated wrapper around OpenTelemetry, supporting traces, metrics, and logs. The Python SDK for logfire is open source, while the server application for recording and displaying data is closed source.

github

: 4.0k

For similar jobs

db2rest

DB2Rest is a modern low-code REST DATA API platform that simplifies the development of intelligent applications. It seamlessly integrates existing and new databases with language models (LMs/LLMs) and vector stores, enabling the rapid delivery of context-aware, reasoning applications without vendor lock-in.

github

: 167

mage-ai

Mage is an open-source data pipeline tool for transforming and integrating data. It offers an easy developer experience, engineering best practices built-in, and data as a first-class citizen. Mage makes it easy to build, preview, and launch data pipelines, and provides observability and scaling capabilities. It supports data integrations, streaming pipelines, and dbt integration.

github

: 7.8k

airbyte

Airbyte is an open-source data integration platform that makes it easy to move data from any source to any destination. With Airbyte, you can build and manage data pipelines without writing any code. Airbyte provides a library of pre-built connectors that make it easy to connect to popular data sources and destinations. You can also create your own connectors using Airbyte's no-code Connector Builder or low-code CDK. Airbyte is used by data engineers and analysts at companies of all sizes to build and manage their data pipelines.

github

: 20.8k

labelbox-python

Labelbox is a data-centric AI platform for enterprises to develop, optimize, and use AI to solve problems and power new products and services. Enterprises use Labelbox to curate data, generate high-quality human feedback data for computer vision and LLMs, evaluate model performance, and automate tasks by combining AI and human-centric workflows. The academic & research community uses Labelbox for cutting-edge AI research.

github

: 135

telemetry-airflow

This repository codifies the Airflow cluster that is deployed at workflow.telemetry.mozilla.org (behind SSO) and commonly referred to as "WTMO" or simply "Airflow". Some links relevant to users and developers of WTMO: * The `dags` directory in this repository contains some custom DAG definitions * Many of the DAGs registered with WTMO don't live in this repository, but are instead generated from ETL task definitions in bigquery-etl * The Data SRE team maintains a WTMO Developer Guide (behind SSO)

github

: 185

airflow

Apache Airflow (or simply Airflow) is a platform to programmatically author, schedule, and monitor workflows. When workflows are defined as code, they become more maintainable, versionable, testable, and collaborative. Use Airflow to author workflows as directed acyclic graphs (DAGs) of tasks. The Airflow scheduler executes your tasks on an array of workers while following the specified dependencies. Rich command line utilities make performing complex surgeries on DAGs a snap. The rich user interface makes it easy to visualize pipelines running in production, monitor progress, and troubleshoot issues when needed.

github

: 44.4k

airbyte-platform

Airbyte is an open-source data integration platform that makes it easy to move data from any source to any destination. With Airbyte, you can build and manage data pipelines without writing any code. Airbyte provides a library of pre-built connectors that make it easy to connect to popular data sources and destinations. You can also create your own connectors using Airbyte's low-code Connector Development Kit (CDK). Airbyte is used by data engineers and analysts at companies of all sizes to move data for a variety of purposes, including data warehousing, data analysis, and machine learning.

github

: 273

chronon

Chronon is a platform that simplifies and improves ML workflows by providing a central place to define features, ensuring point-in-time correctness for backfills, simplifying orchestration for batch and streaming pipelines, offering easy endpoints for feature fetching, and guaranteeing and measuring consistency. It offers benefits over other approaches by enabling the use of a broad set of data for training, handling large aggregations and other computationally intensive transformations, and abstracting away the infrastructure complexity of data plumbing.

github

: 766