sample-apps
Repository of sample applications for https://vespa.ai, the open big data serving engine
Stars: 396
Vespa is an open-source search and AI engine that provides a unified platform for building and deploying search and AI applications. Vespa sample applications showcase various use cases and features of Vespa, including basic search, recommendation, semantic search, image search, text ranking, e-commerce search, question answering, search-as-you-type, and ML inference serving.
README:
The Vespa sample applications are created to run both self-hosted and on Vespa Cloud.
You can easily deploy the sample applications to Vespa Cloud without changing the files -
just follow the same steps as for
Managed Vector Search using Vespa Cloud,
adding security credentials.
First-time users should go through the getting-started guides first.
Explore the examples for smaller applications helping you getting started with a particular feature, and see operations for operational examples.
We try to build each sample application with a standard structure, so you can easily find your way around. This section gives you an overview of this structure.
Click to expand
The standard structure is as follows:
one-sample-app/
├─ README.md
├─ dataset/
│ ├─ a-document.json
│ ├─ another-doc.json
│ ├─ ...
├─ app/
│ ├─ security/
│ ├─ schemas/
│ │ ├─ my_vespa_schema.sd
│ ├─ services.xml
│ ├─ ...
├─ script/
│ ├─ build.sh # when required
│ ├─ ...
There might be other files and folders than the ones listed here, but this should be the bare minimum that you can expect.
Let's analyze the structure:
-
README.md- self explainatory: the README specific for the sample application, we recommend reading it through. -
dataset/- contains the dataset used for the sample application. This is usually a small set of documents, but it can be larger.Note: Sometimes this might be missing, and will be generated by the build script (see below), this is usually because the dataset is too large to be included in the repository.
-
app/- contains the Vespa application package, this is the actual definition of the Vespa deployment. The bare minimum includes:-
security/- this is where you will place yourclients.pemcertificate for data-plane authentication. (Refer to our Security Guide) -
schemas/- contains the schema files used for the application. This is usually a single file, but it can be multiple files. -
services.xml- contains the services configuration for the application. This is usually a single file. - Other files and folders that are required for the application to run.
-
-
script/- contains the optional scripts used to help building the application package. This is usually a single file, but it can be multiple files.-
build.sh- this is the main script, if multiple files are present, this will be the only one you'll need to worry about.Note: We recomend refering to the README for the sample application for details on how to run it.
-
[!NOTE] We are in the process of updating the sample applications to follow the structure described here, but some applications may not be fully updated yet, so please check the README files in each application for details.
Album Recommendations is the intro application to Vespa.
Learn how to configure the schema for simple recommendation and search use cases.
Pyvespa: Hybrid Search - Quickstart and
Pyvespa: Hybrid Search - Quickstart on Vespa Cloud
create a hybrid text search application combining traditional keyword matching with semantic vector search (dense retrieval).
They also demonstrate the Vespa native embedder functionality.
These are intro-level applications for Python users using more advanced Vespa features.
Use Pyvespa: Authenticating to Vespa Cloud for Vespa Cloud credentials.
Pyvespa: Querying Vespa
is a good start for Python users, exploring how to query Vespa using the Vespa Query Language (YQL).
Pyvespa: Read and write operations
documents ways to feed, get, update, and delete data;
Using context manager for efficiently managing resources and feeding streams of data using feed_iter,
which can feed from streams, Iterables, Lists, and files by the use of generators.
Pyvespa: Application packages
is a good intro to the concept of application packages in Vespa.
Try Pyvespa: Advanced Configuration for Vespa Services configuration.
Pyvespa: Examples
is a repository of small snippets and examples, e.g., really simple vector distance search applications.
The News and Recommendation Tutorial
demonstrates basic search functionality and is a great place to start exploring Vespa features.
It creates a recommendation system where the approximate nearest neighbor search in a shared user/item embedding space
is used to retrieve recommended content for a user.
This app also demonstrates using parent-child relationships.
The Text Search Tutorial
demonstrates traditional text search using
BM25/Vespa nativeRank,
and is a good start to using the MS Marco dataset.
There is a growing interest in AI-powered vector representations of unstructured multimodal data
and searching efficiently over these representations.
Managed Vector Search using Vespa Cloud
describes how to unlock the full potential of multimodal AI-powered vector representations using Vespa Cloud.
Vespa Multi-Vector Indexing with HNSW and
Pyvespa: Multi-vector indexing with HNSW
demonstrate how to index multiple vectors per document field for semantic search for longer documents.
These are more advanced than the Hybrid Search examples in the Getting Started section.
Vector Streaming Search
uses vector streaming search for naturally partitioned data, see the
blog post for details.
Multilingual Search with multilingual embeddings
demonstrates multilingual semantic search with multilingual text embedding models.
Simple hybrid search with SPLADE
uses the Vespa splade-embedder for
semantic search using sparse vector representations,
and is a good intro to SPLADE and sparse learned weights for ranking.
Customizing Frozen Data Embeddings in Vespa
demonstrates how to adapt frozen embeddings from foundational embedding models -
see the blog post.
Frozen data embeddings from foundational models is an emerging industry practice
for reducing the complexity of maintaining and versioning embeddings.
The frozen data embeddings are re-used for various tasks, such as classification, search, or recommendations.
Pyvespa: Using Cohere Binary Embeddings in Vespa
demonstrates how to use the Cohere binary vectors with Vespa,
including a re-ranking phase that uses the float query vector version for improved accuracy.
Pyvespa: Billion-scale vector search with Cohere binary embeddings in Vespa
uses the Cohere int8 & binary Embeddings
with a coarse-to-fine search and re-ranking pipeline;
This reduces costs but offers the same retrieval (nDCG) accuracy.
The packed binary vector representation is stored in memory,
with an optional HNSW index using
hamming distance.
The int8 vector representation is stored on disk
using Vespa’s paged option.
Pyvespa: Multilingual Hybrid Search with Cohere binary embeddings and Vespa
demonstrates:
- Building a multilingual search application over a sample of the German split of Wikipedia using binarized Cohere embeddings.
- Indexing multiple binary embeddings per document without having to split the chunks across multiple retrievable units.
- Hybrid search, combining the lexical matching capabilities of Vespa with Cohere binary embeddings.
- Re-scoring the binarized vectors for improved accuracy.
Pyvespa: BGE-M3 - The Mother of all embedding models
demonstrates how to use the BGE-M3 embeddings
and represent all three embedding representations in Vespa.
This code is inspired by the BAAI/bge-m3 README.
Pyvespa: Evaluating retrieval with Snowflake arctic embed
shows how different rank profiles in Vespa can be set up and evaluated.
For the rank profiles that use semantic search,
we will use the small version of Snowflake’s arctic embed model series for generating embeddings.
Pyvespa: Exploring the potential of OpenAI Matryoshka 🪆 embeddings with Vespa
demonstrates the effectiveness of using the recently released (as of January 2024) OpenAI text-embedding-3 embeddings with Vespa.
Specifically, we are interested in the Matryoshka Representation Learning technique used in training,
which lets us "shorten embeddings (i.e., remove some numbers from the end of the sequence) without the embedding losing its concept-representing properties".
This allows us to trade off a small amount of accuracy in exchange for much smaller embedding sizes,
so we can store more documents and search them faster.
Pyvespa: Using Mixedbread.ai embedding model with support for binary vectors
shows how to use the mixedbread-ai/mxbai-embed-large-v1 model
with support for binary vectors with Vespa.
The notebook example also includes a re-ranking phase that uses the float query vector version for improved accuracy.
The re-ranking step makes the model perform at 96.45% of the full float version,
with a 32x decrease in storage footprint.
Retrieval Augmented Generation (RAG) in Vespa
is an end-to-end RAG application where all the steps are run within Vespa.
This application focuses on the generation part of RAG,
with a simple text search using BM25.
This application has three versions of an end-to-end RAG application:
- Using an external LLM service to generate the final response.
- Using local LLM inference to generate the final response.
- Deploying to Vespa Cloud and using GPU-accelerated LLM inference to generate the final response. This includes using Vespa Cloud's Secret Store to save the OpenAI API key.
Pyvespa: Building cost-efficient retrieval-augmented personal AI assistants
uses streaming mode
for cost-efficient retrieval for applications that store and retrieve personal data.
This notebook connects a custom LlamaIndex Retriever
with a Vespa app using streaming mode to retrieve personal data.
Pyvespa: Turbocharge RAG with LangChain and Vespa Streaming Mode for Partitioned Data
uses streaming mode
to build cost-efficient RAG applications over naturally sharded data - also available as a blog post:
Turbocharge RAG with LangChain and Vespa Streaming Mode for Sharded Data.
This example features using elementSimilarity
in search results to easily inspect each chunk's closeness to the query embedding.
Also try Pyvespa: Chat with your pdfs with ColBERT, LangChain, and Vespa -
this demonstrates how you can now use ColBERT ranking natively in Vespa,
which handles the ColBERT embedding process with no custom code.
Pyvespa: Visual PDF RAG with Vespa - ColPali demo application
is an end-to-end demo application for visual retrieval of PDF pages, including a frontend web application -
try vespa-engine-colpali-vespa-visual-retrieval.hf.space for a live demo.
The main goal of the demo is to make it easy to create your own PDF Enterprise Search application using Vespa!
Pyvespa: Vespa 🤝 ColPali: Efficient Document Retrieval with Vision Language Models
demonstrates how to retrieve PDF pages using the embeddings generated by the ColPali model.
ColPali is a powerful Vision Language Model (VLM) that can generate embeddings for images and text.
This notebook uses ColPali to generate embeddings for images of PDF pages and store them in Vespa.
We also store the base64-encoded image of the PDF page and some metadata like title and url.
Pyvespa: Scaling ColPALI (VLM) Retrieval
demonstrates how to represent ColPali in Vespa and to scale to large collections.
Also see the Scaling ColPali to billions of PDFs with Vespa blog post.
Pyvespa: ColPali Ranking Experiments on DocVQA
shows how to reproduce the ColPali results on DocVQA with Vespa.
The dataset consists of PDF documents with questions and answers.
We demonstrate how we can binarize the patch embeddings
and replace the float MaxSim scoring
with a hamming-based MaxSim
without much loss in ranking accuracy but with a significant speedup (close to 4x) and reducing the memory (and storage) requirements by 32x.
Pyvespa: PDF-Retrieval using ColQWen2 (ColPali) with Vespa
is a continuation of the notebooks related to the ColPali models (above) for complex document retrieval,
and demonstrates use of the ColQWen2 model checkpoint.
Billion-Scale Image Search
demonstrates billion-scale image search using a CLIP model
exported in ONNX-format for retrieval.
It features separation of compute from storage and query-time vector similarity de-duping.
It uses PCA to reduce from 768 to 128 dimensions.
Text-video search is a notebook that
downloads a set of videos, converts from .avi to .mp4, creates CLIP embeddings,
feeds to Vespa and lets you query the videos in text using a Streamlit application.
It is a good start for creating a video search application using Vespa!
Video Search and Retrieval with Vespa and TwelveLabs is a notebook
showcasing the use of TwelveLabs state-of-the-art generation and embedding models
for video processing. It demonstrates how to generate rich metadata (including summaries and keywords) for videos
using TwelveLabs' technology, and how to embed video chunks for efficient retrieval. The notebook processes three
sample videos, segments them into chunks, and stores their embeddings along with metadata in Vespa's multi-vector
tensors. You can perform hybrid searches to find specific video scenes based on natural language descriptions.
This serves as an excellent starting point for implementing advanced video retrieval with Vespa!
MS Marco Passage Ranking
shows how to represent state-of-the-art text ranking using Transformer (BERT) models.
It uses the MS Marco passage ranking datasets and features
bi-encoders, cross-encoders, and late-interaction models (ColBERT):
- Simple single-stage sparse retrieval accelerated by the WAND dynamic pruning algorithm with BM25 ranking.
- Dense (vector) search retrieval for efficient candidate retrieval using Vespa's support for approximate nearest neighbor search.
- Re-ranking using the Late contextual interaction over BERT (ColBERT) model.
- Re-ranking using a cross-encoder with cross attention between the query and document terms.
- Multiphase retrieval and ranking combining efficient retrieval (WAND or ANN) with re-ranking stages.
- Using Vespa embedder functionality.
- Hybrid ranking.
With Vespa’s phased ranking capabilities,
doing cross-encoder inference for a subset of documents at a later stage in the ranking pipeline
can be a good trade-off between ranking performance and latency.
Pyvespa: Using Mixedbread.ai cross-encoder for reranking in Vespa.ai
shows how to use the Mixedbread.ai
cross-encoder for global-phase reranking in Vespa.
Pyvespa: Standalone ColBERT with Vespa for end-to-end retrieval and ranking
illustrates using the colbert-ai package to produce token vectors,
instead of using the native Vespa ColBERT embedder.
The guide illustrates how to feed and query using a single passage representation:
- Compress token vectors using binarization compatible with Vespa's
unpack_bitsused in ranking. This implements the binarization of token-level vectors usingnumpy. - Use Vespa hex feed format for binary vectors.
- Query examples.
As a bonus, this also demonstrates how to use ColBERT end-to-end with Vespa for both retrieval and ranking. The retrieval step searches the binary token-level representations using hamming distance. This uses 32 nearestNeighbor operators in the same query, each finding 100 nearest hits in hamming space. Then, the results are re-ranked using the full-blown MaxSim calculation.
ColBERT token-level embeddings:
-
Simple hybrid search with ColBERT
uses a single vector embedding model for retrieval and ColBERT (multi-token vector representation) for re-ranking.
This semantic search application demonstrates the colbert-embedder
and the tensor expressions for ColBERT MaxSim.
It also features reciprocal rank fusion
to fuse different rankings.
-
Long-Context ColBERT
demonstrates Long-Context ColBERT (multi-token vector representation) with extended context windows for long-document retrieval,
as announced in Vespa Long-Context ColBERT.
The app demonstrates the colbert-embedder
and the tensor expressions for performing two types of extended ColBERT late-interaction for long-context retrieval.
This app uses trec-eval for evaluation using
nDCG.
-
Pyvespa: Standalone ColBERT + Vespa for long-context ranking
is a guide on how to use the ColBERT package to produce token-level vectors,
as an alternative to using the native Vespa ColBERT embedder.
It illustrates how to feed multiple passages per Vespa document (long-context):
- Compress token vectors using binarization that is compatible with Vespa's
unpack_bits. - Use Vespa hex feed format for binary vectors with mixed vespa tensors.
- How to query Vespa with the ColBERT query tensor representation.
- Compress token vectors using binarization that is compatible with Vespa's
Pyvespa: LightGBM: Training the model with Vespa features
deploys and uses a LightGBM model in a Vespa application.
The tutorial runs through how to:
- Train a LightGBM classification model with variable names supported by Vespa.
- Create Vespa application package files and export them to an application folder.
- Export the trained LightGBM model to the Vespa application folder.
- Deploy the Vespa application using the application folder.
- Feed data to the Vespa application.
- Assert that the LightGBM predictions from the deployed model are correct.
Pyvespa: LightGBM: Mapping model features to Vespa features
shows how to deploy a LightGBM model with feature names that do not match Vespa feature names.
In addition to the steps in the app above, this tutorial:
- Trains a LightGBM classification model with generic feature names that will not be available in the Vespa application.
- Creates an application package and includes a mapping from Vespa feature names to LightGBM model feature names.
Pyvespa: Feeding performance
intends to shine some light on the different modes of feeding documents to Vespa, looking at 4 different methods:
- Using
VespaSync - Using
VespaAsync - Using
feed_iterable() - Using Vespa CLI
Use Feeding to Vespa Cloud
to test feeding using Vespa Cloud.
The e-commerce application is an end-to-end shopping engine,
using the Amazon product data set.
This use case bundles a front-end application.
It demonstrates building next-generation E-commerce Search using Vespa,
and is a good intro to using the Vespa Cloud CI/CD tests.
Data in e-commerce applications is structured,
so Gradient Boosted Decision Trees (GBDT)
models are popular in this domain.
Try Vespa Product Ranking for using
learning-to-rank (LTR) techniques (using XGBoost and LightGBM)
for improving product search ranking.
In Vespa, faceting (the attribute filtering) is called grouping.
Grouping Results
is a quick intro to implementing faceting/grouping in Vespa.
Recommendations are integral to e-commerce applications.
The recommendation tutorial is a good starting point.
Finally, search as you type and query suggestions lets users quickly create good queries.
Incremental Search shows search-as-you-type functionality,
where for each keystroke of the user, it retrieves matching documents.
It also demonstrates search suggestions (query auto-completion).
Stateless model evaluation demonstrates
using Vespa as a stateless ML model inference server
where Vespa takes care of distributing ML models to multiple serving containers,
offering horizontal scaling and safe deployment.
It features model versioning and a feature processing pipeline,
as well as using custom code in Searchers,
Document Processors and
Request Handlers.
Vespa Documentation Search
is the search application that powers search.vespa.ai -
refer to this for GitHub Actions automation.
This sample app is a good start for automated deployments,
as it has system, staging and production test examples.
It uses the Document API
both for regular PUT operations but also for UPDATE with create-if-nonexistent.
It also has Vespa Components
for custom code.
cord19.vespa.ai is a full-featured application,
based on the Covid-19 Open Research Dataset:
- cord-19: frontend
- cord-19-search: search backend
This application uses embeddings to implement "similar documents" search.
Note: Applications with pom.xml are Java/Maven projects and must be built before deployment. Refer to the Developer Guide for more information.
Contribute to the Vespa sample applications.
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for sample-apps
Similar Open Source Tools
sample-apps
Vespa is an open-source search and AI engine that provides a unified platform for building and deploying search and AI applications. Vespa sample applications showcase various use cases and features of Vespa, including basic search, recommendation, semantic search, image search, text ranking, e-commerce search, question answering, search-as-you-type, and ML inference serving.
CosmosAIGraph
CosmosAIGraph is an AI-powered graph and RAG implementation of OmniRAG pattern, utilizing Azure Cosmos DB and other sources. It includes presentations, reference application documentation, FAQs, and a reference dataset of Python libraries pre-vectorized. The project focuses on Azure Cosmos DB for NoSQL and Apache Jena implementation for the in-memory RDF graph. It provides DockerHub images, with plans to add RBAC and Microsoft Entra ID/AAD authentication support, update AI model to gpt-4.5, and offer generic graph examples with a graph generation solution.
kdbai-samples
KDB.AI is a time-based vector database that allows developers to build scalable, reliable, and real-time applications by providing advanced search, recommendation, and personalization for Generative AI applications. It supports multiple index types, distance metrics, top-N and metadata filtered retrieval, as well as Python and REST interfaces. The repository contains samples demonstrating various use-cases such as temporal similarity search, document search, image search, recommendation systems, sentiment analysis, and more. KDB.AI integrates with platforms like ChatGPT, Langchain, and LlamaIndex. The setup steps require Unix terminal, Python 3.8+, and pip installed. Users can install necessary Python packages and run Jupyter notebooks to interact with the samples.
PulsarRPA
PulsarRPA is a high-performance, distributed, open-source Robotic Process Automation (RPA) framework designed to handle large-scale RPA tasks with ease. It provides a comprehensive solution for browser automation, web content understanding, and data extraction. PulsarRPA addresses challenges of browser automation and accurate web data extraction from complex and evolving websites. It incorporates innovative technologies like browser rendering, RPA, intelligent scraping, advanced DOM parsing, and distributed architecture to ensure efficient, accurate, and scalable web data extraction. The tool is open-source, customizable, and supports cutting-edge information extraction technology, making it a preferred solution for large-scale web data extraction.
nucliadb
NucliaDB is a robust database that allows storing and searching on unstructured data. It is an out of the box hybrid search database, utilizing vector, full text and graph indexes. NucliaDB is written in Rust and Python. We designed it to index large datasets and provide multi-teanant support. When utilizing NucliaDB with Nuclia cloud, you are able to the power of an NLP database without the hassle of data extraction, enrichment and inference. We do all the hard work for you.
llm-search
pyLLMSearch is an advanced RAG system that offers a convenient question-answering system with a simple YAML-based configuration. It enables interaction with multiple collections of local documents, with improvements in document parsing, hybrid search, chat history, deep linking, re-ranking, customizable embeddings, and more. The package is designed to work with custom Large Language Models (LLMs) from OpenAI or installed locally. It supports various document formats, incremental embedding updates, dense and sparse embeddings, multiple embedding models, 'Retrieve and Re-rank' strategy, HyDE (Hypothetical Document Embeddings), multi-querying, chat history, and interaction with embedded documents using different models. It also offers simple CLI and web interfaces, deep linking, offline response saving, and an experimental API.
bocoel
BoCoEL is a tool that leverages Bayesian Optimization to efficiently evaluate large language models by selecting a subset of the corpus for evaluation. It encodes individual entries into embeddings, uses Bayesian optimization to select queries, retrieves from the corpus, and provides easily managed evaluations. The tool aims to reduce computation costs during evaluation with a dynamic budget, supporting models like GPT2, Pythia, and LLAMA through integration with Hugging Face transformers and datasets. BoCoEL offers a modular design and efficient representation of the corpus to enhance evaluation quality.
R1-Searcher
R1-searcher is a tool designed to incentivize the search capability in large reasoning models (LRMs) via reinforcement learning. It enables LRMs to invoke web search and obtain external information during the reasoning process by utilizing a two-stage outcome-supervision reinforcement learning approach. The tool does not require instruction fine-tuning for cold start and is compatible with existing Base LLMs or Chat LLMs. It includes training code, inference code, model checkpoints, and a detailed technical report.
LLMs-from-scratch
This repository contains the code for coding, pretraining, and finetuning a GPT-like LLM and is the official code repository for the book Build a Large Language Model (From Scratch). In _Build a Large Language Model (From Scratch)_, you'll discover how LLMs work from the inside out. In this book, I'll guide you step by step through creating your own LLM, explaining each stage with clear text, diagrams, and examples. The method described in this book for training and developing your own small-but-functional model for educational purposes mirrors the approach used in creating large-scale foundational models such as those behind ChatGPT.
hi-ml
The Microsoft Health Intelligence Machine Learning Toolbox is a repository that provides low-level and high-level building blocks for Machine Learning / AI researchers and practitioners. It simplifies and streamlines work on deep learning models for healthcare and life sciences by offering tested components such as data loaders, pre-processing tools, deep learning models, and cloud integration utilities. The repository includes two Python packages, 'hi-ml-azure' for helper functions in AzureML, 'hi-ml' for ML components, and 'hi-ml-cpath' for models and workflows related to histopathology images.
NeMo
NeMo Framework is a generative AI framework built for researchers and pytorch developers working on large language models (LLMs), multimodal models (MM), automatic speech recognition (ASR), and text-to-speech synthesis (TTS). The primary objective of NeMo is to provide a scalable framework for researchers and developers from industry and academia to more easily implement and design new generative AI models by being able to leverage existing code and pretrained models.
graphrag-local-ollama
GraphRAG Local Ollama is a repository that offers an adaptation of Microsoft's GraphRAG, customized to support local models downloaded using Ollama. It enables users to leverage local models with Ollama for large language models (LLMs) and embeddings, eliminating the need for costly OpenAPI models. The repository provides a simple setup process and allows users to perform question answering over private text corpora by building a graph-based text index and generating community summaries for closely-related entities. GraphRAG Local Ollama aims to improve the comprehensiveness and diversity of generated answers for global sensemaking questions over datasets.
NeMo
NVIDIA NeMo Framework is a scalable and cloud-native generative AI framework built for researchers and PyTorch developers working on Large Language Models (LLMs), Multimodal Models (MMs), Automatic Speech Recognition (ASR), Text to Speech (TTS), and Computer Vision (CV) domains. It is designed to help you efficiently create, customize, and deploy new generative AI models by leveraging existing code and pre-trained model checkpoints.
burn
Burn is a new comprehensive dynamic Deep Learning Framework built using Rust with extreme flexibility, compute efficiency and portability as its primary goals.
HuggingFists
HuggingFists is a low-code data flow tool that enables convenient use of LLM and HuggingFace models. It provides functionalities similar to Langchain, allowing users to design, debug, and manage data processing workflows, create and schedule workflow jobs, manage resources environment, and handle various data artifact resources. The tool also offers account management for users, allowing centralized management of data source accounts and API accounts. Users can access Hugging Face models through the Inference API or locally deployed models, as well as datasets on Hugging Face. HuggingFists supports breakpoint debugging, branch selection, function calls, workflow variables, and more to assist users in developing complex data processing workflows.
DistiLlama
DistiLlama is a Chrome extension that leverages a locally running Large Language Model (LLM) to perform various tasks, including text summarization, chat, and document analysis. It utilizes Ollama as the locally running LLM instance and LangChain for text summarization. DistiLlama provides a user-friendly interface for interacting with the LLM, allowing users to summarize web pages, chat with documents (including PDFs), and engage in text-based conversations. The extension is easy to install and use, requiring only the installation of Ollama and a few simple steps to set up the environment. DistiLlama offers a range of customization options, including the choice of LLM model and the ability to configure the summarization chain. It also supports multimodal capabilities, allowing users to interact with the LLM through text, voice, and images. DistiLlama is a valuable tool for researchers, students, and professionals who seek to leverage the power of LLMs for various tasks without compromising data privacy.
For similar tasks
LLMStack
LLMStack is a no-code platform for building generative AI agents, workflows, and chatbots. It allows users to connect their own data, internal tools, and GPT-powered models without any coding experience. LLMStack can be deployed to the cloud or on-premise and can be accessed via HTTP API or triggered from Slack or Discord.
ai-guide
This guide is dedicated to Large Language Models (LLMs) that you can run on your home computer. It assumes your PC is a lower-end, non-gaming setup.
onnxruntime-genai
ONNX Runtime Generative AI is a library that provides the generative AI loop for ONNX models, including inference with ONNX Runtime, logits processing, search and sampling, and KV cache management. Users can call a high level `generate()` method, or run each iteration of the model in a loop. It supports greedy/beam search and TopP, TopK sampling to generate token sequences, has built in logits processing like repetition penalties, and allows for easy custom scoring.
jupyter-ai
Jupyter AI connects generative AI with Jupyter notebooks. It provides a user-friendly and powerful way to explore generative AI models in notebooks and improve your productivity in JupyterLab and the Jupyter Notebook. Specifically, Jupyter AI offers: * An `%%ai` magic that turns the Jupyter notebook into a reproducible generative AI playground. This works anywhere the IPython kernel runs (JupyterLab, Jupyter Notebook, Google Colab, Kaggle, VSCode, etc.). * A native chat UI in JupyterLab that enables you to work with generative AI as a conversational assistant. * Support for a wide range of generative model providers, including AI21, Anthropic, AWS, Cohere, Gemini, Hugging Face, NVIDIA, and OpenAI. * Local model support through GPT4All, enabling use of generative AI models on consumer grade machines with ease and privacy.
khoj
Khoj is an open-source, personal AI assistant that extends your capabilities by creating always-available AI agents. You can share your notes and documents to extend your digital brain, and your AI agents have access to the internet, allowing you to incorporate real-time information. Khoj is accessible on Desktop, Emacs, Obsidian, Web, and Whatsapp, and you can share PDF, markdown, org-mode, notion files, and GitHub repositories. You'll get fast, accurate semantic search on top of your docs, and your agents can create deeply personal images and understand your speech. Khoj is self-hostable and always will be.
langchain_dart
LangChain.dart is a Dart port of the popular LangChain Python framework created by Harrison Chase. LangChain provides a set of ready-to-use components for working with language models and a standard interface for chaining them together to formulate more advanced use cases (e.g. chatbots, Q&A with RAG, agents, summarization, extraction, etc.). The components can be grouped into a few core modules: * **Model I/O:** LangChain offers a unified API for interacting with various LLM providers (e.g. OpenAI, Google, Mistral, Ollama, etc.), allowing developers to switch between them with ease. Additionally, it provides tools for managing model inputs (prompt templates and example selectors) and parsing the resulting model outputs (output parsers). * **Retrieval:** assists in loading user data (via document loaders), transforming it (with text splitters), extracting its meaning (using embedding models), storing (in vector stores) and retrieving it (through retrievers) so that it can be used to ground the model's responses (i.e. Retrieval-Augmented Generation or RAG). * **Agents:** "bots" that leverage LLMs to make informed decisions about which available tools (such as web search, calculators, database lookup, etc.) to use to accomplish the designated task. The different components can be composed together using the LangChain Expression Language (LCEL).
danswer
Danswer is an open-source Gen-AI Chat and Unified Search tool that connects to your company's docs, apps, and people. It provides a Chat interface and plugs into any LLM of your choice. Danswer can be deployed anywhere and for any scale - on a laptop, on-premise, or to cloud. Since you own the deployment, your user data and chats are fully in your own control. Danswer is MIT licensed and designed to be modular and easily extensible. The system also comes fully ready for production usage with user authentication, role management (admin/basic users), chat persistence, and a UI for configuring Personas (AI Assistants) and their Prompts. Danswer also serves as a Unified Search across all common workplace tools such as Slack, Google Drive, Confluence, etc. By combining LLMs and team specific knowledge, Danswer becomes a subject matter expert for the team. Imagine ChatGPT if it had access to your team's unique knowledge! It enables questions such as "A customer wants feature X, is this already supported?" or "Where's the pull request for feature Y?"
infinity
Infinity is an AI-native database designed for LLM applications, providing incredibly fast full-text and vector search capabilities. It supports a wide range of data types, including vectors, full-text, and structured data, and offers a fused search feature that combines multiple embeddings and full text. Infinity is easy to use, with an intuitive Python API and a single-binary architecture that simplifies deployment. It achieves high performance, with 0.1 milliseconds query latency on million-scale vector datasets and up to 15K QPS.
For similar jobs
sweep
Sweep is an AI junior developer that turns bugs and feature requests into code changes. It automatically handles developer experience improvements like adding type hints and improving test coverage.
teams-ai
The Teams AI Library is a software development kit (SDK) that helps developers create bots that can interact with Teams and Microsoft 365 applications. It is built on top of the Bot Framework SDK and simplifies the process of developing bots that interact with Teams' artificial intelligence capabilities. The SDK is available for JavaScript/TypeScript, .NET, and Python.
ai-guide
This guide is dedicated to Large Language Models (LLMs) that you can run on your home computer. It assumes your PC is a lower-end, non-gaming setup.
classifai
Supercharge WordPress Content Workflows and Engagement with Artificial Intelligence. Tap into leading cloud-based services like OpenAI, Microsoft Azure AI, Google Gemini and IBM Watson to augment your WordPress-powered websites. Publish content faster while improving SEO performance and increasing audience engagement. ClassifAI integrates Artificial Intelligence and Machine Learning technologies to lighten your workload and eliminate tedious tasks, giving you more time to create original content that matters.
chatbot-ui
Chatbot UI is an open-source AI chat app that allows users to create and deploy their own AI chatbots. It is easy to use and can be customized to fit any need. Chatbot UI is perfect for businesses, developers, and anyone who wants to create a chatbot.
BricksLLM
BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM: * Set LLM usage limits for users on different pricing tiers * Track LLM usage on a per user and per organization basis * Block or redact requests containing PIIs * Improve LLM reliability with failovers, retries and caching * Distribute API keys with rate limits and cost limits for internal development/production use cases * Distribute API keys with rate limits and cost limits for students
uAgents
uAgents is a Python library developed by Fetch.ai that allows for the creation of autonomous AI agents. These agents can perform various tasks on a schedule or take action on various events. uAgents are easy to create and manage, and they are connected to a fast-growing network of other uAgents. They are also secure, with cryptographically secured messages and wallets.
griptape
Griptape is a modular Python framework for building AI-powered applications that securely connect to your enterprise data and APIs. It offers developers the ability to maintain control and flexibility at every step. Griptape's core components include Structures (Agents, Pipelines, and Workflows), Tasks, Tools, Memory (Conversation Memory, Task Memory, and Meta Memory), Drivers (Prompt and Embedding Drivers, Vector Store Drivers, Image Generation Drivers, Image Query Drivers, SQL Drivers, Web Scraper Drivers, and Conversation Memory Drivers), Engines (Query Engines, Extraction Engines, Summary Engines, Image Generation Engines, and Image Query Engines), and additional components (Rulesets, Loaders, Artifacts, Chunkers, and Tokenizers). Griptape enables developers to create AI-powered applications with ease and efficiency.