denser-retriever

denser-retriever

An enterprise-grade AI retriever designed to streamline AI integration into your applications, ensuring cutting-edge accuracy.

Stars: 119

Visit
 screenshot

Denser Retriever is an enterprise-grade AI retriever designed to streamline AI integration into applications, combining keyword-based searches, vector databases, and machine learning rerankers using xgboost. It provides state-of-the-art accuracy on MTEB Retrieval benchmarking and supports various heterogeneous retrievers for end-to-end applications like chatbots and semantic search.

README:

denser logo Denser Retriever

Python Version Dependencies Status

Code style: ruff Security: bandit Pre-commit Semantic Versions License Coverage Report

An enterprise-grade AI retriever designed to streamline AI integration into your applications, ensuring cutting-edge accuracy.

📝 Description

Denser Retriever combines multiple search technologies into a single platform. It utilizes gradient boosting ( xgboost) machine learning technique to combine:

  • Keyword-based searches that focus on fetching precisely what the query mentions.
  • Vector databases that are great for finding a wide range of potentially relevant answers.
  • Machine Learning rerankers that fine-tune the results to ensure the most relevant answers top the list.

Our experiments on MTEB datasets show that the combination of keyword search, vector search and a reranker via an xgboost model (denoted as ES+VS+RR_n) can significantly improve the vector search (VS) baseline.

mteb_ndcg_plot

🚀 Features

The initial release of Denser Retriever provides the following features.

  • Supporting heterogeneous retrievers such as keyword search, vector search, and ML model reranking
  • Leveraging xgboost ML technique to effectively combine heterogeneous retrievers
  • State-of-the-art accuracy on MTEB Retrieval benchmarking
  • Demonstrating how to use Denser retriever to power an end-to-end applications such as chatbot and semantic search

📦 Installation

We recommend installing Python via Anaconda, as we have received feedback about issues with Numpy installation when using the installer from https://www.python.org/downloads/. We are working on providing a solution to this problem. To install Denser Retriever, you can run:

Pip

pip install git+https://github.com/denser-org/denser-retriever.git#main

Poetry

poetry add git+https://github.com/denser-org/denser-retriever.git#main

📃 Documentation

The official documentation is hosted on retriever.denser.ai. Click here to get started.

👨🏼‍💻 Development

You can start developing Denser Retriever on your local machine.

See DEVELOPMENT.md for more details.

🛡 License

License

This project is licensed under the terms of the MIT license. See LICENSE for more details.

📃 Citation

@misc{denser-retriever,
  author = {denser-org},
  title = {An enterprise-grade AI retriever designed to streamline AI integration into your applications, ensuring cutting-edge accuracy.},
  year = {2024},
  publisher = {GitHub},
  journal = {GitHub repository},
  howpublished = {\url{https://github.com/denser-org/denser-retriever}}
}

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for denser-retriever

Similar Open Source Tools

For similar tasks

For similar jobs