kalavai-client

kalavai-client

Turns devices into a scalable LLM platform

Stars: 100

Visit
 screenshot

Kalavai is an open-source platform that transforms everyday devices into an AI supercomputer by aggregating resources from multiple machines. It facilitates matchmaking of resources for large AI projects, making AI hardware accessible and affordable. Users can create local and public pools, connect with the community's resources, and share computing power. The platform aims to be a management layer for research groups and organizations, enabling users to unlock the power of existing hardware without needing a devops team. Kalavai CLI tool helps manage both versions of the platform.

README:

Kalavai logo

GitHub Release PyPI - Downloads GitHub contributors GitHub License GitHub Repo stars Dynamic JSON Badge Signup

⭐⭐⭐ Kalavai and our LLM pools are open source, and free to use in both commercial and non-commercial purposes. If you find it useful, consider supporting us by giving a star to our GitHub project, joining our discord channel, follow our Substack and give us a review on Product Hunt.

Kalavai: turn your devices into a scalable LLM platform

Taming the adoption of Large Language Models

Kalavai is an open source tool that turns everyday devices into your very own LLM platform. It aggregates resources from multiple machines, including desktops and laptops, and is compatible with most model engines to make LLM deployment and orchestration simple and reliable.

Kalavai - The first platform to crowdsource AI computation | Product Hunt

What can Kalavai do?

Kalavai's goal is to make using LLMs in real applications accessible and affordable to all. It's a magic box that integrates all the components required to make LLM useful in the age of massive computing, from sourcing computing power, managing distributed infrastructure and storage, using industry-standard model engines and orchestration of LLMs.

Aggregate multiple devices in an LLM pool

https://github.com/user-attachments/assets/4be59886-1b76-4400-ab5c-c803e3e414ec

Deploy LLMs across the pool

https://github.com/user-attachments/assets/ea57a2ab-3924-4097-be2a-504e0988fbb1

Single point of entry for all models (GUI + API)

https://github.com/user-attachments/assets/7df73bbc-d129-46aa-8ce5-0735177dedeb

Self-hosted LLM pools

https://github.com/user-attachments/assets/0d2316f3-79ea-46ac-b41e-8ef720f52672

News updates

Support for LLM engines

We currently support out of the box the following LLM engines:

Coming soon:

Not what you were looking for? Tell us what engines you'd like to see.

Kalavai is at an early stage of its development. We encourage people to use it and give us feedback! Although we are trying to minimise breaking changes, these may occur until we have a stable version (v1.0).

Want to know more?

Getting started

The kalavai client is the main tool to interact with the Kalavai platform, to create and manage both local and public pools and also to interact with them (e.g. deploy models). Let's go over its installation.

From release v0.5.0, you can now install kalavai client in non-worker computers. You can run a pool on a set of machines and have the client on a remote computer from which you access the LLM pool. Because the client only requires having python installed, this means more computers are now supported to run it.

Requirements for a worker machine

Requirements to run the client

  • Python 3.10+

If you see the following error:

fatal error: Python.h: No such file or directory | #include <Python.h>

Make sure you also install python3-dev package. For ubuntu distros:

sudo apt install python3-dev

Install the client

The client is a python package and can be installed with one command:

pip install kalavai-client

Public LLM pools: crowdsource community resources

This is the easiest and most powerful way to experience Kalavai. It affords users the full resource capabilities of the community and access to all its deployed LLMs, via an OpenAI-compatible endpoint as well as a UI-based playground.

Check out our guide on how to join and start deploying LLMs.

Createa a local, private LLM pool

Kalavai is free to use, no caps, for both commercial and non-commercial purposes. All you need to get started is one or more computers that can see each other (i.e. within the same network), and you are good to go. If you wish to join computers in different locations / networks, check managed kalavai.

1. Start a seed node

Simply use the client to start your seed node:

kalavai pool start <pool-name>

Now you are ready to add worker nodes to this seed. To do so, generate a joining token:

$ kalavai pool token --user

Join token: <token>

2. Add worker nodes

Increase the power of your AI pool by inviting others to join.

Copy the joining token. On the worker node, run:

kalavai pool join <token>

Enough already, let's run stuff!

Check our examples to put your new AI pool to good use!

Compatibility matrix

If your system is not currently supported, open an issue and request it. We are expanding this list constantly.

OS compatibility

Since worker nodes run inside docker, any machine that can run docker should be compatible with Kalavai. Here are instructions for linux, Windows and MacOS.

The kalavai client, which controls and access pools, can be installed on any machine that has python 3.10+.

Hardware compatibility:

Roadmap

  • [x] Kalavai client on Linux
  • [x] [TEMPLATE] Distributed LLM deployment
  • [x] Kalavai client on Windows (with WSL2)
  • [x] Public LLM pools
  • [x] Self-hosted LLM pools
  • [x] Collaborative LLM deployment
  • [x] Ray cluster support
  • [x] Kalavai client on Mac
  • [ ] [TEMPLATE] GPUStack support
  • [ ] [TEMPLATE] exo support
  • [ ] Support for AMD GPUs
  • [x] Docker install path

Anything missing here? Give us a shout in the discussion board

Contribute

Star History

Star History Chart

Build from source

Requirements

Python version <= 3.12.

sudo add-apt-repository ppa:deadsnakes/ppa
sudo apt update
sudo apt install python3.10 python3.10-dev
virtualenv -p python3.10 env
source env/bin/activate
sudo apt install  python3.10-venv python3.10-dev -y
pip install -e .[dev]

Build python wheels:

bash publish.sh build

Unit tests

To run the unit tests, use:

python -m unittest

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for kalavai-client

Similar Open Source Tools

For similar tasks

For similar jobs