LLMUnity

Create characters in Unity with LLMs!

Stars: 581

Visit

LLM for Unity enables seamless integration of Large Language Models (LLMs) within the Unity engine, allowing users to create intelligent characters for immersive player interactions. The tool supports major LLM models, runs locally without internet access, offers fast inference on CPU and GPU, and is easy to set up with a single line of code. It is free for both personal and commercial use, tested on Unity 2021 LTS, 2022 LTS, and 2023. Users can build multiple AI characters efficiently, use remote servers for processing, and customize model settings for text generation.

README:

Create characters in Unity with LLMs!

LLM for Unity enables seamless integration of Large Language Models (LLMs) within the Unity engine.
It allows to create intelligent characters that your players can interact with for an immersive experience.
LLM for Unity is built on top of the awesome llama.cpp and llamafile libraries.

_{At a glance  •
How to help  •
Games using LLM for Unity  •
Setup  •
How to use  •
Examples  •
LLM model management  •
Options  •
License}

At a glance

💻 Cross-platform! Windows, Linux, macOS and Android
🏠 Runs locally without internet access. No data ever leaves the game!
⚡ Blazing fast inference on CPU and GPU (Nvidia, AMD, Apple Metal)
🤗 Supports all major LLM models
🔧 Easy to setup, call with a single line of code
💰 Free to use for both personal and commercial purposes

🧪 Tested on Unity: 2021 LTS, 2022 LTS, 2023
🚦 Upcoming Releases

How to help

⭐ Star the repo, leave us a review and spread the word about the project!
Join us at Discord and say hi.
Contribute by submitting feature requests, bugs or even your own PR.
this work to allow even cooler features!

Games using LLM for Unity

Setup

Method 1: Install using the asset store

Open the LLM for Unity asset page and click Add to My Assets
Open the Package Manager in Unity: Window > Package Manager
Select the Packages: My Assets option from the drop-down
Select the LLM for Unity package, click Download and then Import

Method 2: Install using the GitHub repo:

Open the Package Manager in Unity: Window > Package Manager
Click the + button and select Add package from git URL
Use the repository URL https://github.com/undreamai/LLMUnity.git and click Add

How to use

First you will setup the LLM for your game 🏎:

Create an empty GameObject.
In the GameObject Inspector click Add Component and select the LLM script.
Download one of the default models with the Download Model button (~GBs).
Or load your own .gguf model with the Load model button (see LLM model management).

Then you can setup each of your characters as follows 🙋‍♀️:

Create an empty GameObject for the character.
In the GameObject Inspector click Add Component and select the LLMCharacter script.
Define the role of your AI in the Prompt. You can define the name of the AI (AI Name) and the player (Player Name).
(Optional) Select the LLM constructed above in the LLM field if you have more than one LLM GameObjects.

You can also adjust the LLM and character settings according to your preference (see Options).

In your script you can then use it as follows 🦄:

using LLMUnity;

public class MyScript {
  public LLMCharacter llmCharacter;
  
  void HandleReply(string reply){
    // do something with the reply from the model
    Debug.Log(reply);
  }
  
  void Game(){
    // your game function
    ...
    string message = "Hello bot!";
    _ = llmCharacter.Chat(message, HandleReply);
    ...
  }
}

You can also specify a function to call when the model reply has been completed.
This is useful if the Stream option is enabled for continuous output from the model (default behaviour):

  void ReplyCompleted(){
    // do something when the reply from the model is complete
    Debug.Log("The AI replied");
  }
  
  void Game(){
    // your game function
    ...
    string message = "Hello bot!";
    _ = llmCharacter.Chat(message, HandleReply, ReplyCompleted);
    ...
  }

To stop the chat without waiting for its completion you can use:

    llmCharacter.CancelRequests();

Finally, in the Inspector of the GameObject of your script, select the LLMCharacter GameObject created above as the llmCharacter property.

That's all ✨!

You can also:

Build a mobile app on Android

To build an Android app you need to specify the IL2CPP scripting backend and the ARM64 as the target architecture in the player settings.
These settings can be accessed from the Edit > Project Settings menu within the Player > Other Settings section.

It is also a good idea to enable the Download on Build option in the LLM GameObject to download the model on launch in order to keep the app size small.

Save / Load your chat history

To automatically save / load your chat history, you can specify the Save parameter of the LLMCharacter to the filename (or relative path) of your choice. The file is saved in the persistentDataPath folder of Unity. This also saves the state of the LLM which means that the previously cached prompt does not need to be recomputed.

To manually save your chat history, you can use:

    llmCharacter.Save("filename");

and to load the history:

    llmCharacter.Load("filename");

where filename the filename or relative path of your choice.

Process the prompt at the beginning of your app for faster initial processing time

  void WarmupCompleted(){
    // do something when the warmup is complete
    Debug.Log("The AI is nice and ready");
  }

  void Game(){
    // your game function
    ...
    _ = llmCharacter.Warmup(WarmupCompleted);
    ...
  }

Decide whether or not to add the message to the chat/prompt history

The last argument of the Chat function is a boolean that specifies whether to add the message to the history (default: true):

  void Game(){
    // your game function
    ...
    string message = "Hello bot!";
    _ = llmCharacter.Chat(message, HandleReply, ReplyCompleted, false);
    ...
  }

Use pure text completion

  void Game(){
    // your game function
    ...
    string message = "The cat is away";
    _ = llmCharacter.Complete(message, HandleReply, ReplyCompleted);
    ...
  }

Wait for the reply before proceeding to the next lines of code

For this you can use the async/await functionality:

  async void Game(){
    // your game function
    ...
    string message = "Hello bot!";
    string reply = await llmCharacter.Chat(message, HandleReply, ReplyCompleted);
    Debug.Log(reply);
    ...
  }

Add a LLM / LLMCharacter component programmatically

using UnityEngine;
using LLMUnity;

public class MyScript : MonoBehaviour
{
    LLM llm;
    LLMCharacter llmCharacter;

    async void Start()
    {
        // disable gameObject so that theAwake is not called immediately
        gameObject.SetActive(false);

        // Add an LLM object
        llm = gameObject.AddComponent<LLM>();
        // set the model using the filename of the model.
        // The model needs to be added to the LLM model manager (see LLM model management) by loading or downloading it.
        // Otherwise the model file can be copied directly inside the StreamingAssets folder.
        llm.SetModel("Phi-3-mini-4k-instruct-q4.gguf");
        // optional: you can also set loras in a similar fashion and set their weights (if needed)
        llm.AddLora("my-lora.gguf");
        llm.SetLoraWeight(0.5f);
        // optional: you can set the chat template of the model if it is not correctly identified
        // You can find a list of chat templates in the ChatTemplate.templates.Keys
        llm.SetTemplate("phi-3");
        // optional: set number of threads
        llm.numThreads = -1;
        // optional: enable GPU by setting the number of model layers to offload to it
        llm.numGPULayers = 10;

        // Add an LLMCharacter object
        llmCharacter = gameObject.AddComponent<LLMCharacter>();
        // set the LLM object that handles the model
        llmCharacter.llm = llm;
        // set the character prompt
        llmCharacter.SetPrompt("A chat between a curious human and an artificial intelligence assistant.");
        // set the AI and player name
        llmCharacter.AIName = "AI";
        llmCharacter.playerName = "Human";
        // optional: set streaming to false to get the complete result in one go
        // llmCharacter.stream = true;
        // optional: set a save path
        // llmCharacter.save = "AICharacter1";
        // optional: enable the save cache to avoid recomputation when loading a save file (requires ~100 MB)
        // llmCharacter.saveCache = true;
        // optional: set a grammar
        // await llmCharacter.SetGrammar("json.gbnf");

        // re-enable gameObject
        gameObject.SetActive(true);
    }
}

Use a remote server

You can use a remote server to carry out the processing and implement characters that interact with it.

Create the server
To create the server:

Create a project with a GameObject using the LLM script as described above
Enable the Remote option of the LLM and optionally configure the server parameters: port, API key, SSL certificate, SSL key
Build and run to start the server

Alternatively you can use a server binary for easier deployment:

Run the above scene from the Editor and copy the command from the Debug messages (starting with "Server command:")
Download the server binaries and DLLs and extract them into the same folder
Find the architecture you are interested in from the folder above e.g. for Windows and CUDA use the windows-cuda-cu12.2.0.
You can also check the architecture that works for your system from the Debug messages (starting with "Using architecture").
From command line change directory to the architecture folder selected and start the server by running the command copied from above.

Create the characters
Create a second project with the game characters using the LLMCharacter script as described above. Enable the Remote option and configure the host with the IP address (starting with "http://") and port of the server.

Compute embeddings using a LLM

The Embeddings function can be used to obtain the emdeddings of a phrase:

    List<float> embeddings = await llmCharacter.Embeddings("hi, how are you?");

A detailed documentation on function level can be found here:

Examples

The Samples~ folder contains several examples of interaction 🤖:

SimpleInteraction: Demonstrates a simple interaction with an AI character
MultipleCharacters: Demonstrates a simple interaction using multiple AI characters
KnowledgeBaseGame: Simple detective game using a knowledge base to provide information to the LLM based on google/mysteryofthreebots
ChatBot: Demonstrates interaction between a player and a AI with a UI similar to a messaging app (see image below)
AndroidDemo: Example Android app with an initial screen with model download progress

To install a sample:

Open the Package Manager: Window > Package Manager
Select the LLM for Unity Package. From the Samples Tab, click Import next to the sample you want to install.

The samples can be run with the Scene.unity scene they contain inside their folder.
In the scene, select the LLM GameObject and click the Download Model button to download a default model or Load model to load your own model (see LLM model management).
Save the scene, run and enjoy!

LLM model management

LLM for Unity implements a model manager that allows to load or download LLMs and ship them directly in your game.
The model manager can be found as part of the LLM GameObject:

You can download models with the Download model button.
LLM for Unity includes different state of the art models built-in for different model sizes, quantised with the Q4_K_M method.
Alternative models can be downloaded from HuggingFace in the .gguf format.
You can download a model locally and load it with the Load model button, or copy the URL in the Download model > Custom URL field to directly download it.
If a HuggingFace model does not provide a gguf file, it can be converted to gguf with this online converter.

The chat template used for constructing the prompts is determined automatically from the model (if a relevant entry exists) or the model name.
If incorrecly identified, you can select another template from the chat template dropdown.

Models added in the model manager are copied to the game during the building process.
You can omit a model from being built in by deselecting the "Build" checkbox.
To remove the model (but not delete it from disk) you can click the bin button.
The the path and URL (if downloaded) of each added model is diplayed in the expanded view of the model manager access with the >> button:

You can create lighter builds by selecting the Download on Build option.
Using this option the models will be downloaded the first time the game starts instead of copied in the build.
If you have loaded a model locally you need to set its URL through the expanded view, otherwise it will be copied in the build.

❕ Before using any model make sure you check their license ❕

Options

LLM Settings

Show/Hide Advanced Options Toggle to show/hide advanced options from below
Log Level select how verbose the log messages are
Use extras select to install and allow the use of extra features (flash attention and IQ quants)

💻 Setup Settings

Remote select to provide remote access to the LLM
Port port to run the LLM server (if Remote is set)
Num Threads number of threads to use (default: -1 = all)
Num GPU Layers number of model layers to offload to the GPU. If set to 0 the GPU is not used. Use a large number i.e. >30 to utilise the GPU as much as possible. Note that higher values of context size will use more VRAM. If the user's GPU is not supported, the LLM will fall back to the CPU
Debug select to log the output of the model in the Unity Editor
Advanced options
- Parallel Prompts number of prompts / slots that can happen in parallel (default: -1 = number of LLMCharacter objects). Note that the context size is divided among the slots.
  If you want to retain as much context for the LLM and don't need all the characters present at the same time, you can set this number and specify the slot for each LLMCharacter object.
e.g. Setting Parallel Prompts to 1 and slot 0 for all LLMCharacter objects will use the full context, but the entire prompt will need to be computed (no caching) whenever a LLMCharacter object is used for chat.
- Dont Destroy On Load select to not destroy the LLM GameObject when loading a new Scene

Server Security Settings

API key API key to use to allow access to requests from LLMCharacter objects (if Remote is set)
Advanced options
- Load SSL certificate allows to load a SSL certificate for end-to-end encryption of requests (if Remote is set). Requires SSL key as well.
- Load SSL key allows to load a SSL key for end-to-end encryption of requests (if Remote is set). Requires SSL certificate as well.
- SSL certificate path the SSL certificate used for end-to-end encryption of requests (if Remote is set).
- SSL key path the SSL key used for end-to-end encryption of requests (if Remote is set).

🤗 Model Settings

Download model click to download one of the default models
Load model click to load your own model in .gguf format
Download on Start enable to downloaded the LLM models the first time the game starts. Alternatively the LLM models wil be copied directly in the build
Context Size size of the prompt context (0 = context size of the model)
This is the number of tokens the model can take as input when generating responses. Higher values use more RAM or VRAM (if using GPU).
Advanced options
- Download lora click to download a LoRA model in .gguf format
- Load lora click to load a LoRA model in .gguf format
- Batch Size batch size for prompt processing (default: 512)
- Model the path of the model being used (relative to the Assets/StreamingAssets folder)
- Chat Template the chat template being used for the LLM
- Lora the path of the LoRAs being used (relative to the Assets/StreamingAssets folder)
- Lora Weights the weights of the LoRAs being used
- Flash Attention click to use flash attention in the model (if Use extras is enabled)

🗨️ Chat Settings

Advanced options
Base Prompt a common base prompt to use across all LLMCharacter objects using the LLM

LLMCharacter Settings

Show/Hide Advanced Options Toggle to show/hide advanced options from below
Log Level select how verbose the log messages are
Use extras select to install and allow the use of extra features (flash attention and IQ quants)

💻 Setup Settings

Remote whether the LLM used is remote or local
LLM the LLM GameObject (if Remote is not set)
Hort ip of the LLM server (if Remote is set)
Port port of the LLM server (if Remote is set)
Num Retries number of HTTP request retries from the LLM server (if Remote is set)
API key API key of the LLM server (if Remote is set)
Save save filename or relative path
If set, the chat history and LLM state (if save cache is enabled) is automatically saved to file specified.
The chat history is saved with a json suffix and the LLM state with a cache suffix.
Both files are saved in the [persistentDataPath folder of Unity](https://docs.unity3d.com/ScriptReference/Application-persistentDataPath.html).
Save Cache select to save the LLM state along with the chat history. The LLM state is typically around 100MB+.
Debug Prompt select to log the constructed prompts in the Unity Editor

🗨️ Chat Settings

Player Name the name of the player
AI Name the name of the AI
Prompt description of the AI role

🤗 Model Settings

Stream select to receive the reply from the model as it is produced (recommended!).
If it is not selected, the full reply from the model is received in one go
Advanced options
- Load grammar click to load a grammar in .gbnf format
- Grammar the path of the grammar being used (relative to the Assets/StreamingAssets folder)
- Cache Prompt save the ongoing prompt from the chat (default: true)
  Saves the prompt while it is being created by the chat to avoid reprocessing the entire prompt every time
- Slot slot of the server to use for computation. Value can be set from 0 to Parallel Prompts-1 (default: -1 = new slot for each character)
- Seed seed for reproducibility. For random results every time use -1
- Num Predict maximum number of tokens to predict (default: 256, -1 = infinity, -2 = until context filled)
  This is the maximum amount of tokens the model will maximum predict. When N tokens are reached the model will stop generating. This means words / sentences might not get finished if this is too low.
- Temperature LLM temperature, lower values give more deterministic answers (default: 0.2)
  The temperature setting adjusts how random the generated responses are. Turning it up makes the generated choices more varied and unpredictable. Turning it down makes the generated responses more predictable and focused on the most likely options.
- Top K top-k sampling (default: 40, 0 = disabled)
  The top k value controls the top k most probable tokens at each step of generation. This value can help fine tune the output and make this adhere to specific patterns or constraints.
- Top P top-p sampling (default: 0.9, 1.0 = disabled)
  The top p value controls the cumulative probability of generated tokens. The model will generate tokens until this theshold (p) is reached. By lowering this value you can shorten output & encourage / discourage more diverse outputs.
- Min P minimum probability for a token to be used (default: 0.05)
  The probability is defined relative to the probability of the most likely token.
- Repeat Penalty control the repetition of token sequences in the generated text (default: 1.1)
  The penalty is applied to repeated tokens.
- Presence Penalty repeated token presence penalty (default: 0.0, 0.0 = disabled)
  Positive values penalize new tokens based on whether they appear in the text so far, increasing the model's likelihood to talk about new topics.
- Frequency Penalty repeated token frequency penalty (default: 0.0, 0.0 = disabled)
  Positive values penalize new tokens based on their existing frequency in the text so far, decreasing the model's likelihood to repeat the same line verbatim.
- Tfs_z: enable tail free sampling with parameter z (default: 1.0, 1.0 = disabled).
- Typical P: enable locally typical sampling with parameter p (default: 1.0, 1.0 = disabled).
- Repeat Last N: last N tokens to consider for penalizing repetition (default: 64, 0 = disabled, -1 = ctx-size).
- Penalize Nl: penalize newline tokens when applying the repeat penalty (default: true).
- Penalty Prompt: prompt for the purpose of the penalty evaluation. Can be either null, a string or an array of numbers representing tokens (default: null = use original prompt).
- Mirostat: enable Mirostat sampling, controlling perplexity during text generation (default: 0, 0 = disabled, 1 = Mirostat, 2 = Mirostat 2.0).
- Mirostat Tau: set the Mirostat target entropy, parameter tau (default: 5.0).
- Mirostat Eta: set the Mirostat learning rate, parameter eta (default: 0.1).
- N Probs: if greater than 0, the response also contains the probabilities of top N tokens for each generated token (default: 0)
- Ignore Eos: enable to ignore end of stream tokens and continue generating (default: false).

License

The license of LLM for Unity is MIT (LICENSE.md) and uses third-party software with MIT and Apache licenses. Some models included in the asset define their own license terms, please review them before using each model. Third-party licenses can be found in the (Third Party Notices.md).

For Tasks:

Click tags to check more tools for each tasks

create characters interact with players build ai characters customize model settings run inference on cpu/gpu

For Jobs:

game developer ai developer software engineer unity developer game designer

Alternative AI tools for LLMUnity

Similar Open Source Tools

LLMUnity

github

: 581

HuggingFaceGuidedTourForMac

HuggingFaceGuidedTourForMac is a guided tour on how to install optimized pytorch and optionally Apple's new MLX, JAX, and TensorFlow on Apple Silicon Macs. The repository provides steps to install homebrew, pytorch with MPS support, MLX, JAX, TensorFlow, and Jupyter lab. It also includes instructions on running large language models using HuggingFace transformers. The repository aims to help users set up their Macs for deep learning experiments with optimized performance.

github

: 79

SpeziLLM

The Spezi LLM Swift Package includes modules that help integrate LLM-related functionality in applications. It provides tools for local LLM execution, usage of remote OpenAI-based LLMs, and LLMs running on Fog node resources within the local network. The package contains targets like SpeziLLM, SpeziLLMLocal, SpeziLLMLocalDownload, SpeziLLMOpenAI, and SpeziLLMFog for different LLM functionalities. Users can configure and interact with local LLMs, OpenAI LLMs, and Fog LLMs using the provided APIs and platforms within the Spezi ecosystem.

github

: 131

rtdl-num-embeddings

This repository provides the official implementation of the paper 'On Embeddings for Numerical Features in Tabular Deep Learning'. It focuses on transforming scalar continuous features into vectors before integrating them into the main backbone of tabular neural networks, showcasing improved performance. The embeddings for continuous features are shown to enhance the performance of tabular DL models and are applicable to various conventional backbones, offering efficiency comparable to Transformer-based models. The repository includes Python packages for practical usage, exploration of metrics and hyperparameters, and reproducing reported results for different algorithms and datasets.

github

: 287

web-llm

WebLLM is a modular and customizable javascript package that directly brings language model chats directly onto web browsers with hardware acceleration. Everything runs inside the browser with no server support and is accelerated with WebGPU. WebLLM is fully compatible with OpenAI API. That is, you can use the same OpenAI API on any open source models locally, with functionalities including json-mode, function-calling, streaming, etc. We can bring a lot of fun opportunities to build AI assistants for everyone and enable privacy while enjoying GPU acceleration.

github

: 13.1k

lantern

Lantern is an open-source PostgreSQL database extension designed to store vector data, generate embeddings, and handle vector search operations efficiently. It introduces a new index type called 'lantern_hnsw' for vector columns, which speeds up 'ORDER BY ... LIMIT' queries. Lantern utilizes the state-of-the-art HNSW implementation called usearch. Users can easily install Lantern using Docker, Homebrew, or precompiled binaries. The tool supports various distance functions, index construction parameters, and operator classes for efficient querying. Lantern offers features like embedding generation, interoperability with pgvector, parallel index creation, and external index graph generation. It aims to provide superior performance metrics compared to other similar tools and has a roadmap for future enhancements such as cloud-hosted version, hardware-accelerated distance metrics, industry-specific application templates, and support for version control and A/B testing of embeddings.

github

: 756

WindowsAgentArena

Windows Agent Arena (WAA) is a scalable Windows AI agent platform designed for testing and benchmarking multi-modal, desktop AI agents. It provides researchers and developers with a reproducible and realistic Windows OS environment for AI research, enabling testing of agentic AI workflows across various tasks. WAA supports deploying agents at scale using Azure ML cloud infrastructure, allowing parallel running of multiple agents and delivering quick benchmark results for hundreds of tasks in minutes.

github

: 147

py-vectara-agentic

The `vectara-agentic` Python library is designed for developing powerful AI assistants using Vectara and Agentic-RAG. It supports various agent types, includes pre-built tools for domains like finance and legal, and enables easy creation of custom AI assistants and agents. The library provides tools for summarizing text, rephrasing text, legal tasks like summarizing legal text and critiquing as a judge, financial tasks like analyzing balance sheets and income statements, and database tools for inspecting and querying databases. It also supports observability via LlamaIndex and Arize Phoenix integration.

github

: 64

generative-models

Generative Models by Stability AI is a repository that provides various generative models for research purposes. It includes models like Stable Video 4D (SV4D) for video synthesis, Stable Video 3D (SV3D) for multi-view synthesis, SDXL-Turbo for text-to-image generation, and more. The repository focuses on modularity and implements a config-driven approach for building and combining submodules. It supports training with PyTorch Lightning and offers inference demos for different models. Users can access pre-trained models like SDXL-base-1.0 and SDXL-refiner-1.0 under a CreativeML Open RAIL++-M license. The codebase also includes tools for invisible watermark detection in generated images.

github

: 23.6k

magic-cli

Magic CLI is a command line utility that leverages Large Language Models (LLMs) to enhance command line efficiency. It is inspired by projects like Amazon Q and GitHub Copilot for CLI. The tool allows users to suggest commands, search across command history, and generate commands for specific tasks using local or remote LLM providers. Magic CLI also provides configuration options for LLM selection and response generation. The project is still in early development, so users should expect breaking changes and bugs.

github

: 497

probsem

ProbSem is a repository that provides a framework to leverage large language models (LLMs) for assigning context-conditional probability distributions over queried strings. It supports OpenAI engines and HuggingFace CausalLM models, and is flexible for research applications in linguistics, cognitive science, program synthesis, and NLP. Users can define prompts, contexts, and queries to derive probability distributions over possible completions, enabling tasks like cloze completion, multiple-choice QA, semantic parsing, and code completion. The repository offers CLI and API interfaces for evaluation, with options to customize models, normalize scores, and adjust temperature for probability distributions.

github

: 72

py-gpt

github

: 502

ScandEval

ScandEval is a framework for evaluating pretrained language models on mono- or multilingual language tasks. It provides a unified interface for benchmarking models on a variety of tasks, including sentiment analysis, question answering, and machine translation. ScandEval is designed to be easy to use and extensible, making it a valuable tool for researchers and practitioners alike.

github

: 69

tonic_validate

Tonic Validate is a framework for the evaluation of LLM outputs, such as Retrieval Augmented Generation (RAG) pipelines. Validate makes it easy to evaluate, track, and monitor your LLM and RAG applications. Validate allows you to evaluate your LLM outputs through the use of our provided metrics which measure everything from answer correctness to LLM hallucination. Additionally, Validate has an optional UI to visualize your evaluation results for easy tracking and monitoring.

github

: 233

mflux

MFLUX is a line-by-line port of the FLUX implementation in the Huggingface Diffusers library to Apple MLX. It aims to run powerful FLUX models from Black Forest Labs locally on Mac machines. The codebase is minimal and explicit, prioritizing readability over generality and performance. Models are implemented from scratch in MLX, with tokenizers from the Huggingface Transformers library. Dependencies include Numpy and Pillow for image post-processing. Installation can be done using `uv tool` or classic virtual environment setup. Command-line arguments allow for image generation with specified models, prompts, and optional parameters. Quantization options for speed and memory reduction are available. LoRA adapters can be loaded for fine-tuning image generation. Controlnet support provides more control over image generation with reference images. Current limitations include generating images one by one, lack of support for negative prompts, and some LoRA adapters not working.

github

: 723

storm

STORM is a LLM system that writes Wikipedia-like articles from scratch based on Internet search. While the system cannot produce publication-ready articles that often require a significant number of edits, experienced Wikipedia editors have found it helpful in their pre-writing stage. **Try out our [live research preview](https://storm.genie.stanford.edu/) to see how STORM can help your knowledge exploration journey and please provide feedback to help us improve the system 🙏!**

github

: 12.1k

For similar tasks

LLMUnity

github

: 581

SillyTavern

SillyTavern is a user interface you can install on your computer (and Android phones) that allows you to interact with text generation AIs and chat/roleplay with characters you or the community create. SillyTavern is a fork of TavernAI 1.2.8 which is under more active development and has added many major features. At this point, they can be thought of as completely independent programs.

github

: 7.9k

agnai

Agnaistic is an AI roleplay chat tool that allows users to interact with personalized characters using their favorite AI services. It supports multiple AI services, persona schema formats, and features such as group conversations, user authentication, and memory/lore books. Agnaistic can be self-hosted or run using Docker, and it provides a range of customization options through its settings.json file. The tool is designed to be user-friendly and accessible, making it suitable for both casual users and developers.

github

: 525

ragdoll-studio

Ragdoll Studio is a platform offering web apps and libraries for interacting with Ragdoll, enabling users to go beyond fine-tuning and create flawless creative deliverables, rich multimedia, and engaging experiences. It provides various modes such as Story Mode for creating and chatting with characters, Vector Mode for producing vector art, Raster Mode for producing raster art, Video Mode for producing videos, Audio Mode for producing audio, and 3D Mode for producing 3D objects. Users can export their content in various formats and share their creations on the community site. The platform consists of a Ragdoll API and a front-end React application for seamless usage.

github

: 156

character-factory

Character Factory is a Python script designed to generate detailed character cards for SillyTavern, TavernAI, TextGenerationWebUI, and more using Large Language Model (LLM) and Stable Diffusion. It streamlines character generation by leveraging deep learning models to create names, summaries, personalities, greeting messages, and avatars for characters. The tool provides an easy way to create unique and imaginative characters for storytelling, chatting, and other purposes.

github

: 108

ai-anime-art-generator

AI Anime Art Generator is an AI-driven cutting-edge tool for anime arts creation. Perfect for beginners to easily create stunning anime art without any prior experience. It allows users to create detailed character designs, custom avatars for social media, and explore new artistic styles and ideas. Built on Next.js, TailwindCSS, Google Analytics, Vercel, Replicate, CloudFlare R2, and Clerk.

github

: 59

TavernAI

TavernAI is an atmospheric frontend tool for chat and storywriting, compatible with various backends. It offers features like character creation, online character database, group chat, story mode, world info, message swiping, configurable settings, interface themes, backgrounds, message editing, GPT-4.5, and Claude picture recognition. The tool supports backends like Kobold series, Oobabooga's Text Generation Web UI, OpenAI, NovelAI, and Claude. Users can easily install TavernAI on different operating systems and start using it for interactive storytelling and chat experiences.

github

: 2.1k

llm.nvim

llm.nvim is a plugin for Neovim that enables code completion using LLM models. It supports 'ghost-text' code completion similar to Copilot and allows users to choose their model for code generation via HTTP requests. The plugin interfaces with multiple backends like Hugging Face, Ollama, Open AI, and TGI, providing flexibility in model selection and configuration. Users can customize the behavior of suggestions, tokenization, and model parameters to enhance their coding experience. llm.nvim also includes commands for toggling auto-suggestions and manually requesting suggestions, making it a versatile tool for developers using Neovim.

github

: 603

For similar jobs

promptflow

**Prompt flow** is a suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications, from ideation, prototyping, testing, evaluation to production deployment and monitoring. It makes prompt engineering much easier and enables you to build LLM apps with production quality.

github

: 9.2k

deepeval

DeepEval is a simple-to-use, open-source LLM evaluation framework specialized for unit testing LLM outputs. It incorporates various metrics such as G-Eval, hallucination, answer relevancy, RAGAS, etc., and runs locally on your machine for evaluation. It provides a wide range of ready-to-use evaluation metrics, allows for creating custom metrics, integrates with any CI/CD environment, and enables benchmarking LLMs on popular benchmarks. DeepEval is designed for evaluating RAG and fine-tuning applications, helping users optimize hyperparameters, prevent prompt drifting, and transition from OpenAI to hosting their own Llama2 with confidence.

github

: 3.2k

MegaDetector

MegaDetector is an AI model that identifies animals, people, and vehicles in camera trap images (which also makes it useful for eliminating blank images). This model is trained on several million images from a variety of ecosystems. MegaDetector is just one of many tools that aims to make conservation biologists more efficient with AI. If you want to learn about other ways to use AI to accelerate camera trap workflows, check out our of the field, affectionately titled "Everything I know about machine learning and camera traps".

github

: 106

leapfrogai

LeapfrogAI is a self-hosted AI platform designed to be deployed in air-gapped resource-constrained environments. It brings sophisticated AI solutions to these environments by hosting all the necessary components of an AI stack, including vector databases, model backends, API, and UI. LeapfrogAI's API closely matches that of OpenAI, allowing tools built for OpenAI/ChatGPT to function seamlessly with a LeapfrogAI backend. It provides several backends for various use cases, including llama-cpp-python, whisper, text-embeddings, and vllm. LeapfrogAI leverages Chainguard's apko to harden base python images, ensuring the latest supported Python versions are used by the other components of the stack. The LeapfrogAI SDK provides a standard set of protobuffs and python utilities for implementing backends and gRPC. LeapfrogAI offers UI options for common use-cases like chat, summarization, and transcription. It can be deployed and run locally via UDS and Kubernetes, built out using Zarf packages. LeapfrogAI is supported by a community of users and contributors, including Defense Unicorns, Beast Code, Chainguard, Exovera, Hypergiant, Pulze, SOSi, United States Navy, United States Air Force, and United States Space Force.

github

: 255

llava-docker

This Docker image for LLaVA (Large Language and Vision Assistant) provides a convenient way to run LLaVA locally or on RunPod. LLaVA is a powerful AI tool that combines natural language processing and computer vision capabilities. With this Docker image, you can easily access LLaVA's functionalities for various tasks, including image captioning, visual question answering, text summarization, and more. The image comes pre-installed with LLaVA v1.2.0, Torch 2.1.2, xformers 0.0.23.post1, and other necessary dependencies. You can customize the model used by setting the MODEL environment variable. The image also includes a Jupyter Lab environment for interactive development and exploration. Overall, this Docker image offers a comprehensive and user-friendly platform for leveraging LLaVA's capabilities.

github

: 59

carrot

The 'carrot' repository on GitHub provides a list of free and user-friendly ChatGPT mirror sites for easy access. The repository includes sponsored sites offering various GPT models and services. Users can find and share sites, report errors, and access stable and recommended sites for ChatGPT usage. The repository also includes a detailed list of ChatGPT sites, their features, and accessibility options, making it a valuable resource for ChatGPT users seeking free and unlimited GPT services.

github

: 16.9k

TrustLLM

TrustLLM is a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions. Specifically, we first propose a set of principles for trustworthy LLMs that span eight different dimensions. Based on these principles, we further establish a benchmark across six dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics. We then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of over 30 datasets. The document explains how to use the trustllm python package to help you assess the performance of your LLM in trustworthiness more quickly. For more details about TrustLLM, please refer to project website.

github

: 412

AI-YinMei

AI-YinMei is an AI virtual anchor Vtuber development tool (N card version). It supports fastgpt knowledge base chat dialogue, a complete set of solutions for LLM large language models: [fastgpt] + [one-api] + [Xinference], supports docking bilibili live broadcast barrage reply and entering live broadcast welcome speech, supports Microsoft edge-tts speech synthesis, supports Bert-VITS2 speech synthesis, supports GPT-SoVITS speech synthesis, supports expression control Vtuber Studio, supports painting stable-diffusion-webui output OBS live broadcast room, supports painting picture pornography public-NSFW-y-distinguish, supports search and image search service duckduckgo (requires magic Internet access), supports image search service Baidu image search (no magic Internet access), supports AI reply chat box [html plug-in], supports AI singing Auto-Convert-Music, supports playlist [html plug-in], supports dancing function, supports expression video playback, supports head touching action, supports gift smashing action, supports singing automatic start dancing function, chat and singing automatic cycle swing action, supports multi scene switching, background music switching, day and night automatic switching scene, supports open singing and painting, let AI automatically judge the content.

github

: 529