ai-town
A MIT-licensed, deployable starter kit for building and customizing your own version of AI town - a virtual town where AI characters live, chat and socialize.
Stars: 6285
AI Town is a virtual town where AI characters live, chat, and socialize. This project provides a deployable starter kit for building and customizing your own version of AI Town. It features a game engine, database, vector search, auth, text model, deployment, pixel art generation, background music generation, and local inference. You can customize your own simulation by creating characters and stories, updating spritesheets, changing the background, and modifying the background music.
README:
Join our community Discord: AI Stack Devs
AI Town is a virtual town where AI characters live, chat and socialize.
This project is a deployable starter kit for easily building and customizing your own version of AI town. Inspired by the research paper Generative Agents: Interactive Simulacra of Human Behavior.
The primary goal of this project, beyond just being a lot of fun to work on, is to provide a platform with a strong foundation that is meant to be extended. The back-end natively supports shared global state, transactions, and a simulation engine and should be suitable from everything from a simple project to play around with to a scalable, multi-player game. A secondary goal is to make a JS/TS framework available as most simulators in this space (including the original paper above) are written in Python.
- π» Stack
- π§ Installation
- π€ Customize - run YOUR OWN simulated world
- π©βπ» Deploying
- π Credits
- Game engine, database, and vector search: Convex
- Auth (Optional): Clerk
- Default chat model is
llama3and embeddings withmxbai-embed-large. - Local inference: Ollama
- Configurable for other cloud LLMs: Together.ai or antying that speaks the OpenAI API. PRs welcome to add more cloud provider support.
- Pixel Art Generation: Replicate, Fal.ai
- Background Music Generation: Replicate using MusicGen
git clone https://github.com/a16z-infra/ai-town.git
cd ai-town
npm install2. To develop locally with Convex:
Either download a pre-built binary(recommended), or build it from source and run it.
# For new Macs:
curl -L -O https://github.com/get-convex/convex-backend/releases/latest/download/convex-local-backend-aarch64-apple-darwin.zip
unzip convex-local-backend-aarch64-apple-darwin.zip
brew install just
# Runs the server
./convex-local-backendThis also installs just
(e.g. brew install just or cargo install just).
We use just like make to add extra params, so you run just convex ...
instead of npx convex ... for local development.
If you're running the pre-built binary on Mac and there's an Apple warning, go to the folder it's in and right-click it and select "Open" to bypass. From then on you can run it from the commandline. Or you can compile it from source and run it (see above).
To develop against the cloud-hosted version, change the package.json scripts
to use convex ... instead of just convex ....
3. To run a local LLM, download and run Ollama.
You can leave the app running or run ollama serve.
ollama serve will warn you if the app is already running.
Run ollama pull llama3 to have it download llama3.
Test it out with ollama run llama3.
If you want to customize which model to use, adjust convex/util/llm.ts or set
just convex env set OLLAMA_MODEL # model.
Ollama model options can be found here.
You might want to set NUM_MEMORIES_TO_SEARCH to 1 in constants.ts,
to reduce the size of conversation prompts, if you see slowness.
Check out convex/config.ts to configure which models to offer to the UI,
or to set it up to talk to a cloud-hosted LLM.
For Daily background music generation, create a
Replicate account and create a token in your Profile's
API Token page.
npx convex env set REPLICATE_API_TOKEN # token
Specify just instead of npx if you're doing local development.
To run both the front and and back end:
npm run devYou can now visit http://localhost:5173.
If you'd rather run the frontend in a separate terminal from Convex (which syncs your backend functions as they're saved), you can run these two commands:
npm run dev:frontend
npm run dev:backendSee package.json for details, but dev:backend runs just convex dev
Note: The simulation will pause after 5 minutes if the window is idle.
Loading the page will unpause it.
You can also manually freeze & unfreeze the world with a button in the UI.
If you want to run the world without the
browser, you can comment-out the "stop inactive worlds" cron in convex/crons.ts.
To stop the back end, in case of too much activity
This will stop running the engine and agents. You can still run queries and run functions to debug.
just convex run testing:stopTo restart the back end after stopping it
just convex run testing:resumeTo kick the engine in case the game engine or agents aren't running
just convex run testing:kickTo archive the world
If you'd like to reset the world and start from scratch, you can archive the current world:
just convex run testing:archiveThen, you can still look at the world's data in the dashboard, but the engine and agents will no longer run.
You can then create a fresh world with init.
just convex run initTo clear all databases
You can wipe all tables with the wipeAllTables testing function.
just convex run testing:wipeAllTablesTo pause your backend deployment
You can go to the dashboard to your deployment settings to pause and un-pause your deployment. This will stop all functions, whether invoked from the client, scheduled, or as a cron job. See this as a last resort, as there are gentler ways of stopping above. Once you
NOTE: every time you change character data, you should re-run
just convex run testing:wipeAllTables and then
npm run dev to re-upload everything to Convex.
This is because character data is sent to Convex on the initial load.
However, beware that just convex run testing:wipeAllTables WILL wipe all of your data.
-
Create your own characters and stories: All characters and stories, as well as their spritesheet references are stored in characters.ts. You can start by changing character descriptions.
-
Updating spritesheets: in
data/characters.ts, you will see this code:
export const characters = [
{
name: 'f1',
textureUrl: '/assets/32x32folk.png',
spritesheetData: f1SpritesheetData,
speed: 0.1,
},
...
];You should find a sprite sheet for your character, and define sprite motion / assets in the corresponding file (in the above example, f1SpritesheetData was defined in f1.ts)
-
Update the Background (Environment): The map gets loaded in
convex/init.tsfromdata/gentle.js. To update the map, follow these steps:- Use Tiled to export tilemaps as a JSON file (2 layers named bgtiles and objmap)
- Use the
convertMap.jsscript to convert the JSON to a format that the engine can use.
node data/convertMap.js <mapDataPath> <assetPath> <tilesetpxw> <tilesetpxh>-
<mapDataPath>: Path to the Tiled JSON file. -
<assetPath>: Path to tileset images. -
<tilesetpxw>: Tileset width in pixels. -
<tilesetpxh>: Tileset height in pixels. Generatesconverted-map.jsthat you can use likegentle.js
- Change the background music by modifying the prompt in
convex/music.ts - Change how often to generate new music at
convex/crons.tsby modifying thegenerate new background musicjob
Configure convex/util/llm.ts or set these env variables:
# Local Convex
just convex env set LLM_API_HOST # url
just convex env set LLM_MODEL # model
# Cloud Convex
npx convex env set LLM_API_HOST # url
npx convex env set LLM_MODEL # modelThe embeddings model config needs to be changed in code, since you need to specify the embeddings dimension.
For Together.ai, visit https://api.together.xyz/settings/api-keys For OpenAI, visit https://platform.openai.com/account/api-keys
You can run your Convex backend in the cloud by just running
npx convex dev --once --configureAnd updating the package.json scripts to remove just:
change just convex ... to convex ....
You'll then need to set any environment variables you had locally in the cloud
environment with npx convex env set or on the dashboard:
https://dashboard.convex.dev/deployment/settings/environment-variables
To run commands, use npx convex ... where you used to run just convex ....
Before you can run the app, you will need to make sure the Convex functions are deployed to its production environment.
- Run
npx convex deployto deploy the convex functions to production - Run
npx convex run init --prod
If you have existing data you want to clear, you can run npx convex run testing:wipeAllTables --prod
You can add clerk auth back in with git revert b44a436.
Or just look at that diff for what changed to remove it.
Make a Clerk account
- Go to https://dashboard.clerk.com/ and click on "Add Application"
- Name your application and select the sign-in providers you would like to offer users
- Create Application
- Add
VITE_CLERK_PUBLISHABLE_KEYandCLERK_SECRET_KEYto.env.local
VITE_CLERK_PUBLISHABLE_KEY=pk_***
CLERK_SECRET_KEY=sk_***- Go to JWT Templates and create a new Convex Template.
- Copy the JWKS endpoint URL for use below.
npx convex env set CLERK_ISSUER_URL # e.g. https://your-issuer-url.clerk.accounts.dev/- Register an account on Vercel and then install the Vercel CLI.
-
If you are using Github Codespaces: You will need to install the Vercel CLI and authenticate from your codespaces cli by running
vercel login. - Deploy the app to Vercel with
vercel --prod.
We support using Ollama for conversation generations. To have it accessible from the web, you can use Tunnelmole or Ngrok or similar.
Using Tunnelmole
Tunnelmole is an open source tunneling tool.
You can install Tunnelmole using one of the following options:
- NPM:
npm install -g tunnelmole - Linux:
curl -s https://tunnelmole.com/sh/install-linux.sh | sudo bash - Mac:
curl -s https://tunnelmole.com/sh/install-mac.sh --output install-mac.sh && sudo bash install-mac.sh - Windows: Install with NPM, or if you don't have NodeJS installed, download the
exefile for Windows here and put it somewhere in your PATH.
Once Tunnelmole is installed, run the following command:
tmole 11434
Tunnelmole should output a unique url once you run this command.
Using Ngrok
Ngrok is a popular closed source tunneling tool.
Once ngrok is installed and authenticated, run the following command:
ngrok http http://localhost:11434
Ngrok should output a unique url once you run this command.
Add Ollama endpoint to Convex
npx convex env set OLLAMA_HOST # your tunnelmole/ngrok unique url from the previous stepUpdate Ollama domains
Ollama has a list of accepted domains. Add the ngrok domain so it won't reject traffic. see ollama.ai for more details.
- All interactions, background music and rendering on the component in the project are powered by PixiJS.
- Tilesheet:
- https://opengameart.org/content/16x16-game-assets by George Bailey
- https://opengameart.org/content/16x16-rpg-tileset by hilau
- We used https://github.com/pierpo/phaser3-simple-rpg for the original POC of this project. We have since re-wrote the whole app, but appreciated the easy starting point
- Original assets by ansimuz
- The UI is based on original assets by Mounir Tohami
Convex is a hosted backend platform with a
built-in database that lets you write your
database schema and
server functions in
TypeScript. Server-side database
queries automatically
cache and
subscribe to data, powering a
realtime useQuery hook in our
React client. There are also clients for
Python,
Rust,
ReactNative, and
Node, as well as a straightforward
HTTP API.
The database supports NoSQL-style documents with opt-in schema validation, relationships and custom indexes (including on fields in nested objects).
The
query and
mutation server functions have transactional,
low latency access to the database and leverage our
v8 runtime with
determinism guardrails
to provide the strongest ACID guarantees on the market:
immediate consistency,
serializable isolation, and
automatic conflict resolution via
optimistic multi-version concurrency control (OCC / MVCC).
The action server functions have
access to external APIs and enable other side-effects and non-determinism in
either our
optimized v8 runtime or a more
flexible node runtime.
Functions can run in the background via scheduling and cron jobs.
Development is cloud-first, with hot reloads for server function editing via the CLI, preview deployments, logging and exception reporting integrations, There is a dashboard UI to browse and edit data, edit environment variables, view logs, run server functions, and more.
There are built-in features for reactive pagination, file storage, reactive text search, vector search, https endpoints (for webhooks), snapshot import/export, streaming import/export, and runtime validation for function arguments and database data.
Everything scales automatically, and itβs free to start.
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for ai-town
Similar Open Source Tools
ai-town
AI Town is a virtual town where AI characters live, chat, and socialize. This project provides a deployable starter kit for building and customizing your own version of AI Town. It features a game engine, database, vector search, auth, text model, deployment, pixel art generation, background music generation, and local inference. You can customize your own simulation by creating characters and stories, updating spritesheets, changing the background, and modifying the background music.
openui
OpenUI is a tool designed to simplify the process of building UI components by allowing users to describe UI using their imagination and see it rendered live. It supports converting HTML to React, Svelte, Web Components, etc. The tool is open source and aims to make UI development fun, fast, and flexible. It integrates with various AI services like OpenAI, Groq, Gemini, Anthropic, Cohere, and Mistral, providing users with the flexibility to use different models. OpenUI also supports LiteLLM for connecting to various LLM services and allows users to create custom proxy configs. The tool can be run locally using Docker or Python, and it offers a development environment for quick setup and testing.
starter-monorepo
Starter Monorepo is a template repository for setting up a monorepo structure in your project. It provides a basic setup with configurations for managing multiple packages within a single repository. This template includes tools for package management, versioning, testing, and deployment. By using this template, you can streamline your development process, improve code sharing, and simplify dependency management across your project. Whether you are working on a small project or a large-scale application, Starter Monorepo can help you organize your codebase efficiently and enhance collaboration among team members.
aiarena-web
aiarena-web is a website designed for running the aiarena.net infrastructure. It consists of different modules such as core functionality, web API endpoints, frontend templates, and a module for linking users to their Patreon accounts. The website serves as a platform for obtaining new matches, reporting results, featuring match replays, and connecting with Patreon supporters. The project is licensed under GPLv3 in 2019.
polis
Polis is an AI powered sentiment gathering platform that offers a more organic approach than surveys and requires less effort than focus groups. It provides a comprehensive wiki, main deployment at https://pol.is, discussions, issue tracking, and project board for users. Polis can be set up using Docker infrastructure and offers various commands for building and running containers. Users can test their instance, update the system, and deploy Polis for production. The tool also provides developer conveniences for code reloading, type checking, and database connections. Additionally, Polis supports end-to-end browser testing using Cypress and offers troubleshooting tips for common Docker and npm issues.
seer
Seer is a service that provides AI capabilities to Sentry by running inference on Sentry issues and providing user insights. It is currently in early development and not yet compatible with self-hosted Sentry instances. The tool requires access to internal Sentry resources and is intended for internal Sentry employees. Users can set up the environment, download model artifacts, integrate with local Sentry, run evaluations for Autofix AI agent, and deploy to a sandbox staging environment. Development commands include applying database migrations, creating new migrations, running tests, and more. The tool also supports VCRs for recording and replaying HTTP requests.
Open-LLM-VTuber
Open-LLM-VTuber is a project in early stages of development that allows users to interact with Large Language Models (LLM) using voice commands and receive responses through a Live2D talking face. The project aims to provide a minimum viable prototype for offline use on macOS, Linux, and Windows, with features like long-term memory using MemGPT, customizable LLM backends, speech recognition, and text-to-speech providers. Users can configure the project to chat with LLMs, choose different backend services, and utilize Live2D models for visual representation. The project supports perpetual chat, offline operation, and GPU acceleration on macOS, addressing limitations of existing solutions on macOS.
agnai
Agnaistic is an AI roleplay chat tool that allows users to interact with personalized characters using their favorite AI services. It supports multiple AI services, persona schema formats, and features such as group conversations, user authentication, and memory/lore books. Agnaistic can be self-hosted or run using Docker, and it provides a range of customization options through its settings.json file. The tool is designed to be user-friendly and accessible, making it suitable for both casual users and developers.
qrev
QRev is an open-source alternative to Salesforce, offering AI agents to scale sales organizations infinitely. It aims to provide digital workers for various sales roles or a superagent named Qai. The tech stack includes TypeScript for frontend, NodeJS for backend, MongoDB for app server database, ChromaDB for vector database, SQLite for AI server SQL relational database, and Langchain for LLM tooling. The tool allows users to run client app, app server, and AI server components. It requires Node.js and MongoDB to be installed, and provides detailed setup instructions in the README file.
torchchat
torchchat is a codebase showcasing the ability to run large language models (LLMs) seamlessly. It allows running LLMs using Python in various environments such as desktop, server, iOS, and Android. The tool supports running models via PyTorch, chatting, generating text, running chat in the browser, and running models on desktop/server without Python. It also provides features like AOT Inductor for faster execution, running in C++ using the runner, and deploying and running on iOS and Android. The tool supports popular hardware and OS including Linux, Mac OS, Android, and iOS, with various data types and execution modes available.
airbyte_serverless
AirbyteServerless is a lightweight tool designed to simplify the management of Airbyte connectors. It offers a serverless mode for running connectors, allowing users to easily move data from any source to their data warehouse. Unlike the full Airbyte-Open-Source-Platform, AirbyteServerless focuses solely on the Extract-Load process without a UI, database, or transform layer. It provides a CLI tool, 'abs', for managing connectors, creating connections, running jobs, selecting specific data streams, handling secrets securely, and scheduling remote runs. The tool is scalable, allowing independent deployment of multiple connectors. It aims to streamline the connector management process and provide a more agile alternative to the comprehensive Airbyte platform.
actions
Sema4.ai Action Server is a tool that allows users to build semantic actions in Python to connect AI agents with real-world applications. It enables users to create custom actions, skills, loaders, and plugins that securely connect any AI Assistant platform to data and applications. The tool automatically creates and exposes an API based on function declaration, type hints, and docstrings by adding '@action' to Python scripts. It provides an end-to-end stack supporting various connections between AI and user's apps and data, offering ease of use, security, and scalability.
fasttrackml
FastTrackML is an experiment tracking server focused on speed and scalability, fully compatible with MLFlow. It provides a user-friendly interface to track and visualize your machine learning experiments, making it easy to compare different models and identify the best performing ones. FastTrackML is open source and can be easily installed and run with pip or Docker. It is also compatible with the MLFlow Python package, making it easy to integrate with your existing MLFlow workflows.
warc-gpt
WARC-GPT is an experimental retrieval augmented generation pipeline for web archive collections. It allows users to interact with WARC files, extract text, generate text embeddings, visualize embeddings, and interact with a web UI and API. The tool is highly customizable, supporting various LLMs, providers, and embedding models. Users can configure the application using environment variables, ingest WARC files, start the server, and interact with the web UI and API to search for content and generate text completions. WARC-GPT is designed for exploration and experimentation in exploring web archives using AI.
ray-llm
RayLLM (formerly known as Aviary) is an LLM serving solution that makes it easy to deploy and manage a variety of open source LLMs, built on Ray Serve. It provides an extensive suite of pre-configured open source LLMs, with defaults that work out of the box. RayLLM supports Transformer models hosted on Hugging Face Hub or present on local disk. It simplifies the deployment of multiple LLMs, the addition of new LLMs, and offers unique autoscaling support, including scale-to-zero. RayLLM fully supports multi-GPU & multi-node model deployments and offers high performance features like continuous batching, quantization and streaming. It provides a REST API that is similar to OpenAI's to make it easy to migrate and cross test them. RayLLM supports multiple LLM backends out of the box, including vLLM and TensorRT-LLM.
sage
Sage is a tool that allows users to chat with any codebase, providing a chat interface for code understanding and integration. It simplifies the process of learning how a codebase works by offering heavily documented answers sourced directly from the code. Users can set up Sage locally or on the cloud with minimal effort. The tool is designed to be easily customizable, allowing users to swap components of the pipeline and improve the algorithms powering code understanding and generation.
For similar tasks
ai-town
AI Town is a virtual town where AI characters live, chat, and socialize. This project provides a deployable starter kit for building and customizing your own version of AI Town. It features a game engine, database, vector search, auth, text model, deployment, pixel art generation, background music generation, and local inference. You can customize your own simulation by creating characters and stories, updating spritesheets, changing the background, and modifying the background music.
openroleplay.ai
Open Roleplay is an open-source alternative to Character.ai. It allows users to create their own AI characters, customize them, and generate images and voices for them. Open Roleplay also supports group chat and automatic translation. The tool is built with Next.js, React.js, Tailwind CSS, Vercel, Convex, and Clerk.
For similar jobs
weave
Weave is a toolkit for developing Generative AI applications, built by Weights & Biases. With Weave, you can log and debug language model inputs, outputs, and traces; build rigorous, apples-to-apples evaluations for language model use cases; and organize all the information generated across the LLM workflow, from experimentation to evaluations to production. Weave aims to bring rigor, best-practices, and composability to the inherently experimental process of developing Generative AI software, without introducing cognitive overhead.
LLMStack
LLMStack is a no-code platform for building generative AI agents, workflows, and chatbots. It allows users to connect their own data, internal tools, and GPT-powered models without any coding experience. LLMStack can be deployed to the cloud or on-premise and can be accessed via HTTP API or triggered from Slack or Discord.
VisionCraft
The VisionCraft API is a free API for using over 100 different AI models. From images to sound.
kaito
Kaito is an operator that automates the AI/ML inference model deployment in a Kubernetes cluster. It manages large model files using container images, avoids tuning deployment parameters to fit GPU hardware by providing preset configurations, auto-provisions GPU nodes based on model requirements, and hosts large model images in the public Microsoft Container Registry (MCR) if the license allows. Using Kaito, the workflow of onboarding large AI inference models in Kubernetes is largely simplified.
PyRIT
PyRIT is an open access automation framework designed to empower security professionals and ML engineers to red team foundation models and their applications. It automates AI Red Teaming tasks to allow operators to focus on more complicated and time-consuming tasks and can also identify security harms such as misuse (e.g., malware generation, jailbreaking), and privacy harms (e.g., identity theft). The goal is to allow researchers to have a baseline of how well their model and entire inference pipeline is doing against different harm categories and to be able to compare that baseline to future iterations of their model. This allows them to have empirical data on how well their model is doing today, and detect any degradation of performance based on future improvements.
tabby
Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features: * Self-contained, with no need for a DBMS or cloud service. * OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE). * Supports consumer-grade GPUs.
spear
SPEAR (Simulator for Photorealistic Embodied AI Research) is a powerful tool for training embodied agents. It features 300 unique virtual indoor environments with 2,566 unique rooms and 17,234 unique objects that can be manipulated individually. Each environment is designed by a professional artist and features detailed geometry, photorealistic materials, and a unique floor plan and object layout. SPEAR is implemented as Unreal Engine assets and provides an OpenAI Gym interface for interacting with the environments via Python.
Magick
Magick is a groundbreaking visual AIDE (Artificial Intelligence Development Environment) for no-code data pipelines and multimodal agents. Magick can connect to other services and comes with nodes and templates well-suited for intelligent agents, chatbots, complex reasoning systems and realistic characters.