ai-town
A MIT-licensed, deployable starter kit for building and customizing your own version of AI town - a virtual town where AI characters live, chat and socialize.
Stars: 6285
AI Town is a virtual town where AI characters live, chat, and socialize. This project provides a deployable starter kit for building and customizing your own version of AI Town. It features a game engine, database, vector search, auth, text model, deployment, pixel art generation, background music generation, and local inference. You can customize your own simulation by creating characters and stories, updating spritesheets, changing the background, and modifying the background music.
README:
Join our community Discord: AI Stack Devs
AI Town is a virtual town where AI characters live, chat and socialize.
This project is a deployable starter kit for easily building and customizing your own version of AI town. Inspired by the research paper Generative Agents: Interactive Simulacra of Human Behavior.
The primary goal of this project, beyond just being a lot of fun to work on, is to provide a platform with a strong foundation that is meant to be extended. The back-end natively supports shared global state, transactions, and a simulation engine and should be suitable from everything from a simple project to play around with to a scalable, multi-player game. A secondary goal is to make a JS/TS framework available as most simulators in this space (including the original paper above) are written in Python.
- π» Stack
- π§ Installation
- π€ Customize - run YOUR OWN simulated world
- π©βπ» Deploying
- π Credits
- Game engine, database, and vector search: Convex
- Auth (Optional): Clerk
- Default chat model is
llama3
and embeddings withmxbai-embed-large
. - Local inference: Ollama
- Configurable for other cloud LLMs: Together.ai or antying that speaks the OpenAI API. PRs welcome to add more cloud provider support.
- Pixel Art Generation: Replicate, Fal.ai
- Background Music Generation: Replicate using MusicGen
git clone https://github.com/a16z-infra/ai-town.git
cd ai-town
npm install
2. To develop locally with Convex:
Either download a pre-built binary(recommended), or build it from source and run it.
# For new Macs:
curl -L -O https://github.com/get-convex/convex-backend/releases/latest/download/convex-local-backend-aarch64-apple-darwin.zip
unzip convex-local-backend-aarch64-apple-darwin.zip
brew install just
# Runs the server
./convex-local-backend
This also installs just
(e.g. brew install just
or cargo install just
).
We use just
like make
to add extra params, so you run just convex ...
instead of npx convex ...
for local development.
If you're running the pre-built binary on Mac and there's an Apple warning, go to the folder it's in and right-click it and select "Open" to bypass. From then on you can run it from the commandline. Or you can compile it from source and run it (see above).
To develop against the cloud-hosted version, change the package.json scripts
to use convex ...
instead of just convex ...
.
3. To run a local LLM, download and run Ollama.
You can leave the app running or run ollama serve
.
ollama serve
will warn you if the app is already running.
Run ollama pull llama3
to have it download llama3
.
Test it out with ollama run llama3
.
If you want to customize which model to use, adjust convex/util/llm.ts or set
just convex env set OLLAMA_MODEL # model
.
Ollama model options can be found here.
You might want to set NUM_MEMORIES_TO_SEARCH
to 1
in constants.ts,
to reduce the size of conversation prompts, if you see slowness.
Check out convex/config.ts
to configure which models to offer to the UI,
or to set it up to talk to a cloud-hosted LLM.
For Daily background music generation, create a
Replicate account and create a token in your Profile's
API Token page.
npx convex env set REPLICATE_API_TOKEN # token
Specify just
instead of npx
if you're doing local development.
To run both the front and and back end:
npm run dev
You can now visit http://localhost:5173.
If you'd rather run the frontend in a separate terminal from Convex (which syncs your backend functions as they're saved), you can run these two commands:
npm run dev:frontend
npm run dev:backend
See package.json for details, but dev:backend runs just convex dev
Note: The simulation will pause after 5 minutes if the window is idle.
Loading the page will unpause it.
You can also manually freeze & unfreeze the world with a button in the UI.
If you want to run the world without the
browser, you can comment-out the "stop inactive worlds" cron in convex/crons.ts
.
To stop the back end, in case of too much activity
This will stop running the engine and agents. You can still run queries and run functions to debug.
just convex run testing:stop
To restart the back end after stopping it
just convex run testing:resume
To kick the engine in case the game engine or agents aren't running
just convex run testing:kick
To archive the world
If you'd like to reset the world and start from scratch, you can archive the current world:
just convex run testing:archive
Then, you can still look at the world's data in the dashboard, but the engine and agents will no longer run.
You can then create a fresh world with init
.
just convex run init
To clear all databases
You can wipe all tables with the wipeAllTables
testing function.
just convex run testing:wipeAllTables
To pause your backend deployment
You can go to the dashboard to your deployment settings to pause and un-pause your deployment. This will stop all functions, whether invoked from the client, scheduled, or as a cron job. See this as a last resort, as there are gentler ways of stopping above. Once you
NOTE: every time you change character data, you should re-run
just convex run testing:wipeAllTables
and then
npm run dev
to re-upload everything to Convex.
This is because character data is sent to Convex on the initial load.
However, beware that just convex run testing:wipeAllTables
WILL wipe all of your data.
-
Create your own characters and stories: All characters and stories, as well as their spritesheet references are stored in characters.ts. You can start by changing character descriptions.
-
Updating spritesheets: in
data/characters.ts
, you will see this code:
export const characters = [
{
name: 'f1',
textureUrl: '/assets/32x32folk.png',
spritesheetData: f1SpritesheetData,
speed: 0.1,
},
...
];
You should find a sprite sheet for your character, and define sprite motion / assets in the corresponding file (in the above example, f1SpritesheetData
was defined in f1.ts)
-
Update the Background (Environment): The map gets loaded in
convex/init.ts
fromdata/gentle.js
. To update the map, follow these steps:- Use Tiled to export tilemaps as a JSON file (2 layers named bgtiles and objmap)
- Use the
convertMap.js
script to convert the JSON to a format that the engine can use.
node data/convertMap.js <mapDataPath> <assetPath> <tilesetpxw> <tilesetpxh>
-
<mapDataPath>
: Path to the Tiled JSON file. -
<assetPath>
: Path to tileset images. -
<tilesetpxw>
: Tileset width in pixels. -
<tilesetpxh>
: Tileset height in pixels. Generatesconverted-map.js
that you can use likegentle.js
- Change the background music by modifying the prompt in
convex/music.ts
- Change how often to generate new music at
convex/crons.ts
by modifying thegenerate new background music
job
Configure convex/util/llm.ts
or set these env variables:
# Local Convex
just convex env set LLM_API_HOST # url
just convex env set LLM_MODEL # model
# Cloud Convex
npx convex env set LLM_API_HOST # url
npx convex env set LLM_MODEL # model
The embeddings model config needs to be changed in code, since you need to specify the embeddings dimension.
For Together.ai, visit https://api.together.xyz/settings/api-keys For OpenAI, visit https://platform.openai.com/account/api-keys
You can run your Convex backend in the cloud by just running
npx convex dev --once --configure
And updating the package.json
scripts to remove just
:
change just convex ...
to convex ...
.
You'll then need to set any environment variables you had locally in the cloud
environment with npx convex env set
or on the dashboard:
https://dashboard.convex.dev/deployment/settings/environment-variables
To run commands, use npx convex ...
where you used to run just convex ...
.
Before you can run the app, you will need to make sure the Convex functions are deployed to its production environment.
- Run
npx convex deploy
to deploy the convex functions to production - Run
npx convex run init --prod
If you have existing data you want to clear, you can run npx convex run testing:wipeAllTables --prod
You can add clerk auth back in with git revert b44a436
.
Or just look at that diff for what changed to remove it.
Make a Clerk account
- Go to https://dashboard.clerk.com/ and click on "Add Application"
- Name your application and select the sign-in providers you would like to offer users
- Create Application
- Add
VITE_CLERK_PUBLISHABLE_KEY
andCLERK_SECRET_KEY
to.env.local
VITE_CLERK_PUBLISHABLE_KEY=pk_***
CLERK_SECRET_KEY=sk_***
- Go to JWT Templates and create a new Convex Template.
- Copy the JWKS endpoint URL for use below.
npx convex env set CLERK_ISSUER_URL # e.g. https://your-issuer-url.clerk.accounts.dev/
- Register an account on Vercel and then install the Vercel CLI.
-
If you are using Github Codespaces: You will need to install the Vercel CLI and authenticate from your codespaces cli by running
vercel login
. - Deploy the app to Vercel with
vercel --prod
.
We support using Ollama for conversation generations. To have it accessible from the web, you can use Tunnelmole or Ngrok or similar.
Using Tunnelmole
Tunnelmole is an open source tunneling tool.
You can install Tunnelmole using one of the following options:
- NPM:
npm install -g tunnelmole
- Linux:
curl -s https://tunnelmole.com/sh/install-linux.sh | sudo bash
- Mac:
curl -s https://tunnelmole.com/sh/install-mac.sh --output install-mac.sh && sudo bash install-mac.sh
- Windows: Install with NPM, or if you don't have NodeJS installed, download the
exe
file for Windows here and put it somewhere in your PATH.
Once Tunnelmole is installed, run the following command:
tmole 11434
Tunnelmole should output a unique url once you run this command.
Using Ngrok
Ngrok is a popular closed source tunneling tool.
Once ngrok is installed and authenticated, run the following command:
ngrok http http://localhost:11434
Ngrok should output a unique url once you run this command.
Add Ollama endpoint to Convex
npx convex env set OLLAMA_HOST # your tunnelmole/ngrok unique url from the previous step
Update Ollama domains
Ollama has a list of accepted domains. Add the ngrok domain so it won't reject traffic. see ollama.ai for more details.
- All interactions, background music and rendering on the component in the project are powered by PixiJS.
- Tilesheet:
- https://opengameart.org/content/16x16-game-assets by George Bailey
- https://opengameart.org/content/16x16-rpg-tileset by hilau
- We used https://github.com/pierpo/phaser3-simple-rpg for the original POC of this project. We have since re-wrote the whole app, but appreciated the easy starting point
- Original assets by ansimuz
- The UI is based on original assets by Mounir Tohami
Convex is a hosted backend platform with a
built-in database that lets you write your
database schema and
server functions in
TypeScript. Server-side database
queries automatically
cache and
subscribe to data, powering a
realtime useQuery
hook in our
React client. There are also clients for
Python,
Rust,
ReactNative, and
Node, as well as a straightforward
HTTP API.
The database supports NoSQL-style documents with opt-in schema validation, relationships and custom indexes (including on fields in nested objects).
The
query
and
mutation
server functions have transactional,
low latency access to the database and leverage our
v8
runtime with
determinism guardrails
to provide the strongest ACID guarantees on the market:
immediate consistency,
serializable isolation, and
automatic conflict resolution via
optimistic multi-version concurrency control (OCC / MVCC).
The action
server functions have
access to external APIs and enable other side-effects and non-determinism in
either our
optimized v8
runtime or a more
flexible node
runtime.
Functions can run in the background via scheduling and cron jobs.
Development is cloud-first, with hot reloads for server function editing via the CLI, preview deployments, logging and exception reporting integrations, There is a dashboard UI to browse and edit data, edit environment variables, view logs, run server functions, and more.
There are built-in features for reactive pagination, file storage, reactive text search, vector search, https endpoints (for webhooks), snapshot import/export, streaming import/export, and runtime validation for function arguments and database data.
Everything scales automatically, and itβs free to start.
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for ai-town
Similar Open Source Tools
ai-town
AI Town is a virtual town where AI characters live, chat, and socialize. This project provides a deployable starter kit for building and customizing your own version of AI Town. It features a game engine, database, vector search, auth, text model, deployment, pixel art generation, background music generation, and local inference. You can customize your own simulation by creating characters and stories, updating spritesheets, changing the background, and modifying the background music.
aides-jeunes
The user interface (and the main server) of the simulator of aids and social benefits for young people. It is based on the free socio-fiscal simulator Openfisca.
chat-ollama
ChatOllama is an open-source chatbot based on LLMs (Large Language Models). It supports a wide range of language models, including Ollama served models, OpenAI, Azure OpenAI, and Anthropic. ChatOllama supports multiple types of chat, including free chat with LLMs and chat with LLMs based on a knowledge base. Key features of ChatOllama include Ollama models management, knowledge bases management, chat, and commercial LLMs API keys management.
oterm
Oterm is a text-based terminal client for Ollama, a large language model. It provides an intuitive and simple terminal UI, allowing users to interact with Ollama without running servers or frontends. Oterm supports multiple persistent chat sessions, which are stored along with context embeddings and system prompt customizations in a SQLite database. Users can easily customize the model's system prompt and parameters, and select from any of the models they have pulled in Ollama or their own custom models. Oterm also supports keyboard shortcuts for creating new chat sessions, editing existing sessions, renaming sessions, exporting sessions as markdown, deleting sessions, toggling between dark and light themes, quitting the application, switching to multiline input mode, selecting images to include with messages, and navigating through the history of previous prompts. Oterm is licensed under the MIT License.
TypeGPT
TypeGPT is a Python application that enables users to interact with ChatGPT or Google Gemini from any text field in their operating system using keyboard shortcuts. It provides global accessibility, keyboard shortcuts for communication, and clipboard integration for larger text inputs. Users need to have Python 3.x installed along with specific packages and API keys from OpenAI for ChatGPT access. The tool allows users to run the program normally or in the background, manage processes, and stop the program. Users can use keyboard shortcuts like `/ask`, `/see`, `/stop`, `/chatgpt`, `/gemini`, `/check`, and `Shift + Cmd + Enter` to interact with the application in any text field. Customization options are available by modifying files like `keys.txt` and `system_prompt.txt`. Contributions are welcome, and future plans include adding support for other APIs and a user-friendly GUI.
qb
QANTA is a system and dataset for question answering tasks. It provides a script to download datasets, preprocesses questions, and matches them with Wikipedia pages. The system includes various datasets, training, dev, and test data in JSON and SQLite formats. Dependencies include Python 3.6, `click`, and NLTK models. Elastic Search 5.6 is needed for the Guesser component. Configuration is managed through environment variables and YAML files. QANTA supports multiple guesser implementations that can be enabled/disabled. Running QANTA involves using `cli.py` and Luigi pipelines. The system accesses raw Wikipedia dumps for data processing. The QANTA ID numbering scheme categorizes datasets based on events and competitions.
dir-assistant
Dir-assistant is a tool that allows users to interact with their current directory's files using local or API Language Models (LLMs). It supports various platforms and provides API support for major LLM APIs. Users can configure and customize their local LLMs and API LLMs using the tool. Dir-assistant also supports model downloads and configurations for efficient usage. It is designed to enhance file interaction and retrieval using advanced language models.
opencommit
OpenCommit is a tool that auto-generates meaningful commits using AI, allowing users to quickly create commit messages for their staged changes. It provides a CLI interface for easy usage and supports customization of commit descriptions, emojis, and AI models. Users can configure local and global settings, switch between different AI providers, and set up Git hooks for integration with IDE Source Control. Additionally, OpenCommit can be used as a GitHub Action to automatically improve commit messages on push events, ensuring all commits are meaningful and not generic. Payments for OpenAI API requests are handled by the user, with the tool storing API keys locally.
warc-gpt
WARC-GPT is an experimental retrieval augmented generation pipeline for web archive collections. It allows users to interact with WARC files, extract text, generate text embeddings, visualize embeddings, and interact with a web UI and API. The tool is highly customizable, supporting various LLMs, providers, and embedding models. Users can configure the application using environment variables, ingest WARC files, start the server, and interact with the web UI and API to search for content and generate text completions. WARC-GPT is designed for exploration and experimentation in exploring web archives using AI.
WebCraftifyAI
WebCraftifyAI is a software aid that makes it easy to create and build web pages and content. It is designed to be user-friendly and accessible to people of all skill levels. With WebCraftifyAI, you can quickly and easily create professional-looking websites without having to learn complex coding or design skills.
airbyte_serverless
AirbyteServerless is a lightweight tool designed to simplify the management of Airbyte connectors. It offers a serverless mode for running connectors, allowing users to easily move data from any source to their data warehouse. Unlike the full Airbyte-Open-Source-Platform, AirbyteServerless focuses solely on the Extract-Load process without a UI, database, or transform layer. It provides a CLI tool, 'abs', for managing connectors, creating connections, running jobs, selecting specific data streams, handling secrets securely, and scheduling remote runs. The tool is scalable, allowing independent deployment of multiple connectors. It aims to streamline the connector management process and provide a more agile alternative to the comprehensive Airbyte platform.
fabric
Fabric is an open-source framework for augmenting humans using AI. It provides a structured approach to breaking down problems into individual components and applying AI to them one at a time. Fabric includes a collection of pre-defined Patterns (prompts) that can be used for a variety of tasks, such as extracting the most interesting parts of YouTube videos and podcasts, writing essays, summarizing academic papers, creating AI art prompts, and more. Users can also create their own custom Patterns. Fabric is designed to be easy to use, with a command-line interface and a variety of helper apps. It is also extensible, allowing users to integrate it with their own AI applications and infrastructure.
redbox-copilot
Redbox Copilot is a retrieval augmented generation (RAG) app that uses GenAI to chat with and summarise civil service documents. It increases organisational memory by indexing documents and can summarise reports read months ago, supplement them with current work, and produce a first draft that lets civil servants focus on what they do best. The project uses a microservice architecture with each microservice running in its own container defined by a Dockerfile. Dependencies are managed using Python Poetry. Contributions are welcome, and the project is licensed under the MIT License.
blinkid-ios
BlinkID iOS is a mobile SDK that enables developers to easily integrate ID scanning and data extraction capabilities into their iOS applications. The SDK supports scanning and processing various types of identity documents, such as passports, driver's licenses, and ID cards. It provides accurate and fast data extraction, including personal information and document details. With BlinkID iOS, developers can enhance their apps with secure and reliable ID verification functionality, improving user experience and streamlining identity verification processes.
unstructured
The `unstructured` library provides open-source components for ingesting and pre-processing images and text documents, such as PDFs, HTML, Word docs, and many more. The use cases of `unstructured` revolve around streamlining and optimizing the data processing workflow for LLMs. `unstructured` modular functions and connectors form a cohesive system that simplifies data ingestion and pre-processing, making it adaptable to different platforms and efficient in transforming unstructured data into structured outputs.
ChatGPT-OpenAI-Smart-Speaker
ChatGPT Smart Speaker is a project that enables speech recognition and text-to-speech functionalities using OpenAI and Google Speech Recognition. It provides scripts for running on PC/Mac and Raspberry Pi, allowing users to interact with a smart speaker setup. The project includes detailed instructions for setting up the required hardware and software dependencies, along with customization options for the OpenAI model engine, language settings, and response randomness control. The Raspberry Pi setup involves utilizing the ReSpeaker hardware for voice feedback and light shows. The project aims to offer an advanced smart speaker experience with features like wake word detection and response generation using AI models.
For similar tasks
ai-town
AI Town is a virtual town where AI characters live, chat, and socialize. This project provides a deployable starter kit for building and customizing your own version of AI Town. It features a game engine, database, vector search, auth, text model, deployment, pixel art generation, background music generation, and local inference. You can customize your own simulation by creating characters and stories, updating spritesheets, changing the background, and modifying the background music.
openroleplay.ai
Open Roleplay is an open-source alternative to Character.ai. It allows users to create their own AI characters, customize them, and generate images and voices for them. Open Roleplay also supports group chat and automatic translation. The tool is built with Next.js, React.js, Tailwind CSS, Vercel, Convex, and Clerk.
For similar jobs
weave
Weave is a toolkit for developing Generative AI applications, built by Weights & Biases. With Weave, you can log and debug language model inputs, outputs, and traces; build rigorous, apples-to-apples evaluations for language model use cases; and organize all the information generated across the LLM workflow, from experimentation to evaluations to production. Weave aims to bring rigor, best-practices, and composability to the inherently experimental process of developing Generative AI software, without introducing cognitive overhead.
LLMStack
LLMStack is a no-code platform for building generative AI agents, workflows, and chatbots. It allows users to connect their own data, internal tools, and GPT-powered models without any coding experience. LLMStack can be deployed to the cloud or on-premise and can be accessed via HTTP API or triggered from Slack or Discord.
VisionCraft
The VisionCraft API is a free API for using over 100 different AI models. From images to sound.
kaito
Kaito is an operator that automates the AI/ML inference model deployment in a Kubernetes cluster. It manages large model files using container images, avoids tuning deployment parameters to fit GPU hardware by providing preset configurations, auto-provisions GPU nodes based on model requirements, and hosts large model images in the public Microsoft Container Registry (MCR) if the license allows. Using Kaito, the workflow of onboarding large AI inference models in Kubernetes is largely simplified.
PyRIT
PyRIT is an open access automation framework designed to empower security professionals and ML engineers to red team foundation models and their applications. It automates AI Red Teaming tasks to allow operators to focus on more complicated and time-consuming tasks and can also identify security harms such as misuse (e.g., malware generation, jailbreaking), and privacy harms (e.g., identity theft). The goal is to allow researchers to have a baseline of how well their model and entire inference pipeline is doing against different harm categories and to be able to compare that baseline to future iterations of their model. This allows them to have empirical data on how well their model is doing today, and detect any degradation of performance based on future improvements.
tabby
Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features: * Self-contained, with no need for a DBMS or cloud service. * OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE). * Supports consumer-grade GPUs.
spear
SPEAR (Simulator for Photorealistic Embodied AI Research) is a powerful tool for training embodied agents. It features 300 unique virtual indoor environments with 2,566 unique rooms and 17,234 unique objects that can be manipulated individually. Each environment is designed by a professional artist and features detailed geometry, photorealistic materials, and a unique floor plan and object layout. SPEAR is implemented as Unreal Engine assets and provides an OpenAI Gym interface for interacting with the environments via Python.
Magick
Magick is a groundbreaking visual AIDE (Artificial Intelligence Development Environment) for no-code data pipelines and multimodal agents. Magick can connect to other services and comes with nodes and templates well-suited for intelligent agents, chatbots, complex reasoning systems and realistic characters.