airi

airi

💖 アイリ, ultimate Neuro-sama like LLM powered Live2D/VRM living character life pod, near by you.

Stars: 167

Visit
 screenshot

Airi is a VTuber project heavily inspired by Neuro-sama. It is capable of various functions such as playing Minecraft, chatting in Telegram and Discord, audio input from browser and Discord, client side speech recognition, VRM and Live2D model support with animations, and more. The project also includes sub-projects like unspeech, hfup, Drizzle ORM driver for DuckDB WASM, and various other tools. Airi uses models like whisper-large-v3-turbo from Hugging Face and is similar to projects like z-waif, amica, eliza, AI-Waifu-Vtuber, and AIVTuber. The project acknowledges contributions from various sources and implements packages to interact with LLMs and models.

README:

アイリ VTuber

[Try it] [简体中文文档]

Heavily inspired by Neuro-sama

Unlike the other AI driven VTuber open source projects, アイリ VTuber was built with many support of Web technologies such as WebGPU, WebAudio, Web Workers, WebAssembly, WebSocket, etc. from the first day.

This means that アイリ VTuber is capable to run on modern browsers and devices, and even on mobile devices (already done with PWA support), this brought a lot of possibilities for us (the developers) to build and extend the power of アイリ VTuber to the next level, while still left the flexibilities for users to enable features that requires TCP connections or other non-Web technologies such as connect to voice channel to Discord, or playing Minecraft, Factorio with you and your friends.

[!NOTE]

We are still in the early stage of development where we are seeking out talented developers to join us and help us to make アイリ VTuber a reality.

It's ok if you are not familiar with Vue.js, TypeScript, and devtools that required for this project, you can join us as an artist, designer, or even help us to launch our first live stream.

Even you are a big fan of React or Svelte, even Solid, we welcome you, you can open a sub-directory to add features that you want to see in アイリ VTuber, or would like to experiment with.

Fields (and related projects) that we are looking for:

  • Live2D modeller
  • VRM modeller
  • VRChat avatar designer
  • Computer Vision
  • Reinforcement Learning
  • Speech Recognition
  • Speech Synthesis
  • ONNX Runtime
  • Transformers.js
  • vLLM
  • WebGPU
  • Three.js
  • WebXR (checkout the another project we have under @moeru-ai organization)

If you are interested in, why not introduce yourself here? Would like to join part of us to build Airi?

Current progress

Capable of

  • [x] Brain
  • [x] Ears
    • [x] Audio input from browser
    • [x] Audio input from Discord
    • [x] Client side speech recognition
    • [x] Client side talking detection
  • [x] Mouth
  • [x] Body
    • [x] VRM support
      • [x] Control VRM model
    • [x] VRM model animations
      • [x] Auto blink
      • [x] Auto look at
      • [x] Idle eye movement
    • [x] Live2D support
      • [x] Control Live2D model
    • [x] Live2D model animations
      • [x] Auto blink
      • [x] Auto look at
      • [x] Idle eye movement

Development

pnpm i
pnpm dev

Supported the following LLM API Providers (powered by xsai)

Sub-projects born from this project

%%{ init: { 'flowchart': { 'curve': 'catmullRom' } } }%%

flowchart TD
  Core("Core")
  Unspeech["unspeech"]
  DBDriver["@proj-airi/drizzle-duckdb-wasm"]
  MemoryDriver["[WIP] Memory Alaya"]
  DB1["@proj-airi/duckdb-wasm"]
  ICONS["@proj-airi/lobe-icons"]
  UI("@proj-airi/stage-ui")
  Stage("Stage")
  F_AGENT("Factorio Agent")
  F_API["Factorio RCON API"]
  F_MOD1["autorio"]
  SVRT["@proj-airi/server-runtime"]
  MC_AGENT("Minecraft Agent")
  XSAI["xsai"]

  subgraph Airi
    DB1 --> DBDriver --> MemoryDriver --> Memory --> Core
    ICONS --> UI --> Stage --> Core
    Core --> STT
    Core --> SVRT
  end

  STT --> |Speaking|Unspeech
  SVRT --> |Playing Factorio|F_AGENT
  SVRT --> |Playing Minecraft|MC_AGENT

  subgraph Factorio Agent
    F_AGENT --> F_API -..- factorio-server
    subgraph factorio-server-wrapper
      subgraph factorio-server
        F_MOD1
      end
    end
  end

  subgraph Minecraft Agent
    MC_AGENT --> Mineflayer -..- minecraft-server
    subgraph factorio-server-wrapper
      subgraph factorio-server
        F_MOD1
      end
    end
  end

  XSAI --> Core
  XSAI --> F_AGENT
  XSAI --> MC_AGENT
%%{ init: { 'flowchart': { 'curve': 'catmullRom' } } }%%

flowchart TD
  subgraph deploy&bundle
    direction LR
    HFUP["hfup"]
    HF[/"HuggingFace Spaces"\]
    HFUP -...- UI -...-> HF
    HFUP -...- whisper-webgpu -...-> HF
    HFUP -...- moonshine-web -...-> HF
  end

Models used

Similar Projects

Open sourced ones

Non-open-sourced ones

Project Status

Repobeats analytics image

Acknowledgements

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for airi

Similar Open Source Tools

For similar tasks

For similar jobs