worker-vllm
The RunPod worker template for serving our large language model endpoints. Powered by vLLM.
Stars: 158
The worker-vLLM repository provides a serverless endpoint for deploying OpenAI-compatible vLLM models with blazing-fast performance. It supports deploying various model architectures, such as Aquila, Baichuan, BLOOM, ChatGLM, Command-R, DBRX, DeciLM, Falcon, Gemma, GPT-2, GPT BigCode, GPT-J, GPT-NeoX, InternLM, Jais, LLaMA, MiniCPM, Mistral, Mixtral, MPT, OLMo, OPT, Orion, Phi, Phi-3, Qwen, Qwen2, Qwen2MoE, StableLM, Starcoder2, Xverse, and Yi. Users can deploy models using pre-built Docker images or build custom images with specified arguments. The repository also supports OpenAI compatibility for chat completions, completions, and models, with customizable input parameters. Users can modify their OpenAI codebase to use the deployed vLLM worker and access a list of available models for deployment.
README:
Deploy OpenAI-Compatible Blazing-Fast LLM Endpoints powered by the vLLM Inference Engine on RunPod Serverless with just a few clicks.
Update 1.0.0preview is now available, use the image tag runpod/worker-vllm:dev-cuda12.1.0
or runpod/worker-vllm:dev-cuda11.8.0
.
Main Changes:
- vLLM was updated from version
0.3.3
to0.4.2
, adding compatibility for Llama 3 and other models, as well as increasing performance.
We will soon be adding more features from the updates, such as multi-LoRA, multi-modality, and more.
Worker vLLM is now cached on all RunPod machines, resulting in near-instant deployment! Previously, downloading and extracting the image took 3-5 minutes on average.
- Setting up the Serverless Worker
- Usage: OpenAI Compatibility
- Usage: standard
[!NOTE] You can now deploy from the dedicated UI on the RunPod console with all of the settings and choices listed. Try now by accessing in Explore or Serverless pages on the RunPod console!
We now offer a pre-built Docker Image for the vLLM Worker that you can configure entirely with Environment Variables when creating the RunPod Serverless Endpoint:
Below is a summary of the available RunPod Worker images, categorized by image stability and CUDA version compatibility.
CUDA Version | Stable Image Tag | Development Image Tag | Note |
---|---|---|---|
11.8.0 | runpod/worker-vllm:stable-cuda11.8.0 |
runpod/worker-vllm:dev-cuda11.8.0 |
Available on all RunPod Workers without additional selection needed. |
12.1.0 | runpod/worker-vllm:stable-cuda12.1.0 |
runpod/worker-vllm:dev-cuda12.1.0 |
When creating an Endpoint, select CUDA Version 12.3, 12.2 and 12.1 in the filter. |
- RunPod Account
Note:
0
is equivalent toFalse
and1
is equivalent toTrue
for boolean values.
Name | Default | Type/Choices | Description |
---|---|---|---|
LLM Settings | |||
MODEL_NAME *
|
- | str |
Hugging Face Model Repository (e.g., openchat/openchat-3.5-1210 ). |
MODEL_REVISION |
None |
str |
Model revision(branch) to load. |
MAX_MODEL_LEN |
Model's maximum | int |
Maximum number of tokens for the engine to handle per request. |
BASE_PATH |
/runpod-volume |
str |
Storage directory for Huggingface cache and model. Utilizes network storage if attached when pointed at /runpod-volume , which will have only one worker download the model once, which all workers will be able to load. If no network volume is present, creates a local directory within each worker. |
LOAD_FORMAT |
auto |
str |
Format to load model in. |
HF_TOKEN |
- | str |
Hugging Face token for private and gated models. |
QUANTIZATION |
None |
awq , squeezellm , gptq
|
Quantization of given model. The model must already be quantized. |
TRUST_REMOTE_CODE |
0 |
boolean as int
|
Trust remote code for Hugging Face models. Can help with Mixtral 8x7B, Quantized models, and unusual models/architectures. |
SEED |
0 |
int |
Sets random seed for operations. |
KV_CACHE_DTYPE |
auto |
auto , fp8
|
Data type for kv cache storage. Uses DTYPE if set to auto . |
DTYPE |
auto |
auto , half , float16 , bfloat16 , float , float32
|
Sets datatype/precision for model weights and activations. |
Tokenizer Settings | |||
TOKENIZER_NAME |
None |
str |
Tokenizer repository to use a different tokenizer than the model's default. |
TOKENIZER_REVISION |
None |
str |
Tokenizer revision to load. |
CUSTOM_CHAT_TEMPLATE |
None |
str of single-line jinja template |
Custom chat jinja template. More Info |
System, GPU, and Tensor Parallelism(Multi-GPU) Settings | |||
GPU_MEMORY_UTILIZATION |
0.95 |
float |
Sets GPU VRAM utilization. |
MAX_PARALLEL_LOADING_WORKERS |
None |
int |
Load model sequentially in multiple batches, to avoid RAM OOM when using tensor parallel and large models. |
BLOCK_SIZE |
16 |
8 , 16 , 32
|
Token block size for contiguous chunks of tokens. |
SWAP_SPACE |
4 |
int |
CPU swap space size (GiB) per GPU. |
ENFORCE_EAGER |
0 |
boolean as int
|
Always use eager-mode PyTorch. If False(0 ), will use eager mode and CUDA graph in hybrid for maximal performance and flexibility. |
MAX_CONTEXT_LEN_TO_CAPTURE |
8192 |
int |
Maximum context length covered by CUDA graphs. When a sequence has context length larger than this, we fall back to eager mode. |
DISABLE_CUSTOM_ALL_REDUCE |
0 |
int |
Enables or disables custom all reduce. |
Streaming Batch Size Settings: | |||
DEFAULT_BATCH_SIZE |
50 |
int |
Default and Maximum batch size for token streaming to reduce HTTP calls. |
DEFAULT_MIN_BATCH_SIZE |
1 |
int |
Batch size for the first request, which will be multiplied by the growth factor every subsequent request. |
DEFAULT_BATCH_SIZE_GROWTH_FACTOR |
3 |
float |
Growth factor for dynamic batch size. |
The way this works is that the first request will have a batch size of DEFAULT_MIN_BATCH_SIZE , and each subsequent request will have a batch size of previous_batch_size * DEFAULT_BATCH_SIZE_GROWTH_FACTOR . This will continue until the batch size reaches DEFAULT_BATCH_SIZE . E.g. for the default values, the batch sizes will be 1, 3, 9, 27, 50, 50, 50, ... . You can also specify this per request, with inputs max_batch_size , min_batch_size , and batch_size_growth_factor . This has nothing to do with vLLM's internal batching, but rather the number of tokens sent in each HTTP request from the worker |
|||
OpenAI Settings | |||
RAW_OPENAI_OUTPUT |
1 |
boolean as int
|
Enables raw OpenAI SSE format string output when streaming. Required to be enabled (which it is by default) for OpenAI compatibility. |
OPENAI_SERVED_MODEL_NAME_OVERRIDE |
None |
str |
Overrides the name of the served model from model repo/path to specified name, which you will then be able to use the value for the model parameter when making OpenAI requests |
OPENAI_RESPONSE_ROLE |
assistant |
str |
Role of the LLM's Response in OpenAI Chat Completions. |
Serverless Settings | |||
MAX_CONCURRENCY |
300 |
int |
Max concurrent requests per worker. vLLM has an internal queue, so you don't have to worry about limiting by VRAM, this is for improving scaling/load balancing efficiency |
DISABLE_LOG_STATS |
1 |
boolean as int
|
Enables or disables vLLM stats logging. |
DISABLE_LOG_REQUESTS |
1 |
boolean as int
|
Enables or disables vLLM request logging. |
[!TIP] If you are facing issues when using Mixtral 8x7B, Quantized models, or handling unusual models/architectures, try setting
TRUST_REMOTE_CODE
to1
.
To build an image with the model baked in, you must specify the following docker arguments when building the image.
- RunPod Account
- Docker
-
Required
MODEL_NAME
-
Optional
-
MODEL_REVISION
: Model revision to load (default:main
). -
BASE_PATH
: Storage directory where huggingface cache and model will be located. (default:/runpod-volume
, which will utilize network storage if you attach it or create a local directory within the image if you don't. If your intention is to bake the model into the image, you should set this to something like/models
to make sure there are no issues if you were to accidentally attach network storage.) QUANTIZATION
-
WORKER_CUDA_VERSION
:11.8.0
or12.1.0
(default:11.8.0
due to a small number of workers not having CUDA 12.1 support yet.12.1.0
is recommended for optimal performance). -
TOKENIZER_NAME
: Tokenizer repository if you would like to use a different tokenizer than the one that comes with the model. (default:None
, which uses the model's tokenizer) -
TOKENIZER_REVISION
: Tokenizer revision to load (default:main
).
-
For the remaining settings, you may apply them as environment variables when running the container. Supported environment variables are listed in the Environment Variables section.
sudo docker build -t username/image:tag --build-arg MODEL_NAME="openchat/openchat_3.5" --build-arg BASE_PATH="/models" .
If the model you would like to deploy is private or gated, you will need to include it during build time as a Docker secret, which will protect it from being exposed in the image and on DockerHub.
- Enable Docker BuildKit (required for secrets).
export DOCKER_BUILDKIT=1
- Export your Hugging Face token as an environment variable
export HF_TOKEN="your_token_here"
- Add the token as a secret when building
docker build -t username/image:tag --secret id=HF_TOKEN --build-arg MODEL_NAME="openchat/openchat_3.5" .
Below are all supported model architectures (and examples of each) that you can deploy using the vLLM Worker. You can deploy any model on HuggingFace, as long as its base architecture is one of the following:
- Aquila & Aquila2 (
BAAI/AquilaChat2-7B
,BAAI/AquilaChat2-34B
,BAAI/Aquila-7B
,BAAI/AquilaChat-7B
, etc.) - Baichuan & Baichuan2 (
baichuan-inc/Baichuan2-13B-Chat
,baichuan-inc/Baichuan-7B
, etc.) - BLOOM (
bigscience/bloom
,bigscience/bloomz
, etc.) - ChatGLM (
THUDM/chatglm2-6b
,THUDM/chatglm3-6b
, etc.) - Command-R (
CohereForAI/c4ai-command-r-v01
, etc.) - DBRX (
databricks/dbrx-base
,databricks/dbrx-instruct
etc.) - DeciLM (
Deci/DeciLM-7B
,Deci/DeciLM-7B-instruct
, etc.) - Falcon (
tiiuae/falcon-7b
,tiiuae/falcon-40b
,tiiuae/falcon-rw-7b
, etc.) - Gemma (
google/gemma-2b
,google/gemma-7b
, etc.) - GPT-2 (
gpt2
,gpt2-xl
, etc.) - GPT BigCode (
bigcode/starcoder
,bigcode/gpt_bigcode-santacoder
, etc.) - GPT-J (
EleutherAI/gpt-j-6b
,nomic-ai/gpt4all-j
, etc.) - GPT-NeoX (
EleutherAI/gpt-neox-20b
,databricks/dolly-v2-12b
,stabilityai/stablelm-tuned-alpha-7b
, etc.) - InternLM (
internlm/internlm-7b
,internlm/internlm-chat-7b
, etc.) - InternLM2 (
internlm/internlm2-7b
,internlm/internlm2-chat-7b
, etc.) - Jais (
core42/jais-13b
,core42/jais-13b-chat
,core42/jais-30b-v3
,core42/jais-30b-chat-v3
, etc.) - LLaMA, Llama 2, and Meta Llama 3 (
meta-llama/Meta-Llama-3-8B-Instruct
,meta-llama/Meta-Llama-3-70B-Instruct
,meta-llama/Llama-2-70b-hf
,lmsys/vicuna-13b-v1.3
,young-geng/koala
,openlm-research/open_llama_13b
, etc.) - MiniCPM (
openbmb/MiniCPM-2B-sft-bf16
,openbmb/MiniCPM-2B-dpo-bf16
, etc.) - Mistral (
mistralai/Mistral-7B-v0.1
,mistralai/Mistral-7B-Instruct-v0.1
, etc.) - Mixtral (
mistralai/Mixtral-8x7B-v0.1
,mistralai/Mixtral-8x7B-Instruct-v0.1
,mistral-community/Mixtral-8x22B-v0.1
, etc.) - MPT (
mosaicml/mpt-7b
,mosaicml/mpt-30b
, etc.) - OLMo (
allenai/OLMo-1B-hf
,allenai/OLMo-7B-hf
, etc.) - OPT (
facebook/opt-66b
,facebook/opt-iml-max-30b
, etc.) - Orion (
OrionStarAI/Orion-14B-Base
,OrionStarAI/Orion-14B-Chat
, etc.) - Phi (
microsoft/phi-1_5
,microsoft/phi-2
, etc.) - Phi-3 (
microsoft/Phi-3-mini-4k-instruct
,microsoft/Phi-3-mini-128k-instruct
, etc.) - Qwen (
Qwen/Qwen-7B
,Qwen/Qwen-7B-Chat
, etc.) - Qwen2 (
Qwen/Qwen1.5-7B
,Qwen/Qwen1.5-7B-Chat
, etc.) - Qwen2MoE (
Qwen/Qwen1.5-MoE-A2.7B
,Qwen/Qwen1.5-MoE-A2.7B-Chat
, etc.) - StableLM(
stabilityai/stablelm-3b-4e1t
,stabilityai/stablelm-base-alpha-7b-v2
, etc.) - Starcoder2(
bigcode/starcoder2-3b
,bigcode/starcoder2-7b
,bigcode/starcoder2-15b
, etc.) - Xverse (
xverse/XVERSE-7B-Chat
,xverse/XVERSE-13B-Chat
,xverse/XVERSE-65B-Chat
, etc.) - Yi (
01-ai/Yi-6B
,01-ai/Yi-34B
, etc.)
The vLLM Worker is fully compatible with OpenAI's API, and you can use it with any OpenAI Codebase by changing only 3 lines in total. The supported routes are Chat Completions, Completions and Models - with both streaming and non-streaming.
Python (similar to Node.js, etc.):
-
When initializing the OpenAI Client in your code, change the
api_key
to your RunPod API Key and thebase_url
to your RunPod Serverless Endpoint URL in the following format:https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1
, filling in your deployed endpoint ID. For example, if your Endpoint ID isabc1234
, the URL would behttps://api.runpod.ai/v2/abc1234/openai/v1
.- Before:
from openai import OpenAI client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY"))
- After:
from openai import OpenAI client = OpenAI( api_key=os.environ.get("RUNPOD_API_KEY"), base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1", )
-
Change the
model
parameter to your deployed model's name whenever using Completions or Chat Completions.- Before:
response = client.chat.completions.create( model="gpt-3.5-turbo", messages=[{"role": "user", "content": "Why is RunPod the best platform?"}], temperature=0, max_tokens=100, )
- After:
response = client.chat.completions.create( model="<YOUR DEPLOYED MODEL REPO/NAME>", messages=[{"role": "user", "content": "Why is RunPod the best platform?"}], temperature=0, max_tokens=100, )
Using http requests:
- Change the
Authorization
header to your RunPod API Key and theurl
to your RunPod Serverless Endpoint URL in the following format:https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1
- Before:
curl https://api.openai.com/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer $OPENAI_API_KEY" \ -d '{ "model": "gpt-4", "messages": [ { "role": "user", "content": "Why is RunPod the best platform?" } ], "temperature": 0, "max_tokens": 100 }'
- After:
curl https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1/chat/completions \ -H "Content-Type: application/json" \ -H "Authorization: Bearer <YOUR OPENAI API KEY>" \ -d '{ "model": "<YOUR DEPLOYED MODEL REPO/NAME>", "messages": [ { "role": "user", "content": "Why is RunPod the best platform?" } ], "temperature": 0, "max_tokens": 100 }'
When using the chat completion feature of the vLLM Serverless Endpoint Worker, you can customize your requests with the following parameters:
Supported Chat Completions Inputs and Descriptions
Parameter | Type | Default Value | Description |
---|---|---|---|
messages |
Union[str, List[Dict[str, str]]] | List of messages, where each message is a dictionary with a role and content . The model's chat template will be applied to the messages automatically, so the model must have one or it should be specified as CUSTOM_CHAT_TEMPLATE env var. |
|
model |
str | The model repo that you've deployed on your RunPod Serverless Endpoint. If you are unsure what the name is or are baking the model in, use the guide to get the list of available models in the Examples: Using your RunPod endpoint with OpenAI section | |
temperature |
Optional[float] | 0.7 | Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling. |
top_p |
Optional[float] | 1.0 | Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. |
n |
Optional[int] | 1 | Number of output sequences to return for the given prompt. |
max_tokens |
Optional[int] | None | Maximum number of tokens to generate per output sequence. |
seed |
Optional[int] | None | Random seed to use for the generation. |
stop |
Optional[Union[str, List[str]]] | list | List of strings that stop the generation when they are generated. The returned output will not contain the stop strings. |
stream |
Optional[bool] | False | Whether to stream or not |
presence_penalty |
Optional[float] | 0.0 | Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens. |
frequency_penalty |
Optional[float] | 0.0 | Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens. |
logit_bias |
Optional[Dict[str, float]] | None | Unsupported by vLLM |
user |
Optional[str] | None | Unsupported by vLLM |
Additional parameters supported by vLLM: | |||
best_of |
Optional[int] | None | Number of output sequences that are generated from the prompt. From these best_of sequences, the top n sequences are returned. best_of must be greater than or equal to n . This is treated as the beam width when use_beam_search is True. By default, best_of is set to n . |
top_k |
Optional[int] | -1 | Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens. |
ignore_eos |
Optional[bool] | False | Whether to ignore the EOS token and continue generating tokens after the EOS token is generated. |
use_beam_search |
Optional[bool] | False | Whether to use beam search instead of sampling. |
stop_token_ids |
Optional[List[int]] | list | List of tokens that stop the generation when they are generated. The returned output will contain the stop tokens unless the stop tokens are special tokens. |
skip_special_tokens |
Optional[bool] | True | Whether to skip special tokens in the output. |
spaces_between_special_tokens |
Optional[bool] | True | Whether to add spaces between special tokens in the output. Defaults to True. |
add_generation_prompt |
Optional[bool] | True | Read more here |
echo |
Optional[bool] | False | Echo back the prompt in addition to the completion |
repetition_penalty |
Optional[float] | 1.0 | Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the model to repeat tokens. |
min_p |
Optional[float] | 0.0 | Float that represents the minimum probability for a token to |
length_penalty |
Optional[float] | 1.0 | Float that penalizes sequences based on their length. Used in beam search.. |
include_stop_str_in_output |
Optional[bool] | False | Whether to include the stop strings in output text. Defaults to False. |
Supported Completions Inputs and Descriptions
Parameter | Type | Default Value | Description |
---|---|---|---|
model |
str | The model repo that you've deployed on your RunPod Serverless Endpoint. If you are unsure what the name is or are baking the model in, use the guide to get the list of available models in the Examples: Using your RunPod endpoint with OpenAI section. | |
prompt |
Union[List[int], List[List[int]], str, List[str]] | A string, array of strings, array of tokens, or array of token arrays to be used as the input for the model. | |
suffix |
Optional[str] | None | A string to be appended to the end of the generated text. |
max_tokens |
Optional[int] | 16 | Maximum number of tokens to generate per output sequence. |
temperature |
Optional[float] | 1.0 | Float that controls the randomness of the sampling. Lower values make the model more deterministic, while higher values make the model more random. Zero means greedy sampling. |
top_p |
Optional[float] | 1.0 | Float that controls the cumulative probability of the top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. |
n |
Optional[int] | 1 | Number of output sequences to return for the given prompt. |
stream |
Optional[bool] | False | Whether to stream the output. |
logprobs |
Optional[int] | None | Number of log probabilities to return per output token. |
echo |
Optional[bool] | False | Whether to echo back the prompt in addition to the completion. |
stop |
Optional[Union[str, List[str]]] | list | List of strings that stop the generation when they are generated. The returned output will not contain the stop strings. |
seed |
Optional[int] | None | Random seed to use for the generation. |
presence_penalty |
Optional[float] | 0.0 | Float that penalizes new tokens based on whether they appear in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens. |
frequency_penalty |
Optional[float] | 0.0 | Float that penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage the model to use new tokens, while values < 0 encourage the model to repeat tokens. |
best_of |
Optional[int] | None | Number of output sequences that are generated from the prompt. From these best_of sequences, the top n sequences are returned. best_of must be greater than or equal to n . This parameter influences the diversity of the output. |
logit_bias |
Optional[Dict[str, float]] | None | Dictionary of token IDs to biases. |
user |
Optional[str] | None | User identifier for personalizing responses. (Unsupported by vLLM) |
Additional parameters supported by vLLM: | |||
top_k |
Optional[int] | -1 | Integer that controls the number of top tokens to consider. Set to -1 to consider all tokens. |
ignore_eos |
Optional[bool] | False | Whether to ignore the End Of Sentence token and continue generating tokens after the EOS token is generated. |
use_beam_search |
Optional[bool] | False | Whether to use beam search instead of sampling for generating outputs. |
stop_token_ids |
Optional[List[int]] | list | List of tokens that stop the generation when they are generated. The returned output will contain the stop tokens unless the stop tokens are special tokens. |
skip_special_tokens |
Optional[bool] | True | Whether to skip special tokens in the output. |
spaces_between_special_tokens |
Optional[bool] | True | Whether to add spaces between special tokens in the output. Defaults to True. |
repetition_penalty |
Optional[float] | 1.0 | Float that penalizes new tokens based on whether they appear in the prompt and the generated text so far. Values > 1 encourage the model to use new tokens, while values < 1 encourage the model to repeat tokens. |
min_p |
Optional[float] | 0.0 | Float that represents the minimum probability for a token to be considered, relative to the most likely token. Must be in [0, 1]. Set to 0 to disable. |
length_penalty |
Optional[float] | 1.0 | Float that penalizes sequences based on their length. Used in beam search. |
include_stop_str_in_output |
Optional[bool] | False | Whether to include the stop strings in output text. Defaults to False. |
First, initialize the OpenAI Client with your RunPod API Key and Endpoint URL:
from openai import OpenAI
import os
# Initialize the OpenAI Client with your RunPod API Key and Endpoint URL
client = OpenAI(
api_key=os.environ.get("RUNPOD_API_KEY"),
base_url="https://api.runpod.ai/v2/<YOUR ENDPOINT ID>/openai/v1",
)
This is the format used for GPT-4 and focused on instruction-following and chat. Examples of Open Source chat/instruct models include meta-llama/Llama-2-7b-chat-hf
, mistralai/Mixtral-8x7B-Instruct-v0.1
, openchat/openchat-3.5-0106
, NousResearch/Nous-Hermes-2-Mistral-7B-DPO
and more. However, if your model is a completion-style model with no chat/instruct fine-tune and/or does not have a chat template, you can still use this if you provide a chat template with the environment variable CUSTOM_CHAT_TEMPLATE
.
-
Streaming:
# Create a chat completion stream response_stream = client.chat.completions.create( model="<YOUR DEPLOYED MODEL REPO/NAME>", messages=[{"role": "user", "content": "Why is RunPod the best platform?"}], temperature=0, max_tokens=100, stream=True, ) # Stream the response for response in response_stream: print(chunk.choices[0].delta.content or "", end="", flush=True)
-
Non-Streaming:
# Create a chat completion response = client.chat.completions.create( model="<YOUR DEPLOYED MODEL REPO/NAME>", messages=[{"role": "user", "content": "Why is RunPod the best platform?"}], temperature=0, max_tokens=100, ) # Print the response print(response.choices[0].message.content)
This is the format used for models like GPT-3 and is meant for completing the text you provide. Instead of responding to your message, it will try to complete it. Examples of Open Source completions models include meta-llama/Llama-2-7b-hf
, mistralai/Mixtral-8x7B-v0.1
, Qwen/Qwen-72B
, and more. However, you can use any model with this format.
-
Streaming:
# Create a completion stream response_stream = client.completions.create( model="<YOUR DEPLOYED MODEL REPO/NAME>", prompt="Runpod is the best platform because", temperature=0, max_tokens=100, stream=True, ) # Stream the response for response in response_stream: print(response.choices[0].text or "", end="", flush=True)
-
Non-Streaming:
# Create a completion response = client.completions.create( model="<YOUR DEPLOYED MODEL REPO/NAME>", prompt="Runpod is the best platform because", temperature=0, max_tokens=100, ) # Print the response print(response.choices[0].text)
In the case of baking the model into the image, sometimes the repo may not be accepted as the model
in the request. In this case, you can list the available models as shown below and use that name.
models_response = client.models.list()
list_of_models = [model.id for model in models_response]
print(list_of_models)
Click to expand table
You may either use a prompt
or a list of messages
as input. If you use messages
, the model's chat template will be applied to the messages automatically, so the model must have one. If you use prompt
, you may optionally apply the model's chat template to the prompt by setting apply_chat_template
to true
.
Argument | Type | Default | Description |
---|---|---|---|
prompt |
str | Prompt string to generate text based on. | |
messages |
list[dict[str, str]] | List of messages, which will automatically have the model's chat template applied. Overrides prompt . |
|
apply_chat_template |
bool | False | Whether to apply the model's chat template to the prompt . |
sampling_params |
dict | {} | Sampling parameters to control the generation, like temperature, top_p, etc. You can find all available parameters in the Sampling Parameters section below. |
stream |
bool | False | Whether to enable streaming of output. If True, responses are streamed as they are generated. |
max_batch_size |
int | env var DEFAULT_BATCH_SIZE
|
The maximum number of tokens to stream every HTTP POST call. |
min_batch_size |
int | env var DEFAULT_MIN_BATCH_SIZE
|
The minimum number of tokens to stream every HTTP POST call. |
batch_size_growth_factor |
int | env var DEFAULT_BATCH_SIZE_GROWTH_FACTOR
|
The growth factor by which min_batch_size will be multiplied for each call until max_batch_size is reached. |
Below are all available sampling parameters that you can specify in the sampling_params
dictionary. If you do not specify any of these parameters, the default values will be used.
Click to expand table
Argument | Type | Default | Description |
---|---|---|---|
n |
int | 1 | Number of output sequences generated from the prompt. The top n sequences are returned. |
best_of |
Optional[int] | n |
Number of output sequences generated from the prompt. The top n sequences are returned from these best_of sequences. Must be ≥ n . Treated as beam width in beam search. Default is n . |
presence_penalty |
float | 0.0 | Penalizes new tokens based on their presence in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition. |
frequency_penalty |
float | 0.0 | Penalizes new tokens based on their frequency in the generated text so far. Values > 0 encourage new tokens, values < 0 encourage repetition. |
repetition_penalty |
float | 1.0 | Penalizes new tokens based on their appearance in the prompt and generated text. Values > 1 encourage new tokens, values < 1 encourage repetition. |
temperature |
float | 1.0 | Controls the randomness of sampling. Lower values make it more deterministic, higher values make it more random. Zero means greedy sampling. |
top_p |
float | 1.0 | Controls the cumulative probability of top tokens to consider. Must be in (0, 1]. Set to 1 to consider all tokens. |
top_k |
int | -1 | Controls the number of top tokens to consider. Set to -1 to consider all tokens. |
min_p |
float | 0.0 | Represents the minimum probability for a token to be considered, relative to the most likely token. Must be in [0, 1]. Set to 0 to disable. |
use_beam_search |
bool | False | Whether to use beam search instead of sampling. |
length_penalty |
float | 1.0 | Penalizes sequences based on their length. Used in beam search. |
early_stopping |
Union[bool, str] | False | Controls stopping condition in beam search. Can be True , False , or "never" . |
stop |
Union[None, str, List[str]] | None | List of strings that stop generation when produced. The output will not contain these strings. |
stop_token_ids |
Optional[List[int]] | None | List of token IDs that stop generation when produced. Output contains these tokens unless they are special tokens. |
ignore_eos |
bool | False | Whether to ignore the End-Of-Sequence token and continue generating tokens after its generation. |
max_tokens |
int | 16 | Maximum number of tokens to generate per output sequence. |
skip_special_tokens |
bool | True | Whether to skip special tokens in the output. |
spaces_between_special_tokens |
bool | True | Whether to add spaces between special tokens in the output. |
You may either use a prompt
or a list of messages
as input.
-
prompt
The prompt string can be any string, and the model's chat template will not be applied to it unlessapply_chat_template
is set totrue
, in which case it will be treated as a user message.Example:
"prompt": "..."
-
messages
Your list can contain any number of messages, and each message usually can have any role from the following list:user
assistant
system
However, some models may have different roles, so you should check the model's chat template to see which roles are required.
The model's chat template will be applied to the messages automatically, so the model must have one.
Example:
"messages": [ { "role": "system", "content": "..." }, { "role": "user", "content": "..." }, { "role": "assistant", "content": "..." } ]
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for worker-vllm
Similar Open Source Tools
worker-vllm
The worker-vLLM repository provides a serverless endpoint for deploying OpenAI-compatible vLLM models with blazing-fast performance. It supports deploying various model architectures, such as Aquila, Baichuan, BLOOM, ChatGLM, Command-R, DBRX, DeciLM, Falcon, Gemma, GPT-2, GPT BigCode, GPT-J, GPT-NeoX, InternLM, Jais, LLaMA, MiniCPM, Mistral, Mixtral, MPT, OLMo, OPT, Orion, Phi, Phi-3, Qwen, Qwen2, Qwen2MoE, StableLM, Starcoder2, Xverse, and Yi. Users can deploy models using pre-built Docker images or build custom images with specified arguments. The repository also supports OpenAI compatibility for chat completions, completions, and models, with customizable input parameters. Users can modify their OpenAI codebase to use the deployed vLLM worker and access a list of available models for deployment.
airflow-chart
This Helm chart bootstraps an Airflow deployment on a Kubernetes cluster using the Helm package manager. The version of this chart does not correlate to any other component. Users should not expect feature parity between OSS airflow chart and the Astronomer airflow-chart for identical version numbers. To install this helm chart remotely (using helm 3) kubectl create namespace airflow helm repo add astronomer https://helm.astronomer.io helm install airflow --namespace airflow astronomer/airflow To install this repository from source sh kubectl create namespace airflow helm install --namespace airflow . Prerequisites: Kubernetes 1.12+ Helm 3.6+ PV provisioner support in the underlying infrastructure Installing the Chart: sh helm install --name my-release . The command deploys Airflow on the Kubernetes cluster in the default configuration. The Parameters section lists the parameters that can be configured during installation. Upgrading the Chart: First, look at the updating documentation to identify any backwards-incompatible changes. To upgrade the chart with the release name `my-release`: sh helm upgrade --name my-release . Uninstalling the Chart: To uninstall/delete the `my-release` deployment: sh helm delete my-release The command removes all the Kubernetes components associated with the chart and deletes the release. Updating DAGs: Bake DAGs in Docker image The recommended way to update your DAGs with this chart is to build a new docker image with the latest code (`docker build -t my-company/airflow:8a0da78 .`), push it to an accessible registry (`docker push my-company/airflow:8a0da78`), then update the Airflow pods with that image: sh helm upgrade my-release . --set images.airflow.repository=my-company/airflow --set images.airflow.tag=8a0da78 Docker Images: The Airflow image that are referenced as the default values in this chart are generated from this repository: https://github.com/astronomer/ap-airflow. Other non-airflow images used in this chart are generated from this repository: https://github.com/astronomer/ap-vendor. Parameters: The complete list of parameters supported by the community chart can be found on the Parameteres Reference page, and can be set under the `airflow` key in this chart. The following tables lists the configurable parameters of the Astronomer chart and their default values. | Parameter | Description | Default | | :----------------------------- | :-------------------------------------------------------------------------------------------------------- | :---------------------------- | | `ingress.enabled` | Enable Kubernetes Ingress support | `false` | | `ingress.acme` | Add acme annotations to Ingress object | `false` | | `ingress.tlsSecretName` | Name of secret that contains a TLS secret | `~` | | `ingress.webserverAnnotations` | Annotations added to Webserver Ingress object | `{}` | | `ingress.flowerAnnotations` | Annotations added to Flower Ingress object | `{}` | | `ingress.baseDomain` | Base domain for VHOSTs | `~` | | `ingress.auth.enabled` | Enable auth with Astronomer Platform | `true` | | `extraObjects` | Extra K8s Objects to deploy (these are passed through `tpl`). More about Extra Objects. | `[]` | | `sccEnabled` | Enable security context constraints required for OpenShift | `false` | | `authSidecar.enabled` | Enable authSidecar | `false` | | `authSidecar.repository` | The image for the auth sidecar proxy | `nginxinc/nginx-unprivileged` | | `authSidecar.tag` | The image tag for the auth sidecar proxy | `stable` | | `authSidecar.pullPolicy` | The K8s pullPolicy for the the auth sidecar proxy image | `IfNotPresent` | | `authSidecar.port` | The port the auth sidecar exposes | `8084` | | `gitSyncRelay.enabled` | Enables git sync relay feature. | `False` | | `gitSyncRelay.repo.url` | Upstream URL to the git repo to clone. | `~` | | `gitSyncRelay.repo.branch` | Branch of the upstream git repo to checkout. | `main` | | `gitSyncRelay.repo.depth` | How many revisions to check out. Leave as default `1` except in dev where history is needed. | `1` | | `gitSyncRelay.repo.wait` | Seconds to wait before pulling from the upstream remote. | `60` | | `gitSyncRelay.repo.subPath` | Path to the dags directory within the git repository. | `~` | Specify each parameter using the `--set key=value[,key=value]` argument to `helm install`. For example, sh helm install --name my-release --set executor=CeleryExecutor --set enablePodLaunching=false . Walkthrough using kind: Install kind, and create a cluster We recommend testing with Kubernetes 1.25+, example: sh kind create cluster --image kindest/node:v1.25.11 Confirm it's up: sh kubectl cluster-info --context kind-kind Add Astronomer's Helm repo sh helm repo add astronomer https://helm.astronomer.io helm repo update Create namespace + install the chart sh kubectl create namespace airflow helm install airflow -n airflow astronomer/airflow It may take a few minutes. Confirm the pods are up: sh kubectl get pods --all-namespaces helm list -n airflow Run `kubectl port-forward svc/airflow-webserver 8080:8080 -n airflow` to port-forward the Airflow UI to http://localhost:8080/ to confirm Airflow is working. Login as _admin_ and password _admin_. Build a Docker image from your DAGs: 1. Start a project using astro-cli, which will generate a Dockerfile, and load your DAGs in. You can test locally before pushing to kind with `astro airflow start`. `sh mkdir my-airflow-project && cd my-airflow-project astro dev init` 2. Then build the image: `sh docker build -t my-dags:0.0.1 .` 3. Load the image into kind: `sh kind load docker-image my-dags:0.0.1` 4. Upgrade Helm deployment: sh helm upgrade airflow -n airflow --set images.airflow.repository=my-dags --set images.airflow.tag=0.0.1 astronomer/airflow Extra Objects: This chart can deploy extra Kubernetes objects (assuming the role used by Helm can manage them). For Astronomer Cloud and Enterprise, the role permissions can be found in the Commander role. yaml extraObjects: - apiVersion: batch/v1beta1 kind: CronJob metadata: name: "{{ .Release.Name }}-somejob" spec: schedule: "*/10 * * * *" concurrencyPolicy: Forbid jobTemplate: spec: template: spec: containers: - name: myjob image: ubuntu command: - echo args: - hello restartPolicy: OnFailure Contributing: Check out our contributing guide! License: Apache 2.0 with Commons Clause
rwkv.cpp
rwkv.cpp is a port of BlinkDL/RWKV-LM to ggerganov/ggml, supporting FP32, FP16, and quantized INT4, INT5, and INT8 inference. It focuses on CPU but also supports cuBLAS. The project provides a C library rwkv.h and a Python wrapper. RWKV is a large language model architecture with models like RWKV v5 and v6. It requires only state from the previous step for calculations, making it CPU-friendly on large context lengths. Users are advised to test all available formats for perplexity and latency on a representative dataset before serious use.
mistral.rs
Mistral.rs is a fast LLM inference platform written in Rust. We support inference on a variety of devices, quantization, and easy-to-use application with an Open-AI API compatible HTTP server and Python bindings.
Large-Language-Models-play-StarCraftII
Large Language Models Play StarCraft II is a project that explores the capabilities of large language models (LLMs) in playing the game StarCraft II. The project introduces TextStarCraft II, a textual environment for the game, and a Chain of Summarization method for analyzing game information and making strategic decisions. Through experiments, the project demonstrates that LLM agents can defeat the built-in AI at a challenging difficulty level. The project provides benchmarks and a summarization approach to enhance strategic planning and interpretability in StarCraft II gameplay.
llm-structured-output-benchmarks
Benchmark various LLM Structured Output frameworks like Instructor, Mirascope, Langchain, LlamaIndex, Fructose, Marvin, Outlines, LMFormatEnforcer, etc on tasks like multi-label classification, named entity recognition, synthetic data generation. The tool provides benchmark results, methodology, instructions to run the benchmark, add new data, and add a new framework. It also includes a roadmap for framework-related tasks, contribution guidelines, citation information, and feedback request.
BricksLLM
BricksLLM is a cloud native AI gateway written in Go. Currently, it provides native support for OpenAI, Anthropic, Azure OpenAI and vLLM. BricksLLM aims to provide enterprise level infrastructure that can power any LLM production use cases. Here are some use cases for BricksLLM: * Set LLM usage limits for users on different pricing tiers * Track LLM usage on a per user and per organization basis * Block or redact requests containing PIIs * Improve LLM reliability with failovers, retries and caching * Distribute API keys with rate limits and cost limits for internal development/production use cases * Distribute API keys with rate limits and cost limits for students
pr-pilot
PR Pilot is an AI-powered tool designed to assist users in their daily workflow by delegating routine work to AI with confidence and predictability. It integrates seamlessly with popular development tools and allows users to interact with it through a Command-Line Interface, Python SDK, REST API, and Smart Workflows. Users can automate tasks such as generating PR titles and descriptions, summarizing and posting issues, and formatting README files. The tool aims to save time and enhance productivity by providing AI-powered solutions for common development tasks.
stable-diffusion-webui
Stable Diffusion WebUI Docker Image allows users to run Automatic1111 WebUI in a docker container locally or in the cloud. The images do not bundle models or third-party configurations, requiring users to use a provisioning script for container configuration. It supports NVIDIA CUDA, AMD ROCm, and CPU platforms, with additional environment variables for customization and pre-configured templates for Vast.ai and Runpod.io. The service is password protected by default, with options for version pinning, startup flags, and service management using supervisorctl.
receipt-scanner
The receipt-scanner repository is an AI-Powered Receipt and Invoice Scanner for Laravel that allows users to easily extract structured receipt data from images, PDFs, and emails within their Laravel application using OpenAI. It provides a light wrapper around OpenAI Chat and Completion endpoints, supports various input formats, and integrates with Textract for OCR functionality. Users can install the package via composer, publish configuration files, and use it to extract data from plain text, PDFs, images, Word documents, and web content. The scanned receipt data is parsed into a DTO structure with main classes like Receipt, Merchant, and LineItem.
aikit
AIKit is a one-stop shop to quickly get started to host, deploy, build and fine-tune large language models (LLMs). AIKit offers two main capabilities: Inference: AIKit uses LocalAI, which supports a wide range of inference capabilities and formats. LocalAI provides a drop-in replacement REST API that is OpenAI API compatible, so you can use any OpenAI API compatible client, such as Kubectl AI, Chatbot-UI and many more, to send requests to open-source LLMs! Fine Tuning: AIKit offers an extensible fine tuning interface. It supports Unsloth for fast, memory efficient, and easy fine-tuning experience.
scrape-it-now
Scrape It Now is a versatile tool for scraping websites with features like decoupled architecture, CLI functionality, idempotent operations, and content storage options. The tool includes a scraper component for efficient scraping, ad blocking, link detection, markdown extraction, dynamic content loading, and anonymity features. It also offers an indexer component for creating AI search indexes, chunking content, embedding chunks, and enabling semantic search. The tool supports various configurations for Azure services and local storage, providing flexibility and scalability for web scraping and indexing tasks.
gollama
Gollama is a delightful tool that brings Ollama, your offline conversational AI companion, directly into your terminal. It provides a fun and interactive way to generate responses from various models without needing internet connectivity. Whether you're brainstorming ideas, exploring creative writing, or just looking for inspiration, Gollama is here to assist you. The tool offers an interactive interface, customizable prompts, multiple models selection, and visual feedback to enhance user experience. It can be installed via different methods like downloading the latest release, using Go, running with Docker, or building from source. Users can interact with Gollama through various options like specifying a custom base URL, prompt, model, and enabling raw output mode. The tool supports different modes like interactive, piped, CLI with image, and TUI with image. Gollama relies on third-party packages like bubbletea, glamour, huh, and lipgloss. The roadmap includes implementing piped mode, support for extracting codeblocks, copying responses/codeblocks to clipboard, GitHub Actions for automated releases, and downloading models directly from Ollama using the rest API. Contributions are welcome, and the project is licensed under the MIT License.
StableToolBench
StableToolBench is a new benchmark developed to address the instability of Tool Learning benchmarks. It aims to balance stability and reality by introducing features like Virtual API System, Solvable Queries, and Stable Evaluation System. The benchmark ensures consistency through a caching system and API simulators, filters queries based on solvability using LLMs, and evaluates model performance using GPT-4 with metrics like Solvable Pass Rate and Solvable Win Rate.
llama3.java
Llama3.java is a practical Llama 3 inference tool implemented in a single Java file. It serves as the successor of llama2.java and is designed for testing and tuning compiler optimizations and features on the JVM, especially for the Graal compiler. The tool features a GGUF format parser, Llama 3 tokenizer, Grouped-Query Attention inference, support for Q8_0 and Q4_0 quantizations, fast matrix-vector multiplication routines using Java's Vector API, and a simple CLI with 'chat' and 'instruct' modes. Users can download quantized .gguf files from huggingface.co for model usage and can also manually quantize to pure 'Q4_0'. The tool requires Java 21+ and supports running from source or building a JAR file for execution. Performance benchmarks show varying tokens/s rates for different models and implementations on different hardware setups.
aicommit2
AICommit2 is a Reactive CLI tool that streamlines interactions with various AI providers such as OpenAI, Anthropic Claude, Gemini, Mistral AI, Cohere, and unofficial providers like Huggingface and Clova X. Users can request multiple AI simultaneously to generate git commit messages without waiting for all AI responses. The tool runs 'git diff' to grab code changes, sends them to configured AI, and returns the AI-generated commit message. Users can set API keys or Cookies for different providers and configure options like locale, generate number of messages, commit type, proxy, timeout, max-length, and more. AICommit2 can be used both locally with Ollama and remotely with supported providers, offering flexibility and efficiency in generating commit messages.
For similar tasks
worker-vllm
The worker-vLLM repository provides a serverless endpoint for deploying OpenAI-compatible vLLM models with blazing-fast performance. It supports deploying various model architectures, such as Aquila, Baichuan, BLOOM, ChatGLM, Command-R, DBRX, DeciLM, Falcon, Gemma, GPT-2, GPT BigCode, GPT-J, GPT-NeoX, InternLM, Jais, LLaMA, MiniCPM, Mistral, Mixtral, MPT, OLMo, OPT, Orion, Phi, Phi-3, Qwen, Qwen2, Qwen2MoE, StableLM, Starcoder2, Xverse, and Yi. Users can deploy models using pre-built Docker images or build custom images with specified arguments. The repository also supports OpenAI compatibility for chat completions, completions, and models, with customizable input parameters. Users can modify their OpenAI codebase to use the deployed vLLM worker and access a list of available models for deployment.
ai-on-gke
This repository contains assets related to AI/ML workloads on Google Kubernetes Engine (GKE). Run optimized AI/ML workloads with Google Kubernetes Engine (GKE) platform orchestration capabilities. A robust AI/ML platform considers the following layers: Infrastructure orchestration that support GPUs and TPUs for training and serving workloads at scale Flexible integration with distributed computing and data processing frameworks Support for multiple teams on the same infrastructure to maximize utilization of resources
ray
Ray is a unified framework for scaling AI and Python applications. It consists of a core distributed runtime and a set of AI libraries for simplifying ML compute, including Data, Train, Tune, RLlib, and Serve. Ray runs on any machine, cluster, cloud provider, and Kubernetes, and features a growing ecosystem of community integrations. With Ray, you can seamlessly scale the same code from a laptop to a cluster, making it easy to meet the compute-intensive demands of modern ML workloads.
labelbox-python
Labelbox is a data-centric AI platform for enterprises to develop, optimize, and use AI to solve problems and power new products and services. Enterprises use Labelbox to curate data, generate high-quality human feedback data for computer vision and LLMs, evaluate model performance, and automate tasks by combining AI and human-centric workflows. The academic & research community uses Labelbox for cutting-edge AI research.
djl
Deep Java Library (DJL) is an open-source, high-level, engine-agnostic Java framework for deep learning. It is designed to be easy to get started with and simple to use for Java developers. DJL provides a native Java development experience and allows users to integrate machine learning and deep learning models with their Java applications. The framework is deep learning engine agnostic, enabling users to switch engines at any point for optimal performance. DJL's ergonomic API interface guides users with best practices to accomplish deep learning tasks, such as running inference and training neural networks.
mlflow
MLflow is a platform to streamline machine learning development, including tracking experiments, packaging code into reproducible runs, and sharing and deploying models. MLflow offers a set of lightweight APIs that can be used with any existing machine learning application or library (TensorFlow, PyTorch, XGBoost, etc), wherever you currently run ML code (e.g. in notebooks, standalone applications or the cloud). MLflow's current components are:
* `MLflow Tracking
tt-metal
TT-NN is a python & C++ Neural Network OP library. It provides a low-level programming model, TT-Metalium, enabling kernel development for Tenstorrent hardware.
burn
Burn is a new comprehensive dynamic Deep Learning Framework built using Rust with extreme flexibility, compute efficiency and portability as its primary goals.
For similar jobs
weave
Weave is a toolkit for developing Generative AI applications, built by Weights & Biases. With Weave, you can log and debug language model inputs, outputs, and traces; build rigorous, apples-to-apples evaluations for language model use cases; and organize all the information generated across the LLM workflow, from experimentation to evaluations to production. Weave aims to bring rigor, best-practices, and composability to the inherently experimental process of developing Generative AI software, without introducing cognitive overhead.
LLMStack
LLMStack is a no-code platform for building generative AI agents, workflows, and chatbots. It allows users to connect their own data, internal tools, and GPT-powered models without any coding experience. LLMStack can be deployed to the cloud or on-premise and can be accessed via HTTP API or triggered from Slack or Discord.
VisionCraft
The VisionCraft API is a free API for using over 100 different AI models. From images to sound.
kaito
Kaito is an operator that automates the AI/ML inference model deployment in a Kubernetes cluster. It manages large model files using container images, avoids tuning deployment parameters to fit GPU hardware by providing preset configurations, auto-provisions GPU nodes based on model requirements, and hosts large model images in the public Microsoft Container Registry (MCR) if the license allows. Using Kaito, the workflow of onboarding large AI inference models in Kubernetes is largely simplified.
PyRIT
PyRIT is an open access automation framework designed to empower security professionals and ML engineers to red team foundation models and their applications. It automates AI Red Teaming tasks to allow operators to focus on more complicated and time-consuming tasks and can also identify security harms such as misuse (e.g., malware generation, jailbreaking), and privacy harms (e.g., identity theft). The goal is to allow researchers to have a baseline of how well their model and entire inference pipeline is doing against different harm categories and to be able to compare that baseline to future iterations of their model. This allows them to have empirical data on how well their model is doing today, and detect any degradation of performance based on future improvements.
tabby
Tabby is a self-hosted AI coding assistant, offering an open-source and on-premises alternative to GitHub Copilot. It boasts several key features: * Self-contained, with no need for a DBMS or cloud service. * OpenAPI interface, easy to integrate with existing infrastructure (e.g Cloud IDE). * Supports consumer-grade GPUs.
spear
SPEAR (Simulator for Photorealistic Embodied AI Research) is a powerful tool for training embodied agents. It features 300 unique virtual indoor environments with 2,566 unique rooms and 17,234 unique objects that can be manipulated individually. Each environment is designed by a professional artist and features detailed geometry, photorealistic materials, and a unique floor plan and object layout. SPEAR is implemented as Unreal Engine assets and provides an OpenAI Gym interface for interacting with the environments via Python.
Magick
Magick is a groundbreaking visual AIDE (Artificial Intelligence Development Environment) for no-code data pipelines and multimodal agents. Magick can connect to other services and comes with nodes and templates well-suited for intelligent agents, chatbots, complex reasoning systems and realistic characters.