com.openai.unity
A Non-Official OpenAI Rest Client for Unity (UPM)
Stars: 440
com.openai.unity is an OpenAI package for Unity that allows users to interact with OpenAI's API through RESTful requests. It is independently developed and not an official library affiliated with OpenAI. Users can fine-tune models, create assistants, chat completions, and more. The package requires Unity 2021.3 LTS or higher and can be installed via Unity Package Manager or Git URL. Various features like authentication, Azure OpenAI integration, model management, thread creation, chat completions, audio processing, image generation, file management, fine-tuning, batch processing, embeddings, and content moderation are available.
README:
Based on OpenAI-DotNet
A OpenAI package for the Unity to use though their RESTful API. Independently developed, this is not an official library and I am not affiliated with OpenAI. An OpenAI API account is required.
All copyrights, trademarks, logos, and assets are the property of their respective owners.
Requires Unity 2021.3 LTS or higher.
The recommended installation method is though the unity package manager and OpenUPM.
- Open your Unity project settings
- Add the OpenUPM package registry:
- Name:
OpenUPM
- URL:
https://package.openupm.com
- Scope(s):
com.openai
com.utilities
- Name:
- Open the Unity Package Manager window
- Change the Registry from Unity to
My Registries
- Add the
OpenAI
package
- Open your Unity Package Manager
- Add package from git url:
https://github.com/RageAgainstThePixel/com.openai.unity.git#upm
Note: this repo has dependencies on other repositories! You are responsible for adding these on your own.
Check out our new api docs!
https://rageagainstthepixel.github.io/OpenAI-DotNet 🆕
- Authentication
- Azure OpenAI
- OpenAI API Proxy
- Models
-
Assistants
⚠️ 🚧- List Assistants
- Create Assistant
- Retrieve Assistant
- Modify Assistant
- Delete Assistant
-
Assistant Streaming
⚠️ 🚧 -
Threads
⚠️ 🚧 - Vector Stores
- Chat
- Audio
-
Images
⚠️ 🚧 - Files
- Fine Tuning
- Batches
- Embeddings
- Moderations
There are 4 ways to provide your API keys, in order of precedence:
[!WARNING] We recommended using the environment variables to load the API key instead of having it hard coded in your source. It is not recommended use this method in production, but only for accepting user credentials, local testing and quick start scenarios.
-
Pass keys directly with constructor
⚠️ -
Unity Scriptable Object
⚠️ - Load key from configuration file
- Use System Environment Variables
You use the OpenAIAuthentication
when you initialize the API as shown:
[!WARNING] We recommended using the environment variables to load the API key instead of having it hard coded in your source. It is not recommended use this method in production, but only for accepting user credentials, local testing and quick start scenarios.
var api = new OpenAIClient("sk-apiKey");
Or create a OpenAIAuthentication
object manually
var api = new OpenAIClient(new OpenAIAuthentication("sk-apiKey", "org-yourOrganizationId", "proj_yourProjectId"));
You can save the key directly into a scriptable object that is located in the Assets/Resources
folder.
You can create a new one by using the context menu of the project pane and creating a new OpenAIConfiguration
scriptable object.
[!WARNING] Beware checking this file into source control, as other people will be able to see your API key. It is recommended to use the OpenAI-DotNet-Proxy and authenticate users with your preferred OAuth provider.
Attempts to load api keys from a configuration file, by default .openai
in the current directory, optionally traversing up the directory tree or in the user's home directory.
To create a configuration file, create a new text file named .openai
and containing the line:
[!NOTE] Organization and project id entries are optional.
{
"apiKey": "sk-aaaabbbbbccccddddd",
"organizationId": "org-yourOrganizationId",
"projectId": "proj_yourProjectId"
}
OPENAI_API_KEY=sk-aaaabbbbbccccddddd
OPENAI_ORGANIZATION_ID=org-yourOrganizationId
OPENAI_PROJECT_ID=proj_yourProjectId
You can also load the configuration file directly with known path by calling static methods in OpenAIAuthentication
:
- Loads the default
.openai
config in the specified directory:
var api = new OpenAIClient(new OpenAIAuthentication().LoadFromDirectory("path/to/your/directory"));
- Loads the configuration file from a specific path. File does not need to be named
.openai
as long as it conforms to the json format:
var api = new OpenAIClient(new OpenAIAuthentication().LoadFromPath("path/to/your/file.json"));
Use your system's environment variables specify an api key and organization to use.
- Use
OPENAI_API_KEY
for your api key. - Use
OPENAI_ORGANIZATION_ID
to specify an organization. - Use
OPENAI_PROJECT_ID
to specify a project.
var api = new OpenAIClient(new OpenAIAuthentication().LoadFromEnvironment());
You can also choose to use Microsoft's Azure OpenAI deployments as well.
You can find the required information in the Azure Playground by clicking the View Code
button and view a URL like this:
https://{your-resource-name}.openai.azure.com/openai/deployments/{deployment-id}/chat/completions?api-version={api-version}
-
your-resource-name
The name of your Azure OpenAI Resource. -
deployment-id
The deployment name you chose when you deployed the model. -
api-version
The API version to use for this operation. This follows the YYYY-MM-DD format.
To setup the client to use your deployment, you'll need to pass in OpenAISettings
into the client constructor.
var auth = new OpenAIAuthentication("sk-apiKey");
var settings = new OpenAISettings(resourceName: "your-resource-name", deploymentId: "deployment-id", apiVersion: "api-version");
var api = new OpenAIClient(auth, settings);
Authenticate with MSAL as usual and get access token, then use the access token when creating your OpenAIAuthentication
. Then be sure to set useAzureActiveDirectory to true when creating your OpenAISettings
.
Tutorial: Desktop app that calls web APIs: Acquire a token
// get your access token using any of the MSAL methods
var accessToken = result.AccessToken;
var auth = new OpenAIAuthentication(accessToken);
var settings = new OpenAISettings(resourceName: "your-resource", deploymentId: "deployment-id", apiVersion: "api-version", useActiveDirectoryAuthentication: true);
var api = new OpenAIClient(auth, settings);
Using either the OpenAI-DotNet or com.openai.unity packages directly in your front-end app may expose your API keys and other sensitive information. To mitigate this risk, it is recommended to set up an intermediate API that makes requests to OpenAI on behalf of your front-end app. This library can be utilized for both front-end and intermediary host configurations, ensuring secure communication with the OpenAI API.
In the front end example, you will need to securely authenticate your users using your preferred OAuth provider. Once the user is authenticated, exchange your custom auth token with your API key on the backend.
Follow these steps:
- Setup a new project using either the OpenAI-DotNet or com.openai.unity packages.
- Authenticate users with your OAuth provider.
- After successful authentication, create a new
OpenAIAuthentication
object and pass in the custom token with the prefixsess-
. - Create a new
OpenAISettings
object and specify the domain where your intermediate API is located. - Pass your new
auth
andsettings
objects to theOpenAIClient
constructor when you create the client instance.
Here's an example of how to set up the front end:
var authToken = await LoginAsync();
var auth = new OpenAIAuthentication($"sess-{authToken}");
var settings = new OpenAISettings(domain: "api.your-custom-domain.com");
var api = new OpenAIClient(auth, settings);
This setup allows your front end application to securely communicate with your backend that will be using the OpenAI-DotNet-Proxy, which then forwards requests to the OpenAI API. This ensures that your OpenAI API keys and other sensitive information remain secure throughout the process.
In this example, we demonstrate how to set up and use OpenAIProxy
in a new ASP.NET Core web app. The proxy server will handle authentication and forward requests to the OpenAI API, ensuring that your API keys and other sensitive information remain secure.
- Create a new ASP.NET Core minimal web API project.
- Add the OpenAI-DotNet nuget package to your project.
- Powershell install:
Install-Package OpenAI-DotNet-Proxy
- Manually editing .csproj:
<PackageReference Include="OpenAI-DotNet-Proxy" />
- Powershell install:
- Create a new class that inherits from
AbstractAuthenticationFilter
and override theValidateAuthentication
method. This will implement theIAuthenticationFilter
that you will use to check user session token against your internal server. - In
Program.cs
, create a new proxy web application by callingOpenAIProxy.CreateWebApplication
method, passing your customAuthenticationFilter
as a type argument. - Create
OpenAIAuthentication
andOpenAIClientSettings
as you would normally with your API keys, org id, or Azure settings.
public partial class Program
{
private class AuthenticationFilter : AbstractAuthenticationFilter
{
public override void ValidateAuthentication(IHeaderDictionary request)
{
// You will need to implement your own class to properly test
// custom issued tokens you've setup for your end users.
if (!request.Authorization.ToString().Contains(TestUserToken))
{
throw new AuthenticationException("User is not authorized");
}
}
public override async Task ValidateAuthenticationAsync(IHeaderDictionary request)
{
await Task.CompletedTask; // remote resource call
// You will need to implement your own class to properly test
// custom issued tokens you've setup for your end users.
if (!request.Authorization.ToString().Contains(TestUserToken))
{
throw new AuthenticationException("User is not authorized");
}
}
}
public static void Main(string[] args)
{
var auth = OpenAIAuthentication.LoadFromEnv();
var settings = new OpenAIClientSettings(/* your custom settings if using Azure OpenAI */);
using var openAIClient = new OpenAIClient(auth, settings);
OpenAIProxy.CreateWebApplication<AuthenticationFilter>(args, openAIClient).Run();
}
}
Once you have set up your proxy server, your end users can now make authenticated requests to your proxy api instead of directly to the OpenAI API. The proxy server will handle authentication and forward requests to the OpenAI API, ensuring that your API keys and other sensitive information remain secure.
List and describe the various models available in the API. You can refer to the Models documentation to understand what models are available and the differences between them.
Also checkout model endpoint compatibility to understand which models work with which endpoints.
To specify a custom model not pre-defined in this library:
var model = new Model("model-id");
The Models API is accessed via OpenAIClient.ModelsEndpoint
Lists the currently available models, and provides basic information about each one such as the owner and availability.
var api = new OpenAIClient();
var models = await api.ModelsEndpoint.GetModelsAsync();
foreach (var model in models)
{
Debug.Log(model.ToString());
}
Retrieves a model instance, providing basic information about the model such as the owner and permissions.
var api = new OpenAIClient();
var model = await api.ModelsEndpoint.GetModelDetailsAsync("gpt-4o");
Debug.Log(model.ToString());
Delete a fine-tuned model. You must have the Owner role in your organization.
var api = new OpenAIClient();
var isDeleted = await api.ModelsEndpoint.DeleteFineTuneModelAsync("your-fine-tuned-model");
Assert.IsTrue(isDeleted);
[!WARNING] Beta Feature. API subject to breaking changes.
Build assistants that can call models and use tools to perform tasks.
The Assistants API is accessed via OpenAIClient.AssistantsEndpoint
Returns a list of assistants.
var api = new OpenAIClient();
var assistantsList = await api.AssistantsEndpoint.ListAssistantsAsync();
foreach (var assistant in assistantsList.Items)
{
Debug.Log($"{assistant} -> {assistant.CreatedAt}");
}
Create an assistant with a model and instructions.
var api = new OpenAIClient();
var request = new CreateAssistantRequest(Model.GPT4o);
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(request);
Retrieves an assistant.
var api = new OpenAIClient();
var assistant = await api.AssistantsEndpoint.RetrieveAssistantAsync("assistant-id");
Debug.Log($"{assistant} -> {assistant.CreatedAt}");
Modifies an assistant.
var api = new OpenAIClient();
var createRequest = new CreateAssistantRequest(Model.GPT4_Turbo);
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(createRequest);
var modifyRequest = new CreateAssistantRequest(Model.GPT4o);
var modifiedAssistant = await api.AssistantsEndpoint.ModifyAssistantAsync(assistant.Id, modifyRequest);
// OR AssistantExtension for easier use!
var modifiedAssistantEx = await assistant.ModifyAsync(modifyRequest);
Delete an assistant.
var api = new OpenAIClient();
var isDeleted = await api.AssistantsEndpoint.DeleteAssistantAsync("assistant-id");
// OR AssistantExtension for easier use!
var isDeleted = await assistant.DeleteAsync();
Assert.IsTrue(isDeleted);
[!NOTE] Assistant stream events can be easily added to existing thread calls by passing
Func<IServerSentEvent, Task> streamEventHandler
callback to any existing method that supports streaming.
Create Threads that Assistants can interact with.
The Threads API is accessed via OpenAIClient.ThreadsEndpoint
Create a thread.
var api = new OpenAIClient();
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
Debug.Log($"Create thread {thread.Id} -> {thread.CreatedAt}");
Create a thread and run it in one request.
See also: Thread Runs
var api = new OpenAIClient();
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(
new CreateAssistantRequest(
name: "Math Tutor",
instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less.",
model: Model.GPT4o));
var messages = new List<Message> { "I need to solve the equation `3x + 11 = 14`. Can you help me?" };
var threadRequest = new CreateThreadRequest(messages);
var run = await assistant.CreateThreadAndRunAsync(threadRequest);
Debug.Log($"Created thread and run: {run.ThreadId} -> {run.Id} -> {run.CreatedAt}");
Create a thread and run it in one request while streaming events.
var api = new OpenAIClient();
var tools = new List<Tool>
{
Tool.GetOrCreateTool(typeof(WeatherService), nameof(WeatherService.GetCurrentWeatherAsync))
};
var assistantRequest = new CreateAssistantRequest(tools: tools, instructions: "You are a helpful weather assistant. Use the appropriate unit based on geographical location.");
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(assistantRequest);
ThreadResponse thread = null;
async Task StreamEventHandler(IServerSentEvent streamEvent)
{
switch (streamEvent)
{
case ThreadResponse threadResponse:
thread = threadResponse;
break;
case RunResponse runResponse:
if (runResponse.Status == RunStatus.RequiresAction)
{
var toolOutputs = await assistant.GetToolOutputsAsync(runResponse);
foreach (var toolOutput in toolOutputs)
{
Debug.Log($"Tool Output: {toolOutput}");
}
await runResponse.SubmitToolOutputsAsync(toolOutputs, StreamEventHandler);
}
break;
default:
Debug.Log(streamEvent.ToJsonString());
break;
}
}
var run = await assistant.CreateThreadAndRunAsync("I'm in Kuala-Lumpur, please tell me what's the temperature now?", StreamEventHandler);
run = await run.WaitForStatusChangeAsync();
var messages = await thread.ListMessagesAsync();
foreach (var response in messages.Items.Reverse())
{
Debug.Log($"{response.Role}: {response.PrintContent()}");
}
Retrieves a thread.
var api = new OpenAIClient();
var thread = await api.ThreadsEndpoint.RetrieveThreadAsync("thread-id");
// OR if you simply wish to get the latest state of a thread
thread = await thread.UpdateAsync();
Debug.Log($"Retrieve thread {thread.Id} -> {thread.CreatedAt}");
Modifies a thread.
Note: Only the metadata can be modified.
var api = new OpenAIClient();
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
var metadata = new Dictionary<string, string>
{
{ "key", "custom thread metadata" }
}
thread = await api.ThreadsEndpoint.ModifyThreadAsync(thread.Id, metadata);
// OR use extension method for convenience!
thread = await thread.ModifyAsync(metadata);
Debug.Log($"Modify thread {thread.Id} -> {thread.Metadata["key"]}");
Delete a thread.
var api = new OpenAIClient();
var isDeleted = await api.ThreadsEndpoint.DeleteThreadAsync("thread-id");
// OR use extension method for convenience!
var isDeleted = await thread.DeleteAsync();
Assert.IsTrue(isDeleted);
Create messages within threads.
Returns a list of messages for a given thread.
var api = new OpenAIClient();
var messageList = await api.ThreadsEndpoint.ListMessagesAsync("thread-id");
// OR use extension method for convenience!
var messageList = await thread.ListMessagesAsync();
foreach (var message in messageList.Items)
{
Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}");
}
Create a message.
var api = new OpenAIClient();
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
var request = new CreateMessageRequest("Hello world!");
var message = await api.ThreadsEndpoint.CreateMessageAsync(thread.Id, request);
// OR use extension method for convenience!
var message = await thread.CreateMessageAsync("Hello World!");
Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}");
Retrieve a message.
var api = new OpenAIClient();
var message = await api.ThreadsEndpoint.RetrieveMessageAsync("thread-id", "message-id");
// OR use extension methods for convenience!
var message = await thread.RetrieveMessageAsync("message-id");
var message = await message.UpdateAsync();
Debug.Log($"{message.Id}: {message.Role}: {message.PrintContent()}");
Modify a message.
Note: Only the message metadata can be modified.
var api = new OpenAIClient();
var metadata = new Dictionary<string, string>
{
{ "key", "custom message metadata" }
};
var message = await api.ThreadsEndpoint.ModifyMessageAsync("thread-id", "message-id", metadata);
// OR use extension method for convenience!
var message = await message.ModifyAsync(metadata);
Debug.Log($"Modify message metadata: {message.Id} -> {message.Metadata["key"]}");
Represents an execution run on a thread.
Returns a list of runs belonging to a thread.
var api = new OpenAIClient();
var runList = await api.ThreadsEndpoint.ListRunsAsync("thread-id");
// OR use extension method for convenience!
var runList = await thread.ListRunsAsync();
foreach (var run in runList.Items)
{
Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}");
}
Create a run.
var api = new OpenAIClient();
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(
new CreateAssistantRequest(
name: "Math Tutor",
instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less.",
model: Model.GPT4o));
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
var message = await thread.CreateMessageAsync("I need to solve the equation `3x + 11 = 14`. Can you help me?");
var run = await thread.CreateRunAsync(assistant);
Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}");
Create a run and stream the events.
var api = new OpenAIClient();
var assistant = await api.AssistantsEndpoint.CreateAssistantAsync(
new CreateAssistantRequest(
name: "Math Tutor",
instructions: "You are a personal math tutor. Answer questions briefly, in a sentence or less. Your responses should be formatted in JSON.",
model: Model.GPT4o,
responseFormat: ChatResponseFormat.Json));
var thread = await api.ThreadsEndpoint.CreateThreadAsync();
var message = await thread.CreateMessageAsync("I need to solve the equation `3x + 11 = 14`. Can you help me?");
var run = await thread.CreateRunAsync(assistant, async streamEvent =>
{
Debug.Log(streamEvent.ToJsonString());
await Task.CompletedTask;
});
var messages = await thread.ListMessagesAsync();
foreach (var response in messages.Items.Reverse())
{
Debug.Log($"{response.Role}: {response.PrintContent()}");
}
Retrieves a run.
var api = new OpenAIClient();
var run = await api.ThreadsEndpoint.RetrieveRunAsync("thread-id", "run-id");
// OR use extension method for convenience!
var run = await thread.RetrieveRunAsync("run-id");
var run = await run.UpdateAsync();
Debug.Log($"[{run.Id}] {run.Status} | {run.CreatedAt}");
Modifies a run.
Note: Only the metadata can be modified.
var api = new OpenAIClient();
var metadata = new Dictionary<string, string>
{
{ "key", "custom run metadata" }
};
var run = await api.ThreadsEndpoint.ModifyRunAsync("thread-id", "run-id", metadata);
// OR use extension method for convenience!
var run = await run.ModifyAsync(metadata);
Debug.Log($"Modify run {run.Id} -> {run.Metadata["key"]}");
When a run has the status: requires_action
and required_action.type
is submit_tool_outputs
, this endpoint can be used to submit the outputs from the tool calls once they're all completed.
All outputs must be submitted in a single request.
[!NOTE] See Create Thread and Run Streaming example on how to stream tool output events.
var api = new OpenAIClient();
var tools = new List<Tool>
{
// Use a predefined tool
Tool.Retrieval, Tool.CodeInterpreter,
// Or create a tool from a type and the name of the method you want to use for function calling
Tool.GetOrCreateTool(typeof(WeatherService), nameof(WeatherService.GetCurrentWeatherAsync)),
// Pass in an instance of an object to call a method on it
Tool.GetOrCreateTool(api.ImagesEndPoint, nameof(ImagesEndpoint.GenerateImageAsync)),
// Define func<,> callbacks
Tool.FromFunc("name_of_func", () => { /* callback function */ }),
Tool.FromFunc<T1,T2,TResult>("func_with_multiple_params", (t1, t2) => { /* logic that calculates return value */ return tResult; })
};
var assistantRequest = new CreateAssistantRequest(tools: tools, instructions: "You are a helpful weather assistant. Use the appropriate unit based on geographical location.");
var testAssistant = await api.AssistantsEndpoint.CreateAssistantAsync(assistantRequest);
var run = await testAssistant.CreateThreadAndRunAsync("I'm in Kuala-Lumpur, please tell me what's the temperature now?");
// waiting while run is Queued and InProgress
run = await run.WaitForStatusChangeAsync();
// Invoke all of the tool call functions and return the tool outputs.
var toolOutputs = await testAssistant.GetToolOutputsAsync(run.RequiredAction.SubmitToolOutputs.ToolCalls);
foreach (var toolOutput in toolOutputs)
{
Debug.Log($"tool call output: {toolOutput.Output}");
}
// submit the tool outputs
run = await run.SubmitToolOutputsAsync(toolOutputs);
// waiting while run in Queued and InProgress
run = await run.WaitForStatusChangeAsync();
var messages = await run.ListMessagesAsync();
foreach (var message in messages.Items.OrderBy(response => response.CreatedAt))
{
Debug.Log($"{message.Role}: {message.PrintContent()}");
}
Returns a list of run steps belonging to a run.
var api = new OpenAIClient();
var runStepList = await api.ThreadsEndpoint.ListRunStepsAsync("thread-id", "run-id");
// OR use extension method for convenience!
var runStepList = await run.ListRunStepsAsync();
foreach (var runStep in runStepList.Items)
{
Debug.Log($"[{runStep.Id}] {runStep.Status} {runStep.CreatedAt} -> {runStep.ExpiresAt}");
}
Retrieves a run step.
var api = new OpenAIClient();
var runStep = await api.ThreadsEndpoint.RetrieveRunStepAsync("thread-id", "run-id", "step-id");
// OR use extension method for convenience!
var runStep = await run.RetrieveRunStepAsync("step-id");
var runStep = await runStep.UpdateAsync();
Debug.Log($"[{runStep.Id}] {runStep.Status} {runStep.CreatedAt} -> {runStep.ExpiresAt}");
Cancels a run that is in_progress
.
var api = new OpenAIClient();
var isCancelled = await api.ThreadsEndpoint.CancelRunAsync("thread-id", "run-id");
// OR use extension method for convenience!
var isCancelled = await run.CancelAsync();
Assert.IsTrue(isCancelled);
Vector stores are used to store files for use by the file_search
tool.
The Vector Stores API is accessed via OpenAIClient.VectorStoresEndpoint
Returns a list of vector stores.
var api = new OpenAIClient();
var vectorStores = await OpenAIClient.VectorStoresEndpoint.ListVectorStoresAsync();
foreach (var vectorStore in vectorStores.Items)
{
Debug.Log(vectorStore);
}
Create a vector store.
var api = new OpenAIClient();
var createVectorStoreRequest = new CreateVectorStoreRequest("test-vector-store");
var vectorStore = await api.VectorStoresEndpoint.CreateVectorStoreAsync(createVectorStoreRequest);
Debug.Log(vectorStore);
Retrieves a vector store.
var api = new OpenAIClient();
var vectorStore = await api.VectorStoresEndpoint.GetVectorStoreAsync("vector-store-id");
Debug.Log(vectorStore);
Modifies a vector store.
var api = new OpenAIClient();
var metadata = new Dictionary<string, object> { { "Test", DateTime.UtcNow } };
var vectorStore = await api.VectorStoresEndpoint.ModifyVectorStoreAsync("vector-store-id", metadata: metadata);
Debug.Log(vectorStore);
Delete a vector store.
var api = new OpenAIClient();
var isDeleted = await api.VectorStoresEndpoint.DeleteVectorStoreAsync("vector-store-id");
Assert.IsTrue(isDeleted);
Vector store files represent files inside a vector store.
Returns a list of vector store files.
var api = new OpenAIClient();
var files = await api.VectorStoresEndpoint.ListVectorStoreFilesAsync("vector-store-id");
foreach (var file in vectorStoreFiles.Items)
{
Debug.Log(file);
}
Create a vector store file by attaching a file to a vector store.
var api = new OpenAIClient();
var file = await api.VectorStoresEndpoint.CreateVectorStoreFileAsync("vector-store-id", "file-id", new ChunkingStrategy(ChunkingStrategyType.Static));
Debug.Log(file);
Retrieves a vector store file.
var api = new OpenAIClient();
var file = await api.VectorStoresEndpoint.GetVectorStoreFileAsync("vector-store-id", "vector-store-file-id");
Debug.Log(file);
Delete a vector store file. This will remove the file from the vector store but the file itself will not be deleted. To delete the file, use the delete file endpoint.
var api = new OpenAIClient();
var isDeleted = await api.VectorStoresEndpoint.DeleteVectorStoreFileAsync("vector-store-id", vectorStoreFile);
Assert.IsTrue(isDeleted);
Vector store files represent files inside a vector store.
Create a vector store file batch.
var api = new OpenAIClient();
var files = new List<string> { "file_id_1","file_id_2" };
var vectorStoreFileBatch = await api.VectorStoresEndpoint.CreateVectorStoreFileBatchAsync("vector-store-id", files);
Debug.Log(vectorStoreFileBatch);
Retrieves a vector store file batch.
var api = new OpenAIClient();
var vectorStoreFileBatch = await api.VectorStoresEndpoint.GetVectorStoreFileBatchAsync("vector-store-id", "vector-store-file-batch-id");
// you can also use convenience methods!
vectorStoreFileBatch = await vectorStoreFileBatch.UpdateAsync();
vectorStoreFileBatch = await vectorStoreFileBatch.WaitForStatusChangeAsync();
Returns a list of vector store files in a batch.
var api = new OpenAIClient();
var files = await api.VectorStoresEndpoint.ListVectorStoreBatchFilesAsync("vector-store-id", "vector-store-file-batch-id");
foreach (var file in files.Items)
{
Debug.Log(file);
}
Cancel a vector store file batch. This attempts to cancel the processing of files in this batch as soon as possible.
var api = new OpenAIClient();
var isCancelled = await api.VectorStoresEndpoint.CancelVectorStoreFileBatchAsync("vector-store-id", "vector-store-file-batch-id");
Given a chat conversation, the model will return a chat completion response.
The Chat API is accessed via OpenAIClient.ChatEndpoint
Creates a completion for the chat message
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant."),
new Message(Role.User, "Who won the world series in 2020?"),
new Message(Role.Assistant, "The Los Angeles Dodgers won the World Series in 2020."),
new Message(Role.User, "Where was it played?"),
};
var chatRequest = new ChatRequest(messages, Model.GPT4o);
var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
var choice = response.FirstChoice;
Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice.Message} | Finish Reason: {choice.FinishReason}");
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant."),
new Message(Role.User, "Who won the world series in 2020?"),
new Message(Role.Assistant, "The Los Angeles Dodgers won the World Series in 2020."),
new Message(Role.User, "Where was it played?"),
};
var chatRequest = new ChatRequest(messages);
var response = await api.ChatEndpoint.StreamCompletionAsync(chatRequest, async partialResponse =>
{
Debug.Log(partialResponse.FirstChoice.Delta.ToString());
await Task.CompletedTask;
});
var choice = response.FirstChoice;
Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice.Message} | Finish Reason: {choice.FinishReason}");
var api = new OpenAIClient();
var messages = new List<Message>
{
new(Role.System, "You are a helpful weather assistant. Always prompt the user for their location."),
new Message(Role.User, "What's the weather like today?"),
};
foreach (var message in messages)
{
Debug.Log($"{message.Role}: {message}");
}
// Define the tools that the assistant is able to use:
// 1. Get a list of all the static methods decorated with FunctionAttribute
var tools = Tool.GetAllAvailableTools(includeDefaults: false, forceUpdate: true, clearCache: true);
// 2. Define a custom list of tools:
var tools = new List<Tool>
{
Tool.GetOrCreateTool(objectInstance, "TheNameOfTheMethodToCall"),
Tool.FromFunc("a_custom_name_for_your_function", ()=> { /* Some logic to run */ })
};
var chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto");
var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
messages.Add(response.FirstChoice.Message);
Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice} | Finish Reason: {response.FirstChoice.FinishReason}");
var locationMessage = new Message(Role.User, "I'm in Glasgow, Scotland");
messages.Add(locationMessage);
Debug.Log($"{locationMessage.Role}: {locationMessage.Content}");
chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto");
response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
messages.Add(response.FirstChoice.Message);
if (response.FirstChoice.FinishReason == "stop")
{
Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice} | Finish Reason: {response.FirstChoice.FinishReason}");
var unitMessage = new Message(Role.User, "Fahrenheit");
messages.Add(unitMessage);
Debug.Log($"{unitMessage.Role}: {unitMessage.Content}");
chatRequest = new ChatRequest(messages, tools: tools, toolChoice: "auto");
response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
}
// iterate over all tool calls and invoke them
foreach (var toolCall in response.FirstChoice.Message.ToolCalls)
{
Debug.Log($"{response.FirstChoice.Message.Role}: {toolCall.Function.Name} | Finish Reason: {response.FirstChoice.FinishReason}");
Debug.Log($"{toolCall.Function.Arguments}");
// Invokes function to get a generic json result to return for tool call.
var functionResult = await toolCall.InvokeFunctionAsync();
// If you know the return type and do additional processing you can use generic overload
var functionResult = await toolCall.InvokeFunctionAsync<string>();
messages.Add(new Message(toolCall, functionResult));
Debug.Log($"{Role.Tool}: {functionResult}");
}
// System: You are a helpful weather assistant.
// User: What's the weather like today?
// Assistant: Sure, may I know your current location? | Finish Reason: stop
// User: I'm in Glasgow, Scotland
// Assistant: GetCurrentWeather | Finish Reason: tool_calls
// {
// "location": "Glasgow, Scotland",
// "unit": "celsius"
// }
// Tool: The current weather in Glasgow, Scotland is 39°C.
[!WARNING] Beta Feature. API subject to breaking changes.
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant."),
new Message(Role.User, new List<Content>
{
"What's in this image?",
new ImageUrl("https://upload.wikimedia.org/wikipedia/commons/thumb/d/dd/Gfp-wisconsin-madison-the-nature-boardwalk.jpg/2560px-Gfp-wisconsin-madison-the-nature-boardwalk.jpg", ImageDetail.Low)
})
};
var chatRequest = new ChatRequest(messages, model: Model.GPT4o);
var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
Debug.Log($"{response.FirstChoice.Message.Role}: {response.FirstChoice.Message.Content} | Finish Reason: {response.FirstChoice.FinishDetails}");
You can even pass in a Texture2D
!
var api = new OpenAIClient();
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant."),
new Message(Role.User, new List<Content>
{
"What's in this image?",
texture
})
};
var chatRequest = new ChatRequest(messages, model: Model.GPT4o);
var result = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
Debug.Log($"{result.FirstChoice.Message.Role}: {result.FirstChoice} | Finish Reason: {result.FirstChoice.FinishDetails}");
[!WARNING] Beta Feature. API subject to breaking changes.
[!IMPORTANT]
- When using JSON mode, always instruct the model to produce JSON via some message in the conversation, for example via your system message. If you don't include an explicit instruction to generate JSON, the model may generate an unending stream of whitespace and the request may run continually until it reaches the token limit. To help ensure you don't forget, the API will throw an error if the string "JSON" does not appear somewhere in the context.
- The JSON in the message the model returns may be partial (i.e. cut off) if
finish_reason
is length, which indicates the generation exceeded max_tokens or the conversation exceeded the token limit. To guard against this, checkfinish_reason
before parsing the response.- JSON mode will not guarantee the output matches any specific schema, only that it is valid and parses without errors.
var messages = new List<Message>
{
new Message(Role.System, "You are a helpful assistant designed to output JSON."),
new Message(Role.User, "Who won the world series in 2020?"),
};
var chatRequest = new ChatRequest(messages, Model.GPT4o, responseFormat: ChatResponseFormat.Json);
var response = await api.ChatEndpoint.GetCompletionAsync(chatRequest);
foreach (var choice in response.Choices)
{
Debug.Log($"[{choice.Index}] {choice.Message.Role}: {choice} | Finish Reason: {choice.FinishReason}");
}
response.GetUsage();
Converts audio into text.
The Audio API is accessed via OpenAIClient.AudioEndpoint
Generates audio from the input text.
var api = new OpenAIClient();
var request = new SpeechRequest("Hello world!");
var (path, clip) = await api.AudioEndpoint.CreateSpeechAsync(request);
audioSource.PlayOneShot(clip);
Debug.Log(path);
Generate streamed audio from the input text.
var api = new OpenAIClient();
var request = new SpeechRequest("Hello world!");
var (path, clip) = await OpenAIClient.AudioEndpoint.CreateSpeechStreamAsync(request, partialClip => audioSource.PlayOneShot(partialClip));
Debug.Log(path);
Transcribes audio into the input language.
var api = new OpenAIClient();
var request = new AudioTranscriptionRequest(audioClip, language: "en");
var result = await api.AudioEndpoint.CreateTranscriptionAsync(request);
Debug.Log(result);
You can also get detailed information using verbose_json
to get timestamp granularities:
var api = new OpenAIClient();
using var request = new AudioTranscriptionRequest(transcriptionAudio, responseFormat: AudioResponseFormat.Verbose_Json, timestampGranularity: TimestampGranularity.Word, temperature: 0.1f, language: "en");
var response = await api.AudioEndpoint.CreateTranscriptionTextAsync(request);
foreach (var word in response.Words)
{
Debug.Log($"[{word.Start}-{word.End}] \"{word.Word}\"");
}
Translates audio into into English.
var api = new OpenAIClient();
var request = new AudioTranslationRequest(audioClip);
var result = await api.AudioEndpoint.CreateTranslationAsync(request);
Debug.Log(result);
Given a prompt and/or an input image, the model will generate a new image.
The Images API is accessed via OpenAIClient.ImagesEndpoint
Creates an image given a prompt.
var api = new OpenAIClient();
var request = new ImageGenerationRequest("A house riding a velociraptor", Models.Model.DallE_3);
var imageResults = await api.ImagesEndPoint.GenerateImageAsync(request);
foreach (var result in imageResults)
{
Debug.Log(result.ToString());
Assert.IsNotNull(result.Texture);
}
Creates an edited or extended image given an original image and a prompt.
var api = new OpenAIClient();
var request = new ImageEditRequest(Path.GetFullPath(imageAssetPath), Path.GetFullPath(maskAssetPath), "A sunlit indoor lounge area with a pool containing a flamingo", size: ImageSize.Small);
var imageResults = await api.ImagesEndPoint.CreateImageEditAsync(request);
foreach (var result in imageResults)
{
Debug.Log(result.ToString());
Assert.IsNotNull(result.Texture);
}
Creates a variation of a given image.
var api = new OpenAIClient();
var request = new ImageVariationRequest(imageTexture, size: ImageSize.Small);
var imageResults = await api.ImagesEndPoint.CreateImageVariationAsync(request);
foreach (var result in imageResults)
{
Debug.Log(result.ToString());
Assert.IsNotNull(result.Texture);
}
Alternatively, the endpoint can directly take a Texture2D with Read/Write enabled and Compression set to None.
var api = new OpenAIClient();
var request = new ImageVariationRequest(imageTexture, size: ImageSize.Small);
var imageResults = await api.ImagesEndPoint.CreateImageVariationAsync(request);
foreach (var result in imageResults)
{
Debug.Log(result.ToString());
Assert.IsNotNull(result.Texture);
}
Files are used to upload documents that can be used with features like Fine-tuning.
The Files API is accessed via OpenAIClient.FilesEndpoint
Returns a list of files that belong to the user's organization.
var api = new OpenAIClient();
var fileList = await api.FilesEndpoint.ListFilesAsync();
foreach (var file in fileList)
{
Debug.Log($"{file.Id} -> {file.Object}: {file.FileName} | {file.Size} bytes");
}
Upload a file that can be used across various endpoints. The size of all the files uploaded by one organization can be up to 100 GB.
The size of individual files can be a maximum of 512 MB. See the Assistants Tools guide to learn more about the types of files supported. The Fine-tuning API only supports .jsonl files.
var api = new OpenAIClient();
var file = await api.FilesEndpoint.UploadFileAsync("path/to/your/file.jsonl", FilePurpose.FineTune);
Debug.Log(file.Id);
Delete a file.
var api = new OpenAIClient();
var isDeleted = await api.FilesEndpoint.DeleteFileAsync(fileId);
Assert.IsTrue(isDeleted);
Returns information about a specific file.
var api = new OpenAIClient();
var file = await api.FilesEndpoint.GetFileInfoAsync(fileId);
Debug.Log($"{file.Id} -> {file.Object}: {file.FileName} | {file.Size} bytes");
Downloads the file content to the specified directory.
var api = new OpenAIClient();
var downloadedFilePath = await api.FilesEndpoint.DownloadFileAsync(fileId);
Debug.Log(downloadedFilePath);
Assert.IsTrue(File.Exists(downloadedFilePath));
Manage fine-tuning jobs to tailor a model to your specific training data.
Related guide: Fine-tune models
The Files API is accessed via OpenAIClient.FineTuningEndpoint
Creates a job that fine-tunes a specified model from a given dataset.
Response includes details of the enqueued job including job status and the name of the fine-tuned models once complete.
var api = new OpenAIClient();
var fileId = "file-abc123";
var request = new CreateFineTuneRequest(fileId);
var job = await api.FineTuningEndpoint.CreateJobAsync(Model.GPT3_5_Turbo, request);
Debug.Log($"Started {job.Id} | Status: {job.Status}");
List your organization's fine-tuning jobs.
var api = new OpenAIClient();
var jobList = await api.FineTuningEndpoint.ListJobsAsync();
foreach (var job in jobList.Items.OrderByDescending(job => job.CreatedAt)))
{
Debug.Log($"{job.Id} -> {job.CreatedAt} | {job.Status}");
}
Gets info about the fine-tune job.
var api = new OpenAIClient();
var job = await api.FineTuningEndpoint.GetJobInfoAsync(fineTuneJob);
Debug.Log($"{job.Id} -> {job.CreatedAt} | {job.Status}");
Immediately cancel a fine-tune job.
var api = new OpenAIClient();
var isCancelled = await api.FineTuningEndpoint.CancelFineTuneJobAsync(fineTuneJob);
Assert.IsTrue(isCancelled);
Get status updates for a fine-tuning job.
var api = new OpenAIClient();
var eventList = await api.FineTuningEndpoint.ListJobEventsAsync(fineTuneJob);
Debug.Log($"{fineTuneJob.Id} -> status: {fineTuneJob.Status} | event count: {eventList.Events.Count}");
foreach (var @event in eventList.Items.OrderByDescending(@event => @event.CreatedAt))
{
Debug.Log($" {@event.CreatedAt} [{@event.Level}] {@event.Message}");
}
Create large batches of API requests for asynchronous processing. The Batch API returns completions within 24 hours for a 50% discount.
The Batches API is accessed via OpenAIClient.BatchesEndpoint
List your organization's batches.
var api = new OpenAIClient();
var batches = await api.await OpenAIClient.BatchEndpoint.ListBatchesAsync();
foreach (var batch in listResponse.Items)
{
Debug.Log(batch);
}
Creates and executes a batch from an uploaded file of requests
var api = new OpenAIClient();
var batchRequest = new CreateBatchRequest("file-id", Endpoint.ChatCompletions);
var batch = await api.BatchEndpoint.CreateBatchAsync(batchRequest);
Retrieves a batch.
var api = new OpenAIClient();
var batch = await api.BatchEndpoint.RetrieveBatchAsync("batch-id");
// you can also use convenience methods!
batch = await batch.UpdateAsync();
batch = await batch.WaitForStatusChangeAsync();
Cancels an in-progress batch. The batch will be in status cancelling for up to 10 minutes, before changing to cancelled, where it will have partial results (if any) available in the output file.
var api = new OpenAIClient();
var isCancelled = await api.BatchEndpoint.CancelBatchAsync(batch);
Assert.IsTrue(isCancelled);
Get a vector representation of a given input that can be easily consumed by machine learning models and algorithms.
Related guide: Embeddings
The Edits API is accessed via OpenAIClient.EmbeddingsEndpoint
Creates an embedding vector representing the input text.
var api = new OpenAIClient();
var response = await api.EmbeddingsEndpoint.CreateEmbeddingAsync("The food was delicious and the waiter...", Models.Embedding_Ada_002);
Debug.Log(response);
Given a input text, outputs if the model classifies it as violating OpenAI's content policy.
Related guide: Moderations
The Moderations API can be accessed via OpenAIClient.ModerationsEndpoint
Classifies if text violates OpenAI's Content Policy.
var api = new OpenAIClient();
var isViolation = await api.ModerationsEndpoint.GetModerationAsync("I want to kill them.");
Assert.IsTrue(isViolation);
Additionally you can also get the scores of a given input.
var api = new OpenAIClient();
var response = await api.ModerationsEndpoint.CreateModerationAsync(new ModerationsRequest("I love you"));
Assert.IsNotNull(response);
Debug.Log(response.Results?[0]?.Scores?.ToString());
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for com.openai.unity
Similar Open Source Tools
com.openai.unity
com.openai.unity is an OpenAI package for Unity that allows users to interact with OpenAI's API through RESTful requests. It is independently developed and not an official library affiliated with OpenAI. Users can fine-tune models, create assistants, chat completions, and more. The package requires Unity 2021.3 LTS or higher and can be installed via Unity Package Manager or Git URL. Various features like authentication, Azure OpenAI integration, model management, thread creation, chat completions, audio processing, image generation, file management, fine-tuning, batch processing, embeddings, and content moderation are available.
OpenAI-DotNet
OpenAI-DotNet is a simple C# .NET client library for OpenAI to use through their RESTful API. It is independently developed and not an official library affiliated with OpenAI. Users need an OpenAI API account to utilize this library. The library targets .NET 6.0 and above, working across various platforms like console apps, winforms, wpf, asp.net, etc., and on Windows, Linux, and Mac. It provides functionalities for authentication, interacting with models, assistants, threads, chat, audio, images, files, fine-tuning, embeddings, and moderations.
ElevenLabs-DotNet
ElevenLabs-DotNet is a non-official Eleven Labs voice synthesis RESTful client that allows users to convert text to speech. The library targets .NET 8.0 and above, working across various platforms like console apps, winforms, wpf, and asp.net, and across Windows, Linux, and Mac. Users can authenticate using API keys directly, from a configuration file, or system environment variables. The tool provides functionalities for text to speech conversion, streaming text to speech, accessing voices, dubbing audio or video files, generating sound effects, managing history of synthesized audio clips, and accessing user information and subscription status.
dashscope-sdk
DashScope SDK for .NET is an unofficial SDK maintained by Cnblogs, providing various APIs for text embedding, generation, multimodal generation, image synthesis, and more. Users can interact with the SDK to perform tasks such as text completion, chat generation, function calls, file operations, and more. The project is under active development, and users are advised to check the Release Notes before upgrading.
suno-api
Suno AI API is an open-source project that allows developers to integrate the music generation capabilities of Suno.ai into their own applications. The API provides a simple and convenient way to generate music, lyrics, and other audio content using Suno.ai's powerful AI models. With Suno AI API, developers can easily add music generation functionality to their apps, websites, and other projects.
island-ai
island-ai is a TypeScript toolkit tailored for developers engaging with structured outputs from Large Language Models. It offers streamlined processes for handling, parsing, streaming, and leveraging AI-generated data across various applications. The toolkit includes packages like zod-stream for interfacing with LLM streams, stream-hooks for integrating streaming JSON data into React applications, and schema-stream for JSON streaming parsing based on Zod schemas. Additionally, related packages like @instructor-ai/instructor-js focus on data validation and retry mechanisms, enhancing the reliability of data processing workflows.
acte
Acte is a framework designed to build GUI-like tools for AI Agents. It aims to address the issues of cognitive load and freedom degrees when interacting with multiple APIs in complex scenarios. By providing a graphical user interface (GUI) for Agents, Acte helps reduce cognitive load and constraints interaction, similar to how humans interact with computers through GUIs. The tool offers APIs for starting new sessions, executing actions, and displaying screens, accessible via HTTP requests or the SessionManager class.
aiotdlib
aiotdlib is a Python asyncio Telegram client based on TDLib. It provides automatic generation of types and functions from tl schema, validation, good IDE type hinting, and high-level API methods for simpler work with tdlib. The package includes prebuilt TDLib binaries for macOS (arm64) and Debian Bullseye (amd64). Users can use their own binary by passing `library_path` argument to `Client` class constructor. Compatibility with other versions of the library is not guaranteed. The tool requires Python 3.9+ and users need to get their `api_id` and `api_hash` from Telegram docs for installation and usage.
ai
The Vercel AI SDK is a library for building AI-powered streaming text and chat UIs. It provides React, Svelte, Vue, and Solid helpers for streaming text responses and building chat and completion UIs. The SDK also includes a React Server Components API for streaming Generative UI and first-class support for various AI providers such as OpenAI, Anthropic, Mistral, Perplexity, AWS Bedrock, Azure, Google Gemini, Hugging Face, Fireworks, Cohere, LangChain, Replicate, Ollama, and more. Additionally, it offers Node.js, Serverless, and Edge Runtime support, as well as lifecycle callbacks for saving completed streaming responses to a database in the same request.
clarifai-python
The Clarifai Python SDK offers a comprehensive set of tools to integrate Clarifai's AI platform to leverage computer vision capabilities like classification , detection ,segementation and natural language capabilities like classification , summarisation , generation , Q&A ,etc into your applications. With just a few lines of code, you can leverage cutting-edge artificial intelligence to unlock valuable insights from visual and textual content.
aio-pika
Aio-pika is a wrapper around aiormq for asyncio and humans. It provides a completely asynchronous API, object-oriented API, transparent auto-reconnects with complete state recovery, Python 3.7+ compatibility, transparent publisher confirms support, transactions support, and complete type-hints coverage.
funcchain
Funcchain is a Python library that allows you to easily write cognitive systems by leveraging Pydantic models as output schemas and LangChain in the backend. It provides a seamless integration of LLMs into your apps, utilizing OpenAI Functions or LlamaCpp grammars (json-schema-mode) for efficient structured output. Funcchain compiles the Funcchain syntax into LangChain runnables, enabling you to invoke, stream, or batch process your pipelines effortlessly.
llama_ros
This repository provides a set of ROS 2 packages to integrate llama.cpp into ROS 2. By using the llama_ros packages, you can easily incorporate the powerful optimization capabilities of llama.cpp into your ROS 2 projects by running GGUF-based LLMs and VLMs.
venom
Venom is a high-performance system developed with JavaScript to create a bot for WhatsApp, support for creating any interaction, such as customer service, media sending, sentence recognition based on artificial intelligence and all types of design architecture for WhatsApp.
ivy
Ivy is an open-source machine learning framework that enables you to: * 🔄 **Convert code into any framework** : Use and build on top of any model, library, or device by converting any code from one framework to another using `ivy.transpile`. * ⚒️ **Write framework-agnostic code** : Write your code once in `ivy` and then choose the most appropriate ML framework as the backend to leverage all the benefits and tools. Join our growing community 🌍 to connect with people using Ivy. **Let's** unify.ai **together 🦾**
orch
orch is a library for building language model powered applications and agents for the Rust programming language. It can be used for tasks such as text generation, streaming text generation, structured data generation, and embedding generation. The library provides functionalities for executing various language model tasks and can be integrated into different applications and contexts. It offers flexibility for developers to create language model-powered features and applications in Rust.
For similar tasks
TaskingAI
TaskingAI brings Firebase's simplicity to **AI-native app development**. The platform enables the creation of GPTs-like multi-tenant applications using a wide range of LLMs from various providers. It features distinct, modular functions such as Inference, Retrieval, Assistant, and Tool, seamlessly integrated to enhance the development process. TaskingAI’s cohesive design ensures an efficient, intelligent, and user-friendly experience in AI application development.
experts
Experts.js is a tool that simplifies the creation and deployment of OpenAI's Assistants, allowing users to link them together as Tools to create a Panel of Experts system with expanded memory and attention to detail. It leverages the new Assistants API from OpenAI, which offers advanced features such as referencing attached files & images as knowledge sources, supporting instructions up to 256,000 characters, integrating with 128 tools, and utilizing the Vector Store API for efficient file search. Experts.js introduces Assistants as Tools, enabling the creation of Multi AI Agent Systems where each Tool is an LLM-backed Assistant that can take on specialized roles or fulfill complex tasks.
cherry-studio
Cherry Studio is a desktop client that supports multiple Large Language Model (LLM) providers, available on Windows, Mac, and Linux. It allows users to create multiple Assistants and topics, use multiple models to answer questions in the same conversation, and supports drag-and-drop sorting, code highlighting, and Mermaid chart. The tool is designed to enhance productivity and streamline the process of interacting with various language models.
com.openai.unity
com.openai.unity is an OpenAI package for Unity that allows users to interact with OpenAI's API through RESTful requests. It is independently developed and not an official library affiliated with OpenAI. Users can fine-tune models, create assistants, chat completions, and more. The package requires Unity 2021.3 LTS or higher and can be installed via Unity Package Manager or Git URL. Various features like authentication, Azure OpenAI integration, model management, thread creation, chat completions, audio processing, image generation, file management, fine-tuning, batch processing, embeddings, and content moderation are available.
mindsdb
MindsDB is a platform for customizing AI from enterprise data. You can create, serve, and fine-tune models in real-time from your database, vector store, and application data. MindsDB "enhances" SQL syntax with AI capabilities to make it accessible for developers worldwide. With MindsDB’s nearly 200 integrations, any developer can create AI customized for their purpose, faster and more securely. Their AI systems will constantly improve themselves — using companies’ own data, in real-time.
training-operator
Kubeflow Training Operator is a Kubernetes-native project for fine-tuning and scalable distributed training of machine learning (ML) models created with various ML frameworks such as PyTorch, Tensorflow, XGBoost, MPI, Paddle and others. Training Operator allows you to use Kubernetes workloads to effectively train your large models via Kubernetes Custom Resources APIs or using Training Operator Python SDK. > Note: Before v1.2 release, Kubeflow Training Operator only supports TFJob on Kubernetes. * For a complete reference of the custom resource definitions, please refer to the API Definition. * TensorFlow API Definition * PyTorch API Definition * Apache MXNet API Definition * XGBoost API Definition * MPI API Definition * PaddlePaddle API Definition * For details of all-in-one operator design, please refer to the All-in-one Kubeflow Training Operator * For details on its observability, please refer to the monitoring design doc.
helix
HelixML is a private GenAI platform that allows users to deploy the best of open AI in their own data center or VPC while retaining complete data security and control. It includes support for fine-tuning models with drag-and-drop functionality. HelixML brings the best of open source AI to businesses in an ergonomic and scalable way, optimizing the tradeoff between GPU memory and latency.
nntrainer
NNtrainer is a software framework for training neural network models on devices with limited resources. It enables on-device fine-tuning of neural networks using user data for personalization. NNtrainer supports various machine learning algorithms and provides examples for tasks such as few-shot learning, ResNet, VGG, and product rating. It is optimized for embedded devices and utilizes CBLAS and CUBLAS for accelerated calculations. NNtrainer is open source and released under the Apache License version 2.0.
For similar jobs
promptflow
**Prompt flow** is a suite of development tools designed to streamline the end-to-end development cycle of LLM-based AI applications, from ideation, prototyping, testing, evaluation to production deployment and monitoring. It makes prompt engineering much easier and enables you to build LLM apps with production quality.
deepeval
DeepEval is a simple-to-use, open-source LLM evaluation framework specialized for unit testing LLM outputs. It incorporates various metrics such as G-Eval, hallucination, answer relevancy, RAGAS, etc., and runs locally on your machine for evaluation. It provides a wide range of ready-to-use evaluation metrics, allows for creating custom metrics, integrates with any CI/CD environment, and enables benchmarking LLMs on popular benchmarks. DeepEval is designed for evaluating RAG and fine-tuning applications, helping users optimize hyperparameters, prevent prompt drifting, and transition from OpenAI to hosting their own Llama2 with confidence.
MegaDetector
MegaDetector is an AI model that identifies animals, people, and vehicles in camera trap images (which also makes it useful for eliminating blank images). This model is trained on several million images from a variety of ecosystems. MegaDetector is just one of many tools that aims to make conservation biologists more efficient with AI. If you want to learn about other ways to use AI to accelerate camera trap workflows, check out our of the field, affectionately titled "Everything I know about machine learning and camera traps".
leapfrogai
LeapfrogAI is a self-hosted AI platform designed to be deployed in air-gapped resource-constrained environments. It brings sophisticated AI solutions to these environments by hosting all the necessary components of an AI stack, including vector databases, model backends, API, and UI. LeapfrogAI's API closely matches that of OpenAI, allowing tools built for OpenAI/ChatGPT to function seamlessly with a LeapfrogAI backend. It provides several backends for various use cases, including llama-cpp-python, whisper, text-embeddings, and vllm. LeapfrogAI leverages Chainguard's apko to harden base python images, ensuring the latest supported Python versions are used by the other components of the stack. The LeapfrogAI SDK provides a standard set of protobuffs and python utilities for implementing backends and gRPC. LeapfrogAI offers UI options for common use-cases like chat, summarization, and transcription. It can be deployed and run locally via UDS and Kubernetes, built out using Zarf packages. LeapfrogAI is supported by a community of users and contributors, including Defense Unicorns, Beast Code, Chainguard, Exovera, Hypergiant, Pulze, SOSi, United States Navy, United States Air Force, and United States Space Force.
llava-docker
This Docker image for LLaVA (Large Language and Vision Assistant) provides a convenient way to run LLaVA locally or on RunPod. LLaVA is a powerful AI tool that combines natural language processing and computer vision capabilities. With this Docker image, you can easily access LLaVA's functionalities for various tasks, including image captioning, visual question answering, text summarization, and more. The image comes pre-installed with LLaVA v1.2.0, Torch 2.1.2, xformers 0.0.23.post1, and other necessary dependencies. You can customize the model used by setting the MODEL environment variable. The image also includes a Jupyter Lab environment for interactive development and exploration. Overall, this Docker image offers a comprehensive and user-friendly platform for leveraging LLaVA's capabilities.
carrot
The 'carrot' repository on GitHub provides a list of free and user-friendly ChatGPT mirror sites for easy access. The repository includes sponsored sites offering various GPT models and services. Users can find and share sites, report errors, and access stable and recommended sites for ChatGPT usage. The repository also includes a detailed list of ChatGPT sites, their features, and accessibility options, making it a valuable resource for ChatGPT users seeking free and unlimited GPT services.
TrustLLM
TrustLLM is a comprehensive study of trustworthiness in LLMs, including principles for different dimensions of trustworthiness, established benchmark, evaluation, and analysis of trustworthiness for mainstream LLMs, and discussion of open challenges and future directions. Specifically, we first propose a set of principles for trustworthy LLMs that span eight different dimensions. Based on these principles, we further establish a benchmark across six dimensions including truthfulness, safety, fairness, robustness, privacy, and machine ethics. We then present a study evaluating 16 mainstream LLMs in TrustLLM, consisting of over 30 datasets. The document explains how to use the trustllm python package to help you assess the performance of your LLM in trustworthiness more quickly. For more details about TrustLLM, please refer to project website.
AI-YinMei
AI-YinMei is an AI virtual anchor Vtuber development tool (N card version). It supports fastgpt knowledge base chat dialogue, a complete set of solutions for LLM large language models: [fastgpt] + [one-api] + [Xinference], supports docking bilibili live broadcast barrage reply and entering live broadcast welcome speech, supports Microsoft edge-tts speech synthesis, supports Bert-VITS2 speech synthesis, supports GPT-SoVITS speech synthesis, supports expression control Vtuber Studio, supports painting stable-diffusion-webui output OBS live broadcast room, supports painting picture pornography public-NSFW-y-distinguish, supports search and image search service duckduckgo (requires magic Internet access), supports image search service Baidu image search (no magic Internet access), supports AI reply chat box [html plug-in], supports AI singing Auto-Convert-Music, supports playlist [html plug-in], supports dancing function, supports expression video playback, supports head touching action, supports gift smashing action, supports singing automatic start dancing function, chat and singing automatic cycle swing action, supports multi scene switching, background music switching, day and night automatic switching scene, supports open singing and painting, let AI automatically judge the content.