cosmos-predict1

cosmos-predict1

Cosmos-Predict1 is a collection of general-purpose world foundation models for Physical AI that can be fine-tuned into customized world models for downstream applications.

Stars: 99

Visit
 screenshot

README:

NVIDIA Cosmos Header

Cosmos-Predict1 is a key branch of Cosmos World Foundation Models (WFMs) specialized for future state prediction, often referred to as world models. The tree main branches of Cosmos WFMs are cosmos-predict, cosmos-transfer, and cosmos-reason. We visualize the architecture of Cosmos-Predict1 in the following figure.

Cosmos-Predict1 Architecture Diagram

Cosmos-Predict1 includes the following:

  • Diffusion-based world foundation models for Text2World and Video2World generation, where a user can generate visual simulation based on text prompts and video prompts.
  • Autoregressive-based world foundation models for Video2World generation, where a user can generate visual simulation based on video prompts and optional text prompts.
  • Image and video tokenizers for tokenizing videos into continuous tokens (latent vectors) and discrete tokens (integers) efficiently and effectively.
  • Post-training scripts for helping Physical AI builders post-train pre-trained Cosmos-Predict1 for their applications.
  • Pre-training scripts for helping Physical AI builders train their WFMs from scratch.

Example Model Behavior

Cosmos-Predict Text2World

Your browser does not support the video tag.

Cosmos-Predict Video2World

Your browser does not support the video tag.

Getting Started

We provide a comphrehensive set of examples to illustrate how to perform inference, post-training, etc, with Cosmos-Predict1. Click a relevant example below and start your Cosmos journey.

Inference with pre-trained Cosmos-Predict1 models

Post-train pre-trained Cosmos-Predict1 models

Inference with post-trained models:

Cosmos-Predict1 Models

Cosmos-Predict1 include the following models

Diffusion models

Autoregressive models

Tokenizers

License and Contact

This project will download and install additional third-party open source software projects. Review the license terms of these open source projects before use.

NVIDIA Cosmos source code is released under the Apache 2 License.

NVIDIA Cosmos models are released under the NVIDIA Open Model License. For a custom license (such as exemption of guardrail), please contact [email protected].

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for cosmos-predict1

Similar Open Source Tools

For similar tasks

No tools available

For similar jobs

No tools available