prompt-generator-comfyui

prompt-generator-comfyui

Custom AI prompt generator node for the ComfyUI

Stars: 87

Visit
screenshot

Custom AI prompt generator node for ComfyUI. With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset.

README:

prompt-generator-comfyui

Custom AI prompt generator node for ComfyUI. With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset or you can use the pretrained models.

Table Of Contents

Setup

For Portable Installation of the ComfyUI

  • [x] Automatic installation is added for portable version.
  • Clone the repository with git clone https://github.com/alpertunga-bile/prompt-generator-comfyui.git command under custom_nodes folder.
  • Go to the ComfyUI_windows_portable folder and run the run_nvidia_gpu.bat file
  • Open the hires.fixWithPromptGenerator.json or basicWorkflowWithPromptGenerator.json workflow
  • Put your generator under the models/prompt_generators folder. You can create your prompt generator with this repository. You have to put generator as folder. Do not just put pytorch_model.bin file for example.
  • Click Refresh button in ComfyUI

For Manual Installation of the ComfyUI

  • Clone the repository with git clone https://github.com/alpertunga-bile/prompt-generator-comfyui.git command under custom_nodes folder.
  • Run the ComfyUI
  • Open the hires.fixWithPromptGenerator.json or basicWorkflowWithPromptGenerator.json workflow
  • Put your generator under the models/prompt_generators folder. You can create your prompt generator with this repository. You have to put generator as folder. Do not just put pytorch_model.bin file for example.
  • Click Refresh button in ComfyUI

For ComfyUI Manager Users

  • Download the node with ComfyUI Manager
  • Restart the ComfyUI
  • Open the hires.fixWithPromptGenerator.json or basicWorkflowWithPromptGenerator.json workflow
  • Put your generator under the models/prompt_generators folder. You can create your prompt generator with this repository. You have to put generator as folder. Do not just put pytorch_model.bin file for example.
  • Click Refresh button in ComfyUI

Features

  • Multiple output generation is added. You can choose from 5 outputs with the index value. You can check the generated prompts from the log file and terminal. The prompts are logged and printed in order.
  • Randomness is added. See this section.
  • Quantization is added with Quanto and Bitsandbytes packages. See this section.
  • Lora adapter model loading is added with Peft package. (The feature is not full tested in this repository because of my low VRAM but I am using the same implementation in Google Colab for training and inference and it is working there)
  • Optimizations are done with Optimum package.
  • ONNX and transformers models are supported.
  • Preprocessing outputs. See this section.
  • Recursive generation is supported. See this section.
  • Print generated text to terminal and log the node's state under the generated_prompts folder with date as filename.

Example Workflow

example_hires_workflow

example_basic_workflow

  • Prompt Generator Node may look different with final version but workflow in ComfyUI is not going to change

Pretrained Prompt Models

  • You can find the models in this link

  • For to use the pretrained model follow these steps:

    • Download the model and unzip to models/prompt_generators folder.
    • Click Refresh button in ComfyUI.
    • Then select the generator with the node's model_name variable (If you can't see the generator, restart the ComfyUI).

Dataset

Models

  • The model versions are used to differentiate models rather than showing which one is better.

  • The v2 version is the latest trained model and the v4 and v5 models are experimental models.

  • female_positive_generator_v2 | (Training In Process)

  • female_positive_generator_v3 | (Training In Process)

  • female_positive_generator_v4 | Experimental

Variables

Variable Names Definitions
model_name Folder name that contains the model
accelerate Open optimizations. Some of the models are not supported by BetterTransformer (Check your model). If it is not supported switch this option to disable or convert your model to ONNX
quantize Quantize the model. The quantize type is changed based on your OS and torch version. none value disables the quantization. Check this section for more information
token_healing Enable token healing algorithm which is used for fixing unintended bias. Read more from this blog post. To use it, the transformers package version has to be newer from or equal to 4.6 version
prompt Input prompt for the generator
seed Seed value for the model
lock Lock the generation and select from the last generated prompts with index value
random_index Random index value in [1, 5]. If the value is enable, the index value is not used
index User specified index value for selecting prompt from the generated prompts. random_index variable must be disable
cfg CFG is enabled by setting guidance_scale > 1. Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer quality
min_new_tokens The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt.
max_new_tokens The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt.
do_sample Whether or not to use sampling; use greedy decoding otherwise
early_stopping Controls the stopping condition for beam-based methods, like beam-search
num_beams Number of steps for each search path
num_beam_groups Number of groups to divide num_beams into in order to ensure diversity among different groups of beams
diversity_penalty This value is subtracted from a beam’s score if it generates a token same as any beam from other group at a particular time. Note that diversity_penalty is only effective if group beam search is enabled.
temperature How sensitive the algorithm is to selecting low probability options
top_k The number of highest probability vocabulary tokens to keep for top-k-filtering
top_p If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation
repetition_penalty The parameter for repetition penalty. 1.0 means no penalty
no_repeat_ngram_size The size of an n-gram that cannot occur more than once. (0=infinity)
remove_invalid_values Whether to remove possible nan and inf outputs of the model to prevent the generation method to crash. Note that using remove_invalid_values can slow down generation.
self_recursive See this section
recursive_level See this section
preprocess_mode See this section

Quantization

  • Quantization is added with Quanto and Bitsandbytes packages.
  • The Quanto package requires torch >= 2.4 and Bitsandbytes package works out-of-box with Linux OS. So the node is checking which package to use:
    • If requirements are not specified for this packages, you can not use the quantize variable and it has only none value.
    • If the Quanto requirements are filled then you can choose between none, int8, float8, int4 values.
    • If the Bitsandbytes requirements are filled then you can choose between none, int8, int4 values.
  • If your environment can use the Quanto and Bitsandbytes packages, the node selects the Bitsandbytes package.

Random Generation

  • For random generation:

    • Enable do_sample

  • You can find this text generation strategy from the upper link. The strategy is called Multinomial sampling.

  • Changing variable of do_sample to disable gives deterministic generation.

  • For more randomness, you can:

    • Set num_beams to 1
    • Enable random_index variable
    • Increase recursive_level
    • Enable self_recursive

Lock The Generation

  • Enabling the lock variable skip the generation and let you choose from the last generated prompts.
  • You can choose from the index value or use the random_index.
  • If random_index is enabled, the index value is ignored.

How Recursive Works?

  • Let's say we give a, as seed and recursive level is 1. I am going to use the same outputs for this example to explain the functionality more understandable.
  • With self recursive, let's say generator's output is b. So next seed is going to be b and generator's output is c. Final output is a, c. It can be used for generating random outputs.
  • Without self recursive, let's say generator's output is b. So next seed is going to be a, b and generator's output is c. Final output is a, b, c. It can be used for more accurate prompts.

How Preprocess Mode Works?

  • exact_keyword => (masterpiece), ((masterpiece)) is not allowed. Checking the pure keyword without parantheses and weights. The algorithm is adding the prompts from the beginning of the generated text, so add important prompts to prompt variable.
  • exact_prompt => (masterpiece), ((masterpiece)) is allowed but (masterpiece), (masterpiece) is not. Checking the exact match of the prompt.
  • none => Everything is allowed even the repeated prompts.

Example

# ---------------------------------------------------------------------- Original ---------------------------------------------------------------------- #
((masterpiece)), ((masterpiece:1.2)), (masterpiece), blahblah, blah, blah, ((blahblah)), (((((blah))))), ((same prompt)), same prompt, (masterpiece)
# ------------------------------------------------------------- Preprocess (Exact Keyword) ------------------------------------------------------------- #
((masterpiece)), blahblah, blah, ((same prompt))
# ------------------------------------------------------------- Preprocess (Exact Prompt) -------------------------------------------------------------- #
((masterpiece)), ((masterpiece:1.2)), (masterpiece), blahblah, blah, ((blahblah)), (((((blah))))), ((same prompt)), same prompt

Troubleshooting

  • If the below solutions are not fixed your issue please create an issue with bug label

Package Version

  • The node is based on transformers and optimum packages. So most of the problems may be caused from these packages. For overcome these problems you can try to update these packages:

For Manual Installation of the ComfyUI

  1. Activate the virtual environment if there is one.
  2. Run the pip install --upgrade transformers optimum optimum[onnxruntime-gpu] command.

For Portable Installation of the ComfyUI

  1. Go to the ComfyUI_windows_portable folder.
  2. Open the command prompt in this folder.
  3. Run the .\python_embeded\python.exe -s -m pip install --upgrade transformers optimum optimum[onnxruntime-gpu] command.

Automatic Installation

For Manual Installation of the ComfyUI

  • The users have to check if they activate the virtual environment if there is one.

For Portable Installation of the ComfyUI

  • The users have to check that they are starting the ComfyUI in the ComfyUI_windows_portable folder.
  • Because the node is checking the python_embeded folder if it is exists and is using it to install the required packages.

New Updates On The Node

  • Sometimes the variables are changed with updates, so it may broke the workflow. But don't worry, you have to just delete the node in the workflow and add it again.

Contributing

  • Contributions are welcome. If you have an idea and want to implement it by yourself please follow these steps:

    1. Create a fork
    2. Pull request the fork with the description that explaining the new feature

  • If you have an idea but don't know how to implement it, please create an issue with enhancement label.

  • [x] The contributing can be done in several ways. You can contribute to code or to README file.

Example Outputs

first_example second_example third_example

For Tasks:

Click tags to check more tools for each tasks
generate ideas write headlines create outlines expand text translate languages

For Jobs:

content writer copywriter creative writer journalist marketer

Alternative AI tools for prompt-generator-comfyui

Similar Open Source Tools

prompt-generator-comfyui Screenshot
prompt-generator-comfyui

Custom AI prompt generator node for ComfyUI. With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset.

github
: 87
Chatbook Screenshot
Chatbook

Chatbook is a paclet that adds support for LLM-powered notebooks to Wolfram. It allows users to interact with ChatGPT and generate immediately evaluatable Wolfram code. The code can be evaluated in place immediately, making it easy to explore and experiment with ideas.

github
: 96
classifai Screenshot
classifai

Supercharge WordPress Content Workflows and Engagement with Artificial Intelligence. Tap into leading cloud-based services like OpenAI, Microsoft Azure AI, Google Gemini and IBM Watson to augment your WordPress-powered websites. Publish content faster while improving SEO performance and increasing audience engagement. ClassifAI integrates Artificial Intelligence and Machine Learning technologies to lighten your workload and eliminate tedious tasks, giving you more time to create original content that matters.

github
: 669
nvim-aider Screenshot
nvim-aider

Nvim-aider is a plugin for Neovim that provides additional functionality and key mappings to enhance the user's editing experience. It offers features such as code navigation, quick access to commonly used commands, and improved text manipulation tools. With Nvim-aider, users can streamline their workflow and increase productivity while working with Neovim.

github
: 86
note-gen Screenshot
note-gen

Note-gen is a simple tool for generating notes automatically based on user input. It uses natural language processing techniques to analyze text and extract key information to create structured notes. The tool is designed to save time and effort for users who need to summarize large amounts of text or generate notes quickly. With note-gen, users can easily create organized and concise notes for study, research, or any other purpose.

github
: 761
awesome-VLLMs Screenshot
awesome-VLLMs

This repository contains a collection of pre-trained Very Large Language Models (VLLMs) that can be used for various natural language processing tasks. The models are fine-tuned on large text corpora and can be easily integrated into existing NLP pipelines for tasks such as text generation, sentiment analysis, and language translation. The repository also provides code examples and tutorials to help users get started with using these powerful language models in their projects.

github
: 165
prompt-optimizer Screenshot
prompt-optimizer

Prompt Optimizer is a powerful AI prompt optimization tool that helps you write better AI prompts, improving AI output quality. It supports both web application and Chrome extension usage. The tool features intelligent optimization for prompt words, real-time testing to compare before and after optimization, integration with multiple mainstream AI models, client-side processing for security, encrypted local storage for data privacy, responsive design for user experience, and more.

github
: 14.7k
Tiktoken Screenshot
Tiktoken

Tiktoken is a high-performance implementation focused on token count operations. It provides various encodings like o200k_base, cl100k_base, r50k_base, p50k_base, and p50k_edit. Users can easily encode and decode text using the provided API. The repository also includes a benchmark console app for performance tracking. Contributions in the form of PRs are welcome.

github
: 78
MarkFlowy Screenshot
MarkFlowy

MarkFlowy is a lightweight and feature-rich Markdown editor with built-in AI capabilities. It supports one-click export of conversations, translation of articles, and obtaining article abstracts. Users can leverage large AI models like DeepSeek and Chatgpt as intelligent assistants. The editor provides high availability with multiple editing modes and custom themes. Available for Linux, macOS, and Windows, MarkFlowy aims to offer an efficient, beautiful, and data-safe Markdown editing experience for users.

github
: 1.5k
orate Screenshot
orate

Orate is an AI toolkit designed for speech processing tasks. It allows users to generate realistic, human-like speech and transcribe audio using a unified API that integrates with popular AI providers such as OpenAI, ElevenLabs, and AssemblyAI. The toolkit can be easily installed using npm or other package managers. For more details, visit the website.

github
: 363
simple-ai Screenshot
simple-ai

Simple AI is a lightweight Python library for implementing basic artificial intelligence algorithms. It provides easy-to-use functions and classes for tasks such as machine learning, natural language processing, and computer vision. With Simple AI, users can quickly prototype and deploy AI solutions without the complexity of larger frameworks.

github
: 276
CodeWebChat Screenshot
CodeWebChat

Code Web Chat is a versatile, free, and open-source AI pair programming tool with a unique web-based workflow. Users can select files, type instructions, and initialize various chatbots like ChatGPT, Gemini, Claude, and more hands-free. The tool helps users save money with free tiers and subscription-based billing and save time with multi-file edits from a single prompt. It supports chatbot initialization through the Connector browser extension and offers API tools for code completions, editing context, intelligent updates, and commit messages. Users can handle AI responses, code completions, and version control through various commands. The tool is privacy-focused, operates locally, and supports any OpenAI-API compatible provider for its utilities.

github
: 1.1k
trustgraph Screenshot
trustgraph

TrustGraph is a tool that deploys private GraphRAG pipelines to build a RDF style knowledge graph from data, enabling accurate and secure `RAG` requests compatible with cloud LLMs and open-source SLMs. It showcases the reliability and efficiencies of GraphRAG algorithms, capturing contextual language flags missed in conventional RAG approaches. The tool offers features like PDF decoding, text chunking, inference of various LMs, RDF-aligned Knowledge Graph extraction, and more. TrustGraph is designed to be modular, supporting multiple Language Models and environments, with a plug'n'play architecture for easy customization.

github
: 549
cellm Screenshot
cellm

Cellm is an Excel extension that allows users to leverage Large Language Models (LLMs) like ChatGPT within cell formulas. It enables users to extract AI responses to text ranges, making it useful for automating repetitive tasks that involve data processing and analysis. Cellm supports various models from Anthropic, Mistral, OpenAI, and Google, as well as locally hosted models via Llamafiles, Ollama, or vLLM. The tool is designed to simplify the integration of AI capabilities into Excel for tasks such as text classification, data cleaning, content summarization, entity extraction, and more.

github
: 882
memfree Screenshot
memfree

MemFree is an open-source hybrid AI search engine that allows users to simultaneously search their personal knowledge base (bookmarks, notes, documents, etc.) and the Internet. It features a self-hosted super fast serverless vector database, local embedding and rerank service, one-click Chrome bookmarks index, and full code open source. Users can contribute by opening issues for bugs or making pull requests for new features or improvements.

github
: 1.2k
Rodel.Agent Screenshot
Rodel.Agent

Rodel Agent is a Windows desktop application that integrates chat, text-to-image, text-to-speech, and machine translation services, providing users with a comprehensive desktop AI experience. The application supports mainstream AI services and aims to enhance user interaction through various AI functionalities.

github
: 297

For similar tasks

prompt-generator-comfyui Screenshot
prompt-generator-comfyui

Custom AI prompt generator node for ComfyUI. With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset.

github
: 87
lumentis Screenshot
lumentis

Lumentis is a tool that allows users to generate beautiful and comprehensive documentation from meeting transcripts and large documents with a single command. It reads transcripts, asks questions to understand themes and audience, generates an outline, and creates detailed pages with visual variety and styles. Users can switch models for different tasks, control the process, and deploy the generated docs to Vercel. The tool is designed to be open, clean, fast, and easy to use, with upcoming features including folders, PDFs, auto-transcription, website scraping, scientific papers handling, summarization, and continuous updates.

github
: 1.4k
khoj Screenshot
khoj

Khoj is an open-source, personal AI assistant that extends your capabilities by creating always-available AI agents. You can share your notes and documents to extend your digital brain, and your AI agents have access to the internet, allowing you to incorporate real-time information. Khoj is accessible on Desktop, Emacs, Obsidian, Web, and Whatsapp, and you can share PDF, markdown, org-mode, notion files, and GitHub repositories. You'll get fast, accurate semantic search on top of your docs, and your agents can create deeply personal images and understand your speech. Khoj is self-hostable and always will be.

github
: 28.5k
quivr Screenshot
quivr

Quivr is a personal assistant powered by Generative AI, designed to be a second brain for users. It offers fast and efficient access to data, ensuring security and compatibility with various file formats. Quivr is open source and free to use, allowing users to share their brains publicly or keep them private. The marketplace feature enables users to share and utilize brains created by others, boosting productivity. Quivr's offline mode provides anytime, anywhere access to data. Key features include speed, security, OS compatibility, file compatibility, open source nature, public/private sharing options, a marketplace, and offline mode.

github
: 37.6k
SillyTavern Screenshot
SillyTavern

SillyTavern is a user interface you can install on your computer (and Android phones) that allows you to interact with text generation AIs and chat/roleplay with characters you or the community create. SillyTavern is a fork of TavernAI 1.2.8 which is under more active development and has added many major features. At this point, they can be thought of as completely independent programs.

github
: 18.4k
wingman-ai Screenshot
wingman-ai

Wingman-AI is a free and open-source AI coding assistant that brings high-quality AI-assisted coding right to your computer. It offers features such as code completion, interactive chat, and support for multiple AI providers, including Ollama, Hugging Face, and OpenAI. Wingman-AI is designed to enhance your coding workflow by providing real-time assistance and suggestions, making it an ideal tool for developers of all levels.

github
: 210
morphic Screenshot
morphic

Morphic is an AI-powered answer engine with a generative UI. It utilizes a stack of Next.js, Vercel AI SDK, OpenAI, Tavily AI, shadcn/ui, Radix UI, and Tailwind CSS. To get started, fork and clone the repo, install dependencies, fill out secrets in the .env.local file, and run the app locally using 'bun dev'. You can also deploy your own live version of Morphic with Vercel. Verified models that can be specified to writers include Groq, LLaMA3 8b, and LLaMA3 70b.

github
: 7.2k
gpt-engineer Screenshot
gpt-engineer

GPT-Engineer is a tool that allows you to specify a software in natural language, sit back and watch as an AI writes and executes the code, and ask the AI to implement improvements.

github
: 51.9k

For similar jobs

ChatFAQ Screenshot
ChatFAQ

ChatFAQ is an open-source comprehensive platform for creating a wide variety of chatbots: generic ones, business-trained, or even capable of redirecting requests to human operators. It includes a specialized NLP/NLG engine based on a RAG architecture and customized chat widgets, ensuring a tailored experience for users and avoiding vendor lock-in.

github
: 128
anything-llm Screenshot
anything-llm

AnythingLLM is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting. This application allows you to pick and choose which LLM or Vector Database you want to use as well as supporting multi-user management and permissions.

github
: 42.1k
ai-guide Screenshot
ai-guide

This guide is dedicated to Large Language Models (LLMs) that you can run on your home computer. It assumes your PC is a lower-end, non-gaming setup.

github
: 159
classifai Screenshot
classifai

Supercharge WordPress Content Workflows and Engagement with Artificial Intelligence. Tap into leading cloud-based services like OpenAI, Microsoft Azure AI, Google Gemini and IBM Watson to augment your WordPress-powered websites. Publish content faster while improving SEO performance and increasing audience engagement. ClassifAI integrates Artificial Intelligence and Machine Learning technologies to lighten your workload and eliminate tedious tasks, giving you more time to create original content that matters.

github
: 669
mikupad Screenshot
mikupad

mikupad is a lightweight and efficient language model front-end powered by ReactJS, all packed into a single HTML file. Inspired by the likes of NovelAI, it provides a simple yet powerful interface for generating text with the help of various backends.

github
: 300
glide Screenshot
glide

Glide is a cloud-native LLM gateway that provides a unified REST API for accessing various large language models (LLMs) from different providers. It handles LLMOps tasks such as model failover, caching, key management, and more, making it easy to integrate LLMs into applications. Glide supports popular LLM providers like OpenAI, Anthropic, Azure OpenAI, AWS Bedrock (Titan), Cohere, Google Gemini, OctoML, and Ollama. It offers high availability, performance, and observability, and provides SDKs for Python and NodeJS to simplify integration.

github
: 110
onnxruntime-genai Screenshot
onnxruntime-genai

ONNX Runtime Generative AI is a library that provides the generative AI loop for ONNX models, including inference with ONNX Runtime, logits processing, search and sampling, and KV cache management. Users can call a high level `generate()` method, or run each iteration of the model in a loop. It supports greedy/beam search and TopP, TopK sampling to generate token sequences, has built in logits processing like repetition penalties, and allows for easy custom scoring.

github
: 829
firecrawl Screenshot
firecrawl

Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. It crawls all accessible subpages and provides clean markdown for each, without requiring a sitemap. The API is easy to use and can be self-hosted. It also integrates with Langchain and Llama Index. The Python SDK makes it easy to crawl and scrape websites in Python code.

github
: 34.1k