prompt-generator-comfyui
Custom AI prompt generator node for the ComfyUI
Stars: 87
Custom AI prompt generator node for ComfyUI. With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset.
README:
Custom AI prompt generator node for ComfyUI. With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset or you can use the pretrained models.
- prompt-generator-comfyui
- Table Of Contents
- Setup
- Features
- Example Workflow
- Pretrained Prompt Models
- Variables
- Troubleshooting
- Contributing
- Example Outputs
- [x] Automatic installation is added for portable version.
- Clone the repository with
git clone https://github.com/alpertunga-bile/prompt-generator-comfyui.gitcommand undercustom_nodesfolder. - Go to the
ComfyUI_windows_portablefolder and run the run_nvidia_gpu.bat file - Open the
hires.fixWithPromptGenerator.jsonorbasicWorkflowWithPromptGenerator.jsonworkflow - Put your generator under the
models/prompt_generatorsfolder. You can create your prompt generator with this repository. You have to put generator as folder. Do not just putpytorch_model.binfile for example. - Click
Refreshbutton in ComfyUI
- Clone the repository with
git clone https://github.com/alpertunga-bile/prompt-generator-comfyui.gitcommand undercustom_nodesfolder. - Run the ComfyUI
- Open the
hires.fixWithPromptGenerator.jsonorbasicWorkflowWithPromptGenerator.jsonworkflow - Put your generator under the
models/prompt_generatorsfolder. You can create your prompt generator with this repository. You have to put generator as folder. Do not just putpytorch_model.binfile for example. - Click
Refreshbutton in ComfyUI
- Download the node with ComfyUI Manager
- Restart the ComfyUI
- Open the
hires.fixWithPromptGenerator.jsonorbasicWorkflowWithPromptGenerator.jsonworkflow - Put your generator under the
models/prompt_generatorsfolder. You can create your prompt generator with this repository. You have to put generator as folder. Do not just putpytorch_model.binfile for example. - Click
Refreshbutton in ComfyUI
- Multiple output generation is added. You can choose from 5 outputs with the index value. You can check the generated prompts from the log file and terminal. The prompts are logged and printed in order.
- Randomness is added. See this section.
- Quantization is added with Quanto and Bitsandbytes packages. See this section.
- Lora adapter model loading is added with Peft package. (The feature is not full tested in this repository because of my low VRAM but I am using the same implementation in Google Colab for training and inference and it is working there)
- Optimizations are done with Optimum package.
- ONNX and transformers models are supported.
- Preprocessing outputs. See this section.
- Recursive generation is supported. See this section.
- Print generated text to terminal and log the node's state under the
generated_promptsfolder with date as filename.
- Prompt Generator Node may look different with final version but workflow in ComfyUI is not going to change
-
You can find the models in this link
-
For to use the pretrained model follow these steps:
- Download the model and unzip to
models/prompt_generatorsfolder. - Click
Refreshbutton in ComfyUI. - Then select the generator with the node's
model_namevariable (If you can't see the generator, restart the ComfyUI).
- Download the model and unzip to
- Huggingface Dataset
- Process of data cleaning and gathering can be found here
-
The model versions are used to differentiate models rather than showing which one is better.
-
The v2 version is the latest trained model and the v4 and v5 models are experimental models.
-
female_positive_generator_v2 | (Training In Process)
- Base model
- using distilgpt2 model
- ~500 MB
-
female_positive_generator_v3 | (Training In Process)
- Base model
- using bigscience/bloom-560m model
- ~1.3 GB
-
female_positive_generator_v4 | Experimental
- Lora adapter model
- using senseable/WestLake-7B-v2 model as the base model
- Base model is ~14 GB
| Variable Names | Definitions |
|---|---|
| model_name | Folder name that contains the model |
| accelerate | Open optimizations. Some of the models are not supported by BetterTransformer (Check your model). If it is not supported switch this option to disable or convert your model to ONNX |
| quantize | Quantize the model. The quantize type is changed based on your OS and torch version. none value disables the quantization. Check this section for more information |
| token_healing | Enable token healing algorithm which is used for fixing unintended bias. Read more from this blog post. To use it, the transformers package version has to be newer from or equal to 4.6 version |
| prompt | Input prompt for the generator |
| seed | Seed value for the model |
| lock | Lock the generation and select from the last generated prompts with index value |
| random_index | Random index value in [1, 5]. If the value is enable, the index value is not used |
| index | User specified index value for selecting prompt from the generated prompts. random_index variable must be disable |
| cfg | CFG is enabled by setting guidance_scale > 1. Higher guidance scale encourages the model to generate samples that are more closely linked to the input prompt, usually at the expense of poorer quality |
| min_new_tokens | The minimum numbers of tokens to generate, ignoring the number of tokens in the prompt. |
| max_new_tokens | The maximum numbers of tokens to generate, ignoring the number of tokens in the prompt. |
| do_sample | Whether or not to use sampling; use greedy decoding otherwise |
| early_stopping | Controls the stopping condition for beam-based methods, like beam-search |
| num_beams | Number of steps for each search path |
| num_beam_groups | Number of groups to divide num_beams into in order to ensure diversity among different groups of beams |
| diversity_penalty | This value is subtracted from a beam’s score if it generates a token same as any beam from other group at a particular time. Note that diversity_penalty is only effective if group beam search is enabled. |
| temperature | How sensitive the algorithm is to selecting low probability options |
| top_k | The number of highest probability vocabulary tokens to keep for top-k-filtering |
| top_p | If set to float < 1, only the smallest set of most probable tokens with probabilities that add up to top_p or higher are kept for generation |
| repetition_penalty | The parameter for repetition penalty. 1.0 means no penalty |
| no_repeat_ngram_size | The size of an n-gram that cannot occur more than once. (0=infinity) |
| remove_invalid_values | Whether to remove possible nan and inf outputs of the model to prevent the generation method to crash. Note that using remove_invalid_values can slow down generation. |
| self_recursive | See this section |
| recursive_level | See this section |
| preprocess_mode | See this section |
- Quantization is added with Quanto and Bitsandbytes packages.
- The Quanto package requires
torch >= 2.4and Bitsandbytes package works out-of-box with Linux OS. So the node is checking which package to use:- If requirements are not specified for this packages, you can not use the
quantizevariable and it has onlynonevalue. - If the Quanto requirements are filled then you can choose between
none, int8, float8, int4values. - If the Bitsandbytes requirements are filled then you can choose between
none, int8, int4values.
- If requirements are not specified for this packages, you can not use the
- If your environment can use the Quanto and Bitsandbytes packages, the node selects the Bitsandbytes package.
-
For random generation:
- Enable do_sample
-
You can find this text generation strategy from the upper link. The strategy is called Multinomial sampling.
-
Changing variable of do_sample to disable gives deterministic generation.
-
For more randomness, you can:
- Set num_beams to 1
- Enable random_index variable
- Increase recursive_level
- Enable self_recursive
- Enabling the lock variable skip the generation and let you choose from the last generated prompts.
- You can choose from the index value or use the random_index.
- If random_index is enabled, the index value is ignored.
- Let's say we give
a,as seed and recursive level is 1. I am going to use the same outputs for this example to explain the functionality more understandable. - With self recursive, let's say generator's output is
b. So next seed is going to beband generator's output isc. Final output isa, c. It can be used for generating random outputs. - Without self recursive, let's say generator's output is
b. So next seed is going to bea, band generator's output isc. Final output isa, b, c. It can be used for more accurate prompts.
-
exact_keyword =>
(masterpiece), ((masterpiece))is not allowed. Checking the pure keyword without parantheses and weights. The algorithm is adding the prompts from the beginning of the generated text, so add important prompts to prompt variable. -
exact_prompt =>
(masterpiece), ((masterpiece))is allowed but(masterpiece), (masterpiece)is not. Checking the exact match of the prompt. - none => Everything is allowed even the repeated prompts.
# ---------------------------------------------------------------------- Original ---------------------------------------------------------------------- #
((masterpiece)), ((masterpiece:1.2)), (masterpiece), blahblah, blah, blah, ((blahblah)), (((((blah))))), ((same prompt)), same prompt, (masterpiece)
# ------------------------------------------------------------- Preprocess (Exact Keyword) ------------------------------------------------------------- #
((masterpiece)), blahblah, blah, ((same prompt))
# ------------------------------------------------------------- Preprocess (Exact Prompt) -------------------------------------------------------------- #
((masterpiece)), ((masterpiece:1.2)), (masterpiece), blahblah, blah, ((blahblah)), (((((blah))))), ((same prompt)), same prompt
- If the below solutions are not fixed your issue please create an issue with
buglabel
- The node is based on transformers and optimum packages. So most of the problems may be caused from these packages. For overcome these problems you can try to update these packages:
- Activate the virtual environment if there is one.
- Run the
pip install --upgrade transformers optimum optimum[onnxruntime-gpu]command.
- Go to the
ComfyUI_windows_portablefolder. - Open the command prompt in this folder.
- Run the
.\python_embeded\python.exe -s -m pip install --upgrade transformers optimum optimum[onnxruntime-gpu]command.
- The users have to check if they activate the virtual environment if there is one.
- The users have to check that they are starting the ComfyUI in the
ComfyUI_windows_portablefolder. - Because the node is checking the
python_embededfolder if it is exists and is using it to install the required packages.
- Sometimes the variables are changed with updates, so it may broke the workflow. But don't worry, you have to just delete the node in the workflow and add it again.
-
Contributions are welcome. If you have an idea and want to implement it by yourself please follow these steps:
- Create a fork
- Pull request the fork with the description that explaining the new feature
-
If you have an idea but don't know how to implement it, please create an issue with
enhancementlabel. -
[x] The contributing can be done in several ways. You can contribute to code or to README file.
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for prompt-generator-comfyui
Similar Open Source Tools
prompt-generator-comfyui
Custom AI prompt generator node for ComfyUI. With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset.
lector
Lector is a text analysis tool that helps users extract insights from unstructured text data. It provides functionalities such as sentiment analysis, keyword extraction, entity recognition, and text summarization. With Lector, users can easily analyze large volumes of text data to uncover patterns, trends, and valuable information. The tool is designed to be user-friendly and efficient, making it suitable for both beginners and experienced users in the field of natural language processing and text mining.
Top-AI-Tools
Top AI Tools is a comprehensive, community-curated directory that aims to catalog and showcase the most outstanding AI-powered products. This index is not exhaustive, but rather a compilation of our research and contributions from the community.
memfree
MemFree is an open-source hybrid AI search engine that allows users to simultaneously search their personal knowledge base (bookmarks, notes, documents, etc.) and the Internet. It features a self-hosted super fast serverless vector database, local embedding and rerank service, one-click Chrome bookmarks index, and full code open source. Users can contribute by opening issues for bugs or making pull requests for new features or improvements.
omnichain
OmniChain is a tool for building efficient self-updating visual workflows using AI language models, enabling users to automate tasks, create chatbots, agents, and integrate with existing frameworks. It allows users to create custom workflows guided by logic processes, store and recall information, and make decisions based on that information. The tool enables users to create tireless robot employees that operate 24/7, access the underlying operating system, generate and run NodeJS code snippets, and create custom agents and logic chains. OmniChain is self-hosted, open-source, and available for commercial use under the MIT license, with no coding skills required.
enterprise-h2ogpte
Enterprise h2oGPTe - GenAI RAG is a repository containing code examples, notebooks, and benchmarks for the enterprise version of h2oGPTe, a powerful AI tool for generating text based on the RAG (Retrieval-Augmented Generation) architecture. The repository provides resources for leveraging h2oGPTe in enterprise settings, including implementation guides, performance evaluations, and best practices. Users can explore various applications of h2oGPTe in natural language processing tasks, such as text generation, content creation, and conversational AI.
orcish-ai-nextjs-framework
The Orcish AI Next.js Framework is a powerful tool that leverages OpenAI API to seamlessly integrate AI functionalities into Next.js applications. It allows users to generate text, images, and text-to-speech based on specified input. The framework provides an easy-to-use interface for utilizing AI capabilities in application development.
amazon-sagemaker-generativeai
Repository for training and deploying Generative AI models, including text-text, text-to-image generation, prompt engineering playground and chain of thought examples using SageMaker Studio. The tool provides a platform for users to experiment with generative AI techniques, enabling them to create text and image outputs based on input data. It offers a range of functionalities for training and deploying models, as well as exploring different generative AI applications.
ai-chatbot-framework
An AI Chatbot framework built in Python. It allows users to easily create Natural Language conversational scenarios with no coding efforts. The tool continuously learns from conversations to improve its capabilities. It can be integrated with various channels like Messenger and Slack. Users can create AI-powered chatbots without expertise in artificial intelligence.
douyin-chatgpt-bot
Douyin ChatGPT Bot is an AI-driven system for automatic replies on Douyin, including comment and private message replies. It offers features such as comment filtering, customizable robot responses, and automated account management. The system aims to enhance user engagement and brand image on the Douyin platform, providing a seamless experience for managing interactions with followers and potential customers.
mslearn-ai-language
This repository contains lab files for Azure AI Language modules. It provides hands-on exercises and resources for learning about various AI language technologies on the Azure platform. The labs cover topics such as natural language processing, text analytics, language understanding, and more. By following the exercises in this repository, users can gain practical experience in implementing AI language solutions using Azure services.
aide
Aide is a code-first API documentation and utility library for Rust, along with other related utility crates for web-servers. It provides tools for creating API documentation and handling JSON request validation. The repository contains multiple crates that offer drop-in replacements for existing libraries, ensuring compatibility with Aide. Contributions are welcome, and the code is dual licensed under MIT and Apache-2.0. If Aide does not meet your requirements, you can explore similar libraries like paperclip, utoipa, and okapi.
PowerApps-Samples
PowerApps-Samples is a repository containing sample code for Power Apps, covering various aspects such as Dataverse, model-driven apps, canvas apps, Power Apps component framework, portals, and AI Builder. It serves as a valuable resource for developers looking to explore and learn about different functionalities within Power Apps ecosystem.
MoonshotAI-Cookbook
The MoonshotAI-Cookbook provides example code and guides for accomplishing common tasks with the MoonshotAI API. To run these examples, you'll need an MoonshotAI account and associated API key. Most code examples are written in Python, though the concepts can be applied in any language.
hoarder
A self-hostable bookmark-everything app with a touch of AI for data hoarders. Features include bookmarking links, taking notes, storing images, automatic fetching for link details, full-text search, AI-based automatic tagging, Chrome and Firefox plugins, iOS and Android apps, dark mode support, and self-hosting. Built to address the need for archiving and previewing links with automatic tagging. Developed by a systems engineer to stay connected with web development and cater to personal use cases.
recognizer
Recognizer is a Python library for speech recognition. It provides a simple interface to transcribe speech from audio files or live audio input. The library supports multiple speech recognition engines, including Google Speech Recognition, Sphinx, and Wit.ai. Recognizer is easy to use and can be integrated into various applications to enable voice commands, transcription, and speech-to-text functionality.
For similar tasks
prompt-generator-comfyui
Custom AI prompt generator node for ComfyUI. With this node, you can use text generation models to generate prompts. Before using, text generation model has to be trained with prompt dataset.
lumentis
Lumentis is a tool that allows users to generate beautiful and comprehensive documentation from meeting transcripts and large documents with a single command. It reads transcripts, asks questions to understand themes and audience, generates an outline, and creates detailed pages with visual variety and styles. Users can switch models for different tasks, control the process, and deploy the generated docs to Vercel. The tool is designed to be open, clean, fast, and easy to use, with upcoming features including folders, PDFs, auto-transcription, website scraping, scientific papers handling, summarization, and continuous updates.
khoj
Khoj is an open-source, personal AI assistant that extends your capabilities by creating always-available AI agents. You can share your notes and documents to extend your digital brain, and your AI agents have access to the internet, allowing you to incorporate real-time information. Khoj is accessible on Desktop, Emacs, Obsidian, Web, and Whatsapp, and you can share PDF, markdown, org-mode, notion files, and GitHub repositories. You'll get fast, accurate semantic search on top of your docs, and your agents can create deeply personal images and understand your speech. Khoj is self-hostable and always will be.
quivr
Quivr is a personal assistant powered by Generative AI, designed to be a second brain for users. It offers fast and efficient access to data, ensuring security and compatibility with various file formats. Quivr is open source and free to use, allowing users to share their brains publicly or keep them private. The marketplace feature enables users to share and utilize brains created by others, boosting productivity. Quivr's offline mode provides anytime, anywhere access to data. Key features include speed, security, OS compatibility, file compatibility, open source nature, public/private sharing options, a marketplace, and offline mode.
SillyTavern
SillyTavern is a user interface you can install on your computer (and Android phones) that allows you to interact with text generation AIs and chat/roleplay with characters you or the community create. SillyTavern is a fork of TavernAI 1.2.8 which is under more active development and has added many major features. At this point, they can be thought of as completely independent programs.
wingman-ai
Wingman-AI is a free and open-source AI coding assistant that brings high-quality AI-assisted coding right to your computer. It offers features such as code completion, interactive chat, and support for multiple AI providers, including Ollama, Hugging Face, and OpenAI. Wingman-AI is designed to enhance your coding workflow by providing real-time assistance and suggestions, making it an ideal tool for developers of all levels.
morphic
Morphic is an AI-powered answer engine with a generative UI. It utilizes a stack of Next.js, Vercel AI SDK, OpenAI, Tavily AI, shadcn/ui, Radix UI, and Tailwind CSS. To get started, fork and clone the repo, install dependencies, fill out secrets in the .env.local file, and run the app locally using 'bun dev'. You can also deploy your own live version of Morphic with Vercel. Verified models that can be specified to writers include Groq, LLaMA3 8b, and LLaMA3 70b.
gpt-engineer
GPT-Engineer is a tool that allows you to specify a software in natural language, sit back and watch as an AI writes and executes the code, and ask the AI to implement improvements.
For similar jobs
ChatFAQ
ChatFAQ is an open-source comprehensive platform for creating a wide variety of chatbots: generic ones, business-trained, or even capable of redirecting requests to human operators. It includes a specialized NLP/NLG engine based on a RAG architecture and customized chat widgets, ensuring a tailored experience for users and avoiding vendor lock-in.
anything-llm
AnythingLLM is a full-stack application that enables you to turn any document, resource, or piece of content into context that any LLM can use as references during chatting. This application allows you to pick and choose which LLM or Vector Database you want to use as well as supporting multi-user management and permissions.
ai-guide
This guide is dedicated to Large Language Models (LLMs) that you can run on your home computer. It assumes your PC is a lower-end, non-gaming setup.
classifai
Supercharge WordPress Content Workflows and Engagement with Artificial Intelligence. Tap into leading cloud-based services like OpenAI, Microsoft Azure AI, Google Gemini and IBM Watson to augment your WordPress-powered websites. Publish content faster while improving SEO performance and increasing audience engagement. ClassifAI integrates Artificial Intelligence and Machine Learning technologies to lighten your workload and eliminate tedious tasks, giving you more time to create original content that matters.
mikupad
mikupad is a lightweight and efficient language model front-end powered by ReactJS, all packed into a single HTML file. Inspired by the likes of NovelAI, it provides a simple yet powerful interface for generating text with the help of various backends.
glide
Glide is a cloud-native LLM gateway that provides a unified REST API for accessing various large language models (LLMs) from different providers. It handles LLMOps tasks such as model failover, caching, key management, and more, making it easy to integrate LLMs into applications. Glide supports popular LLM providers like OpenAI, Anthropic, Azure OpenAI, AWS Bedrock (Titan), Cohere, Google Gemini, OctoML, and Ollama. It offers high availability, performance, and observability, and provides SDKs for Python and NodeJS to simplify integration.
onnxruntime-genai
ONNX Runtime Generative AI is a library that provides the generative AI loop for ONNX models, including inference with ONNX Runtime, logits processing, search and sampling, and KV cache management. Users can call a high level `generate()` method, or run each iteration of the model in a loop. It supports greedy/beam search and TopP, TopK sampling to generate token sequences, has built in logits processing like repetition penalties, and allows for easy custom scoring.
firecrawl
Firecrawl is an API service that takes a URL, crawls it, and converts it into clean markdown. It crawls all accessible subpages and provides clean markdown for each, without requiring a sitemap. The API is easy to use and can be self-hosted. It also integrates with Langchain and Llama Index. The Python SDK makes it easy to crawl and scrape websites in Python code.