DataDreamer

DataDreamer

DataDreamer: Prompt. Generate Synthetic Data. Train & Align Models.   🤖💤

Stars: 751

Visit
 screenshot

DataDreamer is a powerful open-source Python library designed for prompting, synthetic data generation, and training workflows. It is simple, efficient, and research-grade, allowing users to create prompting workflows, generate synthetic datasets, and train models with ease. The library is built for researchers, by researchers, focusing on correctness, best practices, and reproducibility. It offers features like aggressive caching, resumability, support for bleeding-edge techniques, and easy sharing of datasets and models. DataDreamer enables users to run multi-step prompting workflows, generate synthetic datasets for various tasks, and train models by aligning, fine-tuning, instruction-tuning, and distilling them using existing or synthetic data.

README:

DataDreamer
https://datadreamer.dev

Prompt. Generate Synthetic Data. Train & Align Models.

Tests & Release Ruff

DataDreamer is a powerful open-source Python library for prompting, synthetic data generation, and training workflows. It is designed to be simple, extremely efficient, and research-grade.

Installation

pip3 install datadreamer.dev
demo.py Result of demo.py
                                                                                                   
demo.py

See the full demo script


                                                                                     
Demo

See the synthetic dataset and the trained model

🚀 For more demonstrations and recipes see the Quick Tour page.

With DataDreamer you can:

  • 💬 Create Prompting Workflows: Create and run multi-step, complex, prompting workflows easily with major open source or API-based LLMs.
  • 📊 Generate Synthetic Datasets: Generate synthetic datasets for novel tasks or augment existing datasets with LLMs.
  • ⚙️ Train Models: Align models. Fine-tune models. Instruction-tune models. Distill models. Train on existing data or synthetic data.
  • ... learn more about what's possible in the Overview Guide

DataDreamer is:

  • 🧩 Simple: Simple and approachable to use with sensible defaults, yet powerful with support for bleeding edge techniques.
  • 🔬 Research-Grade: Built for researchers, by researchers, but accessible to all. A focus on correctness, best practices, and reproducibility.
  • 🏎️ Efficient: Aggressive caching and resumability built-in. Support for techniques like quantization, parameter-efficient training (LoRA), and more.
  • 🔄 Reproducible: Workflows built with DataDreamer are easily shareable, reproducible, and extendable.
  • 🤝 Makes Sharing Easy: Publishing datasets and models is simple. Automatically generate data cards and model cards with metadata. Generate a list of any citations required.
  • ... learn more about the motivation and design principles behind DataDreamer.

Citation

Please cite the DataDreamer paper:

@misc{patel2024datadreamer,
      title={DataDreamer: A Tool for Synthetic Data Generation and Reproducible LLM Workflows}, 
      author={Ajay Patel and Colin Raffel and Chris Callison-Burch},
      year={2024},
      eprint={2402.10379},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}

Contact

Please reach out to us via email ([email protected]) or on Discord if you have any questions, comments, or feedback.



Copyright © 2024, Ajay Patel. Released under the MIT License.

Thank you to the maintainers at Hugging Face and LiteLLM for accepting contributions necessary for DataDreamer and providing upstream support.


Funding Acknowledgements

ODNI, IARPA: This research is supported in part by the Office of the Director of National Intelligence (ODNI), Intelligence Advanced Research Projects Activity (IARPA), via the HIATUS Program contract #2022-22072200005. The views and conclusions contained herein are those of the authors and should not be interpreted as necessarily representing the official policies, either expressed or implied, of ODNI, IARPA, or the U.S. Government. The U.S. Government is authorized to reproduce and distribute reprints for governmental purposes notwithstanding any copyright annotation therein.

For Tasks:

Click tags to check more tools for each tasks

For Jobs:

Alternative AI tools for DataDreamer

Similar Open Source Tools

For similar tasks

For similar jobs