
wiseflow
Use LLMs to dig out what you care about from massive amounts of information and a variety of sources daily.
Stars: 7786

Wiseflow is an agile information mining tool that utilizes the thinking and analysis capabilities of large models to accurately extract specific information from various given sources, without the need for manual intervention. The tool focuses on filtering noise from a vast amount of information to reveal valuable insights. It is recommended to use normal language models for information extraction tasks to optimize speed and cost, rather than complex reasoning models. The tool is designed for continuous information gathering based on specified focus points from various sources.
README:
English | 日本語 | 한국어 | Deutsch | Français | العربية
🚀 使用大模型从海量信息、各类信源中每日挖掘你真正感兴趣的要点!
我们缺的不是信息,而是从海量信息中过滤噪音,从而让有价值的信息显露出来
https://github.com/user-attachments/assets/48998353-6c6c-4f8f-acae-dc5c45e2e0e6
4.2版本在4.0、4.1版本基础上重点强化了网页抓取能力,现在程序可以直接调用您本地“真正”的 Chrome 浏览器进行获取。这不仅最大化降低了被目标站点“风控”的概率,而且还带来了可以持久化用户数据、支持页面操作脚本等新特性!(比如部分网站需要用户登录后才能展示完整内容,您现在可以预先登录,然后再使用 wiseflow 获取完整内容)。
因为4.2版本直接使用您本地的 Chrome 浏览器进行抓取,所以现在部署时无需执行 python -m playwright install --with-deps chromium
了,但需要按默认安装路径安装 google chrome 浏览器
除此之外,我们还重构了搜索引擎方案,以及提供了完备的 proxy 方案。具体见 CHANGELOG
4.1版本支持为关注点精准配置搜索源,目前支持 bing、github 和 arxiv三个搜索源,且均使用平台原生接口,无需额外申请第三方服务。
4.1版本支持为 focuspoint 设定角色和目的,从而指导 LLM 以特定视角或目的进行分析和提取。但使用时请注意:
- 如果关注点本身指向性很具体,那么角色和目的的设定对结果影响不大;
- 影响最终结果质量的第一要素永远是信源,一定要提供与关注点高度相关的信源。
有关角色和目的设定对提取结果影响的测评案例,请参考 task1
现在你可以在 pb 界面下创建自己的表单,并配置给特定的关注点,LLM 将按照表单字段进行精准提取。
现在可以指定程序按关注点在社交平台上查找相关内容,并进一步查找内容的创作者主页信息。结合"自定义提取模式",wiseflow可以帮助你在全网搜索潜在客户、合作伙伴或者投资人的联系方式。
“在 LLM 时代,优秀的开发者应该把至少60%的时间花在选择合适的 LLM 模型上”
我们精选了7套来自真实项目的测试样本,并广泛选择了主流的且输出价格不超过 ¥4/M tokens 的模型,进行了详细的wiseflow info extracting任务测试, 得出了如下使用推荐:
- 性能优先场景下,推荐使用:ByteDance-Seed/Seed-OSS-36B-Instruct
- 成本优先场景下,依然推荐使用:Qwen/Qwen3-14B
视觉辅助分析模型,依然可以使用:/Qwen/Qwen2.5-VL-7B-Instruct (wiseflow 任务目前对此的依赖不高)
详细的测试报告,可见 LLM USE TEST
需要说明的是,以上测试结果仅代表模型在 wiseflow 信息提取任务上的表现,不能代表模型的综合能力和全面能力。wiseflow 信息提取任务与其他类型任务(如规划、写作等)可能存在明显不同,另外成本是我们重点考虑的因素之一,因为 wiseflow 任务对模型的使用量消耗会比较大,尤其是在多信源、多关注点情况下。
wiseflow 不限定模型服务提供商,只要兼容 openaiSDK 请求接口格式即可。您可以选择已有的 Maas 服务或者 Ollama 等本地部署模型服务。
对于中国大陆区域用户,我们推荐使用 Siliconflow 的模型服务
🌹 欢迎使用我的 推荐链接 申请,你我都会获赠¥14平台奖励
另外,如果您对 openai 系列模型更加青睐的话,'o3-mini' 和 'openai/gpt-oss-20b' 也是不错的选择,视觉辅助分析可以搭配 gpt-4o-mini。
💰 目前在 wiseflow 应用内可以官方价格的九折使用由 AiHubMix 转发的 openai 系列模型官方接口。
注意: 享受优惠需要切换至 aihubmix 分支,详见 README
我把 wiseflow 的产品定位称为"wide search", 这是相对于目前大火的"deep search"而言。
具体而言"deep search"是面向某一具体问题由 llm 自主动态规划搜索路径,持续探索不同页面,采集到足够的信息后给出答案或者产出报告等;但是有的时候,我们并不带着具体的问题进行搜索,也并不需要深入探索,只需要广泛的信息采集(比如行业情报搜集、对象背景信息搜集、客户信息采集等),这个时候广度明显更有意义。虽然使用"deep search"也能实现这个任务,但那是大炮打蚊子,低效率高成本,而 wiseflow 就是专为这种"wide search"场景打造的利器。
- 全平台的获取能力,包括网页、社交媒体(目前提供对微博和快手平台的支持)、RSS 信源、bing、github、arxiv等;
- 独特的 html 处理流程,自动按关注点提取信息并发现值得进一步探索的链接,且仅需 14b 参数量的大模型即可很好的工作;
- 爬查一体”策略,爬取过程中 LLM 即介入,只抓取与关注点相关的信息,有效降低平台风控概率;
- 面向普通用户(而非开发者),无需人工介入提供 Xpath,"开箱即用";
- 持续迭代带来的高稳定性和高可用性,以及兼顾系统资源和速度的处理效率;
- 将不仅仅是“爬虫”……
(4.x 架构整体规划图。虚线框内为尚未完成的部分,希望有能力的社区开发者加入我们,贡献 PR。 所有contributor都将免费获赠 pro 版本的使用权!)
只需三步即可开始使用!
4.2版本起,必须先安装 google chrome 浏览器(使用默认安装路径)
windows 用户请提前下载 git bash 工具,并在 bash 中执行如下命令 bash下载链接
curl -LsSf https://astral.sh/uv/install.sh | sh
git clone https://github.com/TeamWiseFlow/wiseflow.git
上述操作会完成 uv 的安装。
接下来去 pocketbase docs 下载对应自己系统的 pocketbase 程序放置于 .pb 文件夹下
也可以尝试使用 install_pocketbase.sh (for MacOS/Linux) 或 install_pocketbase.ps1 (for Windows) 来安装。
在 wiseflow 文件夹(项目根目录)参考 env_sample 创建 .env 文件,并填入相关设定信息。
4.x 版本无需用户在.env 中提供 pocketbase 的账密,也不限定 pocketbase 的版本, 同时我们也暂时取消了 Secondary Model 的设定, 因此你其实最少仅需四个参数即可完成配置:
- LLM_API_KEY="" # LLM 服务的 key (任何提供 OpenAI 格式 API 的模型服务商均可,本地使用 ollama 部署则无需设置)
- LLM_API_BASE="https://api.siliconflow.cn/v1" # LLM 服务接口地址(推荐使用siliconflow服务, 欢迎使用我的 推荐链接 申请,你我都会获赠¥14平台奖励)
- PRIMARY_MODEL=ByteDance-Seed/Seed-OSS-36B-Instruct # 价格敏感且提取不复杂的场景可以使用 Qwen3-14B
- VL_MODEL=Pro/Qwen/Qwen2.5-VL-7B-Instruct
cd wiseflow
uv venv # 仅第一次执行需要
source .venv/bin/activate # Linux/macOS
# 或者在 Windows 上:
# .venv\Scripts\activate
uv sync # 仅第一次执行需要
chmod +x run.sh # 仅第一次执行需要
./run.sh
详细使用教程请参考 docs/manual/manual.md
wiseflow 所有抓取数据都会即时存入 pocketbase,因此您可以直接操作 pocketbase 数据库来获取数据。
PocketBase作为流行的轻量级数据库,目前已有 Go/Javascript/Python 等语言的SDK。
欢迎在如下 repo 中分享并推广您的二次开发应用案例!
自4.2版本起,我们更新了开源许可协议,敬请查阅: LICENSE
商用合作,请联系 Email:[email protected]
有任何问题或建议,欢迎通过 issue 留言。
- Crawl4ai(Open-source LLM Friendly Web Crawler & Scraper) https://github.com/unclecode/crawl4ai
- Patchright(Undetected Python version of the Playwright testing and automation library) https://github.com/Kaliiiiiiiiii-Vinyzu/patchright-python
- MediaCrawler(xhs/dy/wb/ks/bilibili/zhihu crawler) https://github.com/NanmiCoder/MediaCrawler
- NoDriver(Providing a blazing fast framework for web automation, webscraping, bots and any other creative ideas...) https://github.com/ultrafunkamsterdam/nodriver
- Pocketbase(Open Source realtime backend in 1 file) https://github.com/pocketbase/pocketbase
- Feedparser(Parse feeds in Python) https://github.com/kurtmckee/feedparser
- SearXNG(a free internet metasearch engine which aggregates results from various search services and databases) https://github.com/searxng/searxng
如果您在相关工作中参考或引用了本项目的部分或全部,请注明如下信息:
Author:Wiseflow Team
https://github.com/TeamWiseFlow/wiseflow
For Tasks:
Click tags to check more tools for each tasksFor Jobs:
Alternative AI tools for wiseflow
Similar Open Source Tools

wiseflow
Wiseflow is an agile information mining tool that utilizes the thinking and analysis capabilities of large models to accurately extract specific information from various given sources, without the need for manual intervention. The tool focuses on filtering noise from a vast amount of information to reveal valuable insights. It is recommended to use normal language models for information extraction tasks to optimize speed and cost, rather than complex reasoning models. The tool is designed for continuous information gathering based on specified focus points from various sources.

local-deep-research
Local Deep Research is a powerful AI-powered research assistant that performs deep, iterative analysis using multiple LLMs and web searches. It can be run locally for privacy or configured to use cloud-based LLMs for enhanced capabilities. The tool offers advanced research capabilities, flexible LLM support, rich output options, privacy-focused operation, enhanced search integration, and academic & scientific integration. It also provides a web interface, command line interface, and supports multiple LLM providers and search engines. Users can configure AI models, search engines, and research parameters for customized research experiences.

mushroom
MRCMS is a Java-based content management system that uses data model + template + plugin implementation, providing built-in article model publishing functionality. The goal is to quickly build small to medium websites.

Awesome-Lists
Awesome-Lists is a curated list of awesome lists across various domains of computer science and beyond, including programming languages, web development, data science, and more. It provides a comprehensive index of articles, books, courses, open source projects, and other resources. The lists are organized by topic and subtopic, making it easy to find the information you need. Awesome-Lists is a valuable resource for anyone looking to learn more about a particular topic or to stay up-to-date on the latest developments in the field.

robustmq
RobustMQ is a next-generation, high-performance, multi-protocol message queue built in Rust. It aims to create a unified messaging infrastructure tailored for modern cloud-native and AI systems. With features like high performance, distributed architecture, multi-protocol support, pluggable storage, cloud-native readiness, multi-tenancy, security features, observability, and user-friendliness, RobustMQ is designed to be production-ready and become a top-level Apache project in the message queue ecosystem by the second half of 2025.

kweaver
KWeaver is an open-source cognitive intelligence development framework that provides data scientists, application developers, and domain experts with the ability for rapid development, comprehensive openness, and high-performance knowledge network generation and cognitive intelligence large model framework. It offers features such as automated and visual knowledge graph construction, visualization and analysis of knowledge graph data, knowledge graph integration, knowledge graph resource management, large model prompt engineering and debugging, and visual configuration for large model access.

R2R
R2R (RAG to Riches) is a fast and efficient framework for serving high-quality Retrieval-Augmented Generation (RAG) to end users. The framework is designed with customizable pipelines and a feature-rich FastAPI implementation, enabling developers to quickly deploy and scale RAG-based applications. R2R was conceived to bridge the gap between local LLM experimentation and scalable production solutions. **R2R is to LangChain/LlamaIndex what NextJS is to React**. A JavaScript client for R2R deployments can be found here. ### Key Features * **🚀 Deploy** : Instantly launch production-ready RAG pipelines with streaming capabilities. * **🧩 Customize** : Tailor your pipeline with intuitive configuration files. * **🔌 Extend** : Enhance your pipeline with custom code integrations. * **⚖️ Autoscale** : Scale your pipeline effortlessly in the cloud using SciPhi. * **🤖 OSS** : Benefit from a framework developed by the open-source community, designed to simplify RAG deployment.

AI-Translation-Assistant-Pro
AI Translation Assistant Pro is a powerful AI-driven platform for multilingual translation and content processing. It offers features such as text translation, image recognition, PDF processing, speech recognition, and video processing. The platform includes a subscription system with different membership levels, user management functionalities, quota management, and real-time usage statistics. It utilizes technologies like Next.js, React, TypeScript for the frontend, Node.js, PostgreSQL for the backend, NextAuth.js for authentication, Stripe for payments, and integrates with cloud services like Aliyun OSS and Tencent Cloud for AI services.

aippt_PresentationGen
A SpringBoot web application that generates PPT files using a llm. The tool preprocesses single-page templates and dynamically combines them to generate PPTX files with text replacement functionality. It utilizes technologies such as SpringBoot, MyBatis, MySQL, Redis, WebFlux, Apache POI, Aspose Slides, OSS, and Vue2. Users can deploy the tool by configuring various parameters in the application.yml file and setting up necessary resources like MySQL, OSS, and API keys. The tool also supports integration with open-source image libraries like Unsplash for adding images to the presentations.

monoscope
Monoscope is an open-source monitoring and observability platform that uses artificial intelligence to understand and monitor systems automatically. It allows users to ingest and explore logs, traces, and metrics in S3 buckets, query in natural language via LLMs, and create AI agents to detect anomalies. Key capabilities include universal data ingestion, AI-powered understanding, natural language interface, cost-effective storage, and zero configuration. Monoscope is designed to reduce alert fatigue, catch issues before they impact users, and provide visibility across complex systems.

finite-monkey-engine
FiniteMonkey is an advanced vulnerability mining engine powered purely by GPT, requiring no prior knowledge base or fine-tuning. Its effectiveness significantly surpasses most current related research approaches. The tool is task-driven, prompt-driven, and focuses on prompt design, leveraging 'deception' and hallucination as key mechanics. It has helped identify vulnerabilities worth over $60,000 in bounties. The tool requires PostgreSQL database, OpenAI API access, and Python environment for setup. It supports various languages like Solidity, Rust, Python, Move, Cairo, Tact, Func, Java, and Fake Solidity for scanning. FiniteMonkey is best suited for logic vulnerability mining in real projects, not recommended for academic vulnerability testing. GPT-4-turbo is recommended for optimal results with an average scan time of 2-3 hours for medium projects. The tool provides detailed scanning results guide and implementation tips for users.

bifrost
Bifrost is a high-performance AI gateway that unifies access to multiple providers through a single OpenAI-compatible API. It offers features like automatic failover, load balancing, semantic caching, and enterprise-grade functionalities. Users can deploy Bifrost in seconds with zero configuration, benefiting from its core infrastructure, advanced features, enterprise and security capabilities, and developer experience. The repository structure is modular, allowing for maximum flexibility. Bifrost is designed for quick setup, easy configuration, and seamless integration with various AI models and tools.

neuropilot
NeuroPilot is an open-source AI-powered education platform that transforms study materials into interactive learning resources. It provides tools like contextual chat, smart notes, flashcards, quizzes, and AI podcasts. Supported by various AI models and embedding providers, it offers features like WebSocket streaming, JSON or vector database support, file-based storage, and configurable multi-provider setup for LLMs and TTS engines. The technology stack includes Node.js, TypeScript, Vite, React, TailwindCSS, JSON database, multiple LLM providers, and Docker for deployment. Users can contribute to the project by integrating AI models, adding mobile app support, improving performance, enhancing accessibility features, and creating documentation and tutorials.

inspector
A developer tool for testing and debugging Model Context Protocol (MCP) servers. It allows users to test the compliance of their MCP servers with the latest MCP specs, supports various transports like STDIO, SSE, and Streamable HTTP, features an LLM Playground for testing server behavior against different models, provides comprehensive logging and error reporting for MCP server development, and offers a modern developer experience with multiple server connections and saved configurations. The tool is built using Next.js and integrates MCP capabilities, AI SDKs from OpenAI, Anthropic, and Ollama, and various technologies like Node.js, TypeScript, and Next.js.

commands
Production-ready slash commands for Claude Code that accelerate development through intelligent automation and multi-agent orchestration. Contains 52 commands organized into workflows and tools categories. Workflows orchestrate complex tasks with multiple agents, while tools provide focused functionality for specific development tasks. Commands can be used with prefixes for organization or flattened for convenience. Best practices include using workflows for complex tasks and tools for specific scopes, chaining commands strategically, and providing detailed context for effective usage.

genkit-plugins
Community plugins repository for Google Firebase Genkit, containing various plugins for AI APIs and Vector Stores. Developed by The Fire Company, this repository offers plugins like genkitx-anthropic, genkitx-cohere, genkitx-groq, genkitx-mistral, genkitx-openai, genkitx-convex, and genkitx-hnsw. Users can easily install and use these plugins in their projects, with examples provided in the documentation. The repository also showcases products like Fireview and Giftit built using these plugins, and welcomes contributions from the community.
For similar tasks

Awesome-Segment-Anything
Awesome-Segment-Anything is a powerful tool for segmenting and extracting information from various types of data. It provides a user-friendly interface to easily define segmentation rules and apply them to text, images, and other data formats. The tool supports both supervised and unsupervised segmentation methods, allowing users to customize the segmentation process based on their specific needs. With its versatile functionality and intuitive design, Awesome-Segment-Anything is ideal for data analysts, researchers, content creators, and anyone looking to efficiently extract valuable insights from complex datasets.

Time-LLM
Time-LLM is a reprogramming framework that repurposes large language models (LLMs) for time series forecasting. It allows users to treat time series analysis as a 'language task' and effectively leverage pre-trained LLMs for forecasting. The framework involves reprogramming time series data into text representations and providing declarative prompts to guide the LLM reasoning process. Time-LLM supports various backbone models such as Llama-7B, GPT-2, and BERT, offering flexibility in model selection. The tool provides a general framework for repurposing language models for time series forecasting tasks.

crewAI
CrewAI is a cutting-edge framework designed to orchestrate role-playing autonomous AI agents. By fostering collaborative intelligence, CrewAI empowers agents to work together seamlessly, tackling complex tasks. It enables AI agents to assume roles, share goals, and operate in a cohesive unit, much like a well-oiled crew. Whether you're building a smart assistant platform, an automated customer service ensemble, or a multi-agent research team, CrewAI provides the backbone for sophisticated multi-agent interactions. With features like role-based agent design, autonomous inter-agent delegation, flexible task management, and support for various LLMs, CrewAI offers a dynamic and adaptable solution for both development and production workflows.

Transformers_And_LLM_Are_What_You_Dont_Need
Transformers_And_LLM_Are_What_You_Dont_Need is a repository that explores the limitations of transformers in time series forecasting. It contains a collection of papers, articles, and theses discussing the effectiveness of transformers and LLMs in this domain. The repository aims to provide insights into why transformers may not be the best choice for time series forecasting tasks.

pytorch-forecasting
PyTorch Forecasting is a PyTorch-based package for time series forecasting with state-of-the-art network architectures. It offers a high-level API for training networks on pandas data frames and utilizes PyTorch Lightning for scalable training on GPUs and CPUs. The package aims to simplify time series forecasting with neural networks by providing a flexible API for professionals and default settings for beginners. It includes a timeseries dataset class, base model class, multiple neural network architectures, multi-horizon timeseries metrics, and hyperparameter tuning with optuna. PyTorch Forecasting is built on pytorch-lightning for easy training on various hardware configurations.

spider
Spider is a high-performance web crawler and indexer designed to handle data curation workloads efficiently. It offers features such as concurrency, streaming, decentralization, headless Chrome rendering, HTTP proxies, cron jobs, subscriptions, smart mode, blacklisting, whitelisting, budgeting depth, dynamic AI prompt scripting, CSS scraping, and more. Users can easily get started with the Spider Cloud hosted service or set up local installations with spider-cli. The tool supports integration with Node.js and Python for additional flexibility. With a focus on speed and scalability, Spider is ideal for extracting and organizing data from the web.

AI_for_Science_paper_collection
AI for Science paper collection is an initiative by AI for Science Community to collect and categorize papers in AI for Science areas by subjects, years, venues, and keywords. The repository contains `.csv` files with paper lists labeled by keys such as `Title`, `Conference`, `Type`, `Application`, `MLTech`, `OpenReviewLink`. It covers top conferences like ICML, NeurIPS, and ICLR. Volunteers can contribute by updating existing `.csv` files or adding new ones for uncovered conferences/years. The initiative aims to track the increasing trend of AI for Science papers and analyze trends in different applications.

pytorch-forecasting
PyTorch Forecasting is a PyTorch-based package designed for state-of-the-art timeseries forecasting using deep learning architectures. It offers a high-level API and leverages PyTorch Lightning for efficient training on GPU or CPU with automatic logging. The package aims to simplify timeseries forecasting tasks by providing a flexible API for professionals and user-friendly defaults for beginners. It includes features such as a timeseries dataset class for handling data transformations, missing values, and subsampling, various neural network architectures optimized for real-world deployment, multi-horizon timeseries metrics, and hyperparameter tuning with optuna. Built on pytorch-lightning, it supports training on CPUs, single GPUs, and multiple GPUs out-of-the-box.
For similar jobs

wiseflow
Wiseflow is an agile information mining tool that utilizes the thinking and analysis capabilities of large models to accurately extract specific information from various given sources, without the need for manual intervention. The tool focuses on filtering noise from a vast amount of information to reveal valuable insights. It is recommended to use normal language models for information extraction tasks to optimize speed and cost, rather than complex reasoning models. The tool is designed for continuous information gathering based on specified focus points from various sources.

lollms-webui
LoLLMs WebUI (Lord of Large Language Multimodal Systems: One tool to rule them all) is a user-friendly interface to access and utilize various LLM (Large Language Models) and other AI models for a wide range of tasks. With over 500 AI expert conditionings across diverse domains and more than 2500 fine tuned models over multiple domains, LoLLMs WebUI provides an immediate resource for any problem, from car repair to coding assistance, legal matters, medical diagnosis, entertainment, and more. The easy-to-use UI with light and dark mode options, integration with GitHub repository, support for different personalities, and features like thumb up/down rating, copy, edit, and remove messages, local database storage, search, export, and delete multiple discussions, make LoLLMs WebUI a powerful and versatile tool.

Azure-Analytics-and-AI-Engagement
The Azure-Analytics-and-AI-Engagement repository provides packaged Industry Scenario DREAM Demos with ARM templates (Containing a demo web application, Power BI reports, Synapse resources, AML Notebooks etc.) that can be deployed in a customer’s subscription using the CAPE tool within a matter of few hours. Partners can also deploy DREAM Demos in their own subscriptions using DPoC.

minio
MinIO is a High Performance Object Storage released under GNU Affero General Public License v3.0. It is API compatible with Amazon S3 cloud storage service. Use MinIO to build high performance infrastructure for machine learning, analytics and application data workloads.

mage-ai
Mage is an open-source data pipeline tool for transforming and integrating data. It offers an easy developer experience, engineering best practices built-in, and data as a first-class citizen. Mage makes it easy to build, preview, and launch data pipelines, and provides observability and scaling capabilities. It supports data integrations, streaming pipelines, and dbt integration.

AiTreasureBox
AiTreasureBox is a versatile AI tool that provides a collection of pre-trained models and algorithms for various machine learning tasks. It simplifies the process of implementing AI solutions by offering ready-to-use components that can be easily integrated into projects. With AiTreasureBox, users can quickly prototype and deploy AI applications without the need for extensive knowledge in machine learning or deep learning. The tool covers a wide range of tasks such as image classification, text generation, sentiment analysis, object detection, and more. It is designed to be user-friendly and accessible to both beginners and experienced developers, making AI development more efficient and accessible to a wider audience.

tidb
TiDB is an open-source distributed SQL database that supports Hybrid Transactional and Analytical Processing (HTAP) workloads. It is MySQL compatible and features horizontal scalability, strong consistency, and high availability.

airbyte
Airbyte is an open-source data integration platform that makes it easy to move data from any source to any destination. With Airbyte, you can build and manage data pipelines without writing any code. Airbyte provides a library of pre-built connectors that make it easy to connect to popular data sources and destinations. You can also create your own connectors using Airbyte's no-code Connector Builder or low-code CDK. Airbyte is used by data engineers and analysts at companies of all sizes to build and manage their data pipelines.