OneKE

🌐Web • 📹Video

• Table of Contents
• 🔔News
• 🌟Overview
• 🚀Quick Start
  • Step1: Environment Setup
    • 🔩Manual Environment Configuration
    • 🐳Building With Docker Image
  • Step2: Start with Examples
    • 🖊️Start with YAML
    • 🖊️Start with Python
    • 🖊️Start with Web UI
• 🔍Further Usage
  • 💡Extraction Task Support
    • 1. Named Entity Recognition
    • 2. Relation Extraction
    • 3. Event Extraction
    • 4. Triple Extraction
      • Knowledge Graph
    • 5. Open Domain IE
  • 💡Data Source Support
  • 💡Extraction Model Support
  • 💡Extraction Method Support
  • 💡Knowledge Base Configuration
    • 1. Schema Repository
    • 2. Case Repository
• 🛠️Network Issue Solutions
• 🎉Contributors
• 🌻Acknowledgement

• [2025/02] We support the local deployment of the DeepSeek-R1 series in addition to the existing API service, as well as vllm acceleration for other LLMs.
• [2025/01] OneKE is accepted by WWW 2025 Demonstration Track 🎉🎉🎉.
• [2024/12] We open source the OneKE framework, supporting multi-agent knowledge extraction across various scenarios.
• [2024/04] We release a new bilingual (Chinese and English) schema-based information extraction model called OneKE based on Chinese-Alpaca-2-13B.

OneKE is a flexible dockerized system for schema-guided knowledge extraction, capable of extracting information from the web and raw PDF books across multiple domains like science and news. It employs a collaborative multi-agent approach and includes a user-customizable knowledge base to enable tailored extraction. Embark on your information extraction journey with OneKE!

OneKE currently offers the following features:

• [x] Various IE Tasks Support
• [x] Various Data Sources Support
• [x] Various LLMs Support
• [x] Various Extraction Method Support
• [x] User-Configurable Knowledge Base

We have developed a webpage demo for OneKE with Gradio, click here try information extraction in an intuitive way.

Note: The demo only displays OneKE's basic capabilities for efficiency. Consider the local deployment steps below for further features.

OneKE supports both manual and docker image environment configuration, choose your preferred method to build.

Conda virtual environments offer a light and flexible setup.

Prerequisites

• Anaconda Installation
• GPU support (recommended CUDA version: 12.4)

Configure Steps

1. Clone the repository:

git clone https://github.com/zjunlp/OneKE.git

2. Enter the working directory, and all subsequent commands should be executed in this directory.

cd OneKE

3. Create a virtual environment using Anaconda.

conda create -n oneke python=3.9
conda activate oneke

4. Install all required Python packages.

pip install -r requirements.txt
# If you encounter network issues, consider setting up a domestic mirror for pip.

Docker image provides greater reliability and stability.

Prerequisites

• Docker Installation
• NVIDIA Container Toolkit
• GPU support (recommended CUDA version: 12.4)

Configure Steps

1. Clone the repository:

git clone https://github.com/zjunlp/OneKE.git

2. Pull the docker image from the mirror repository.

docker pull zjunlp/oneke:v4
# If you encounter network issues, consider setting up domestic registry mirrors for docker.

3. Launch a container from the image.

docker run --gpus all \
  -v ./OneKE:/app/OneKE \
  -it oneke:v4 /bin/bash

If using locally deployed models, ensure the local model path is mapped to the container:

docker run --gpus all \
  -v ./OneKE:/app/OneKE \
  -v your_local_model_path:/app/model/your_model_name \
  -it oneke:v4 /bin/bash

Map any necessary local files to the container paths as shown above, and use container paths in your code and execution.

Upon starting, the container will enter the /app/OneKE directory as its working directory. Just modify the code locally as needed, and the changes will sync to the container through mapping.

We offer three quick-start options. Choose your preferred method to swiftly explore OneKE with predefined examples.

Note:

• Ensure that your working directory is set to the OneKE folder, whether in a virtual environment or a docker container.
• Refer to here to resolve the network issues. If you have more questions, feel free to open an issue with us.

Step1: Prepare the configuration file

Several YAML configuration files are available in the examples/config. These extraction scenarios cover different extraction data, methods, and models, allowing you to easily explore all the features of OneKE.

Web News Extraction:

Here is the example for the web news knowledge extraction scenario, with the source extraction text in HTML format:

# model configuration
model:
  category: DeepSeek  # model category, chosen from ChatGPT, DeepSeek, LLaMA, Qwen, ChatGLM, MiniCPM, OneKE.
  model_name_or_path: deepseek-chat # model name, chosen from deepseek-chat and deepseek-reasoner. Choose deepseek-chat to use DeepSeek-V3 or choose deepseek-reasoner to use DeepSeek-R1.
  api_key: your_api_key # your API key for the model with API service. No need for open-source models.
  base_url: https://api.deepseek.com # base URL for the API service. No need for open-source models.

# extraction configuration
extraction:
  task: Base # task type, chosen from Base, NER, RE, EE.
  instruction: Extract key information from the given text. # description for the task. No need for NER, RE, EE task.
  use_file: true # whether to use a file for the input text. Default set to false.
  file_path: ./data/input_files/Tulsi_Gabbard_News.html # path to the input file. No need if use_file is set to false.
  output_schema: NewsReport # output schema for the extraction task. Selected the from schema repository.
  mode: customized # extraction mode, chosen from quick, detailed, customized. Default set to quick. See src/config.yaml for more details.
  update_case: false # whether to update the case repository. Default set to false.
  show_trajectory: false # whether to display the extracted intermediate steps

Book News Extraction:

Here is the example for the book news extraction scenario, with the source extraction text in PDF format:

model:
  # Recommend using ChatGPT or DeepSeek APIs for complex IE task.
  category: ChatGPT # model category, chosen from ChatGPT, DeepSeek, LLaMA, Qwen, ChatGLM, MiniCPM, OneKE.
  model_name_or_path: gpt-4o-mini # model name, chosen from the model list of the selected category.
  api_key: your_api_key # your API key for the model with API service. No need for open-source models.
  base_url: https://api.openai.com/v1 # # base URL for the API service. No need for open-source models.

extraction:
  task: Base # task type, chosen from Base, NER, RE, EE.
  instruction: Extract main characters and background setting from this chapter. # description for the task. No need for NER, RE, EE task.
  use_file: true # whether to use a file for the input text. Default set to false.
  file_path: ./data/input_files/Harry_Potter_Chapter1.pdf #  # path to the input file. No need if use_file is set to false.
  mode: quick # extraction mode, chosen from quick, detailed, customized. Default set to quick. See src/config.yaml for more details.
  update_case: false # whether to update the case repository. Default set to false.
  show_trajectory: false # whether to display the extracted intermediate steps

The model section contains information about the extraction model, while the extraction section configures the settings for the extraction process.

You can choose an existing configuration file or customize the extraction settings as you wish. Note that when using an API service like ChatGPT and DeepSeek, please set your API key.

Step2: Run the shell script

Specify the configuration file path and run the code to start the extraction process.

config_file=your_yaml_file_path # configuration file path, use the container path if inside a container
python src/run.py --config $config_file # start extraction, executed in the OneKE directory

If you want to deploy the local models using vllm, run the following code:

config_file=your_yaml_file_path # REMEMBER to set vllm_serve to TRUE!
python src/models/vllm_serve.py --config $config_file # deploy local model via vllm, executed in the OneKE directory
python src/run.py --config $config_file # start extraction, executed in the OneKE directory

Refer to here to get an overview of the knowledge extraction results.

You can also try OneKE by directly running the example.py file located in the example directory. Specifically, execute the following commands:

python examples/example.py

This will complete a basic NER task, with the extraction results printed upon completion. You can further modify the code in example.py to suit your extraction task setting or to access detailed extraction trajectory.

Named Entity Extraction:

Specifically, we present a NER case in the example.py file:

import sys
sys.path.append("./src")
from models import *
from pipeline import *
import json

# model configuration
model = ChatGPT(model_name_or_path="gpt-4o-mini", api_key="your_api_key")
pipeline = Pipeline(model)

# extraction configuration
Task = "NER"
Text = "Finally , every other year , ELRA organizes a major conference LREC , the International Language Resources and Evaluation Conference."
Constraint = nationality, country capital, place of death, children, location contains, place of birth, place lived, administrative division of country, country of administrative divisions, company, neighborhood of, company founders

# get extraction result
result, trajectory, frontend_schema, frontend_res = pipeline.get_extract_result(task=Task, text=Text, constraint=Constraint)
print("Trajectory:", json.dumps(trajectory, indent=4))

First, select an appropriate extraction model, then complete the configuration of extraction parameters (such as extraction task, extraction text, etc.). Finally, call the get_extract_result function of the Pipeline class to perform information extraction and obtain the final results.

Refer to here to get an overview of the knowledge extraction results.

Note: Before starting with the web UI, make sure the package gradio 4.44.0 is already installed in your Environment.

Step1: Execute Command

Execute the following commands in the OneKE directory:

python src/webui.py

Step2: Open your Web Browser

The front-end is built with Gradio, and the default port of Gradio is 7860. Therefore, please enter the following URL in your browser's address bar to open the web interface:

 http://127.0.0.1:7860

Similarly, you can visually configure tasks and obtain results through the front-end interface.

1. 🎲 Quick Start with an Example 🎲: Quickly get a simple example to try OneKE.
2. Submit: After configuring your LLM, parameters, and tasks, click this button to run OneKE.
3. Clear: When a task is completed, click this button to restore the initial state.

You can try different types of information extraction tasks within the OneKE framework.

In subsequent code processing, we categorize tasks into four types: NER for Named Entity Recognition, RE for Relation Extraction, EE for Event Extraction, Triple for Triple Extraction, and Base for any other user-defined open-domain extraction tasks.

Named entity recognition seeks to locate and classify named entities mentioned in unstructured text into pre-defined entity types such as person names, organizations, locations, organizations, etc.

Refer to the case defined in examples/config/NER.yaml as an example:

In this task setting, Text represents the text to be extracted, while Entity Types denote the constraint on the types of entities to be extracted. Accordingly, we set the text and constraint attributes in the YAML file to their respective values.

Next, follow the steps below to complete the NER task:

• Complete ./examples/config/NER.yaml:

  configure the necessary model and extraction settings.

• Run the shell script below:

  config_file=./examples/config/NER.yaml
  python src/run.py --config $config_file

  ( Refer to issues for any network issues. )

The final extraction result should be:

Click here to obtain the raw results in json format.

Note: The actual extraction results may not exactly match this due to LLM randomness.

The result indicates that, given the text and entity type constraint, entities of type conference have been extracted: ELRA, conference, International Language Resources and Evaluation Conference.

You can either specify entity type constraints or omit them. Without constraints, OneKE will extract all entities from the sentence.

Relationship extraction is the task of extracting semantic relations between entities from a unstructured text.

Refer to the case defined in examples/config/RE.yaml as an example:

In this task setting, Text represents the text to be extracted, while Relation Types denote the constraint on the types of relations of entities to be extracted. Accordingly, we set the text and constraint attributes in the YAML file to their respective values.

Next, follow the steps below to complete the RE task:

• Complete ./examples/config/RE.yaml: configure the necessary model and extraction settings
• Run the shell script below:

  config_file=./examples/config/RE.yaml
  python src/run.py --config $config_file

  ( Refer to issues for any network issues. )

The final extraction result should be:

Click here to obtain the raw results in json format.

Note: The actual extraction results may not exactly match this due to LLM randomness.

The result indicates that, the relation Country-Capital is extracted from the given text based on the relation list, accompanied by the corresponding head entity Guinea and tail entity Conakry, which denotes that Conakry is the capital of Guinea.

You can either specify relation type constraints or omit them. Without constraints, OneKE will extract all relation triples from the sentence.

Event extraction is the task to extract event type, event trigger words, and event arguments from a unstructed text, which is a more complex IE task compared to the first two.

Refer to the case defined in examples/config/EE.yaml as an example:

The extraction text is:

UConn Health , an academic medical center , says in a media statement that it identified approximately 326,000 potentially impacted individuals whose personal information was contained in the compromised email accounts.

while the event type constraint is formatted as follows:

Each event type has its own corresponding event arguments.

Next, follow the steps below to complete the EE task:

• Complete ./examples/config/EE.yaml: configure the necessary model and extraction settings
• Run the shell script below:

  config_file=./examples/config/EE.yaml
  python src/run.py --config $config_file

  ( Refer to issues for any network issues. )

The final extraction result should be:

Click here to obtain the raw results in json format.

Note: The actual extraction results may not exactly match this due to LLM randomness.

The extraction results show that the data breach event is identified using the trigger compromised, and the specific contents of different event arguments such as compromised data and victim have also been extracted.

You can either specify event constraints or omit them. Without constraints, OneKE will extract all events from the sentence.

Triple Extraction identifies subject-predicate-object triples in text. A triple is a fundamental data structure in information extraction, representing a piece of knowledge or a fact. Knowledge Graph (KG) can be quickly constructed after the Triple Extraction.

Here is an example:

The final extraction result should be:

Let's start in OneKE ~

The constraint can be customed as multiple styles, and it's formatted as follows:

• Define entity types only:

  If you only need to specify the entity types, the constraint should be a single list of strings representing the different entity types.

["Person", "Place", "Event", "property"]

• Define entity types and relation types:

  If you need to specify both entity types and relation types, the constraint should be a nested list. The first list contains the entity types, and the second list contains the relation types.

[["Person", "Place", "Event", "property"], ["Interpersonal", "Located", "Ownership", "Action"]]

• Define subject entities types, relation types, and object entities types:

  If you need to define the types of subject entities, relation types, and object entities, the constraint should be a nested list. The first list contains the subject entity types, the second list contains the relation types, and the third list contains the object entity types.

[["Person"], ["Interpersonal", "Ownership"], ["Person", "property"]]

Next, follow the steps below to complete the Triple extraction task:

• Complete ./examples/config/Triple2KG.yaml:

  configure the necessary model and extraction settings.

• Run the shell script below:

  config_file=./examples/config/Triple2KG.yaml
  python src/run.py --config $config_file

  ( Refer to issues for any network issues. )

Here is an example to start. And access a raw results in JSON format here.

⚠️ Warning: If you do not intend to build a Knowledge Graph, make sure to remove or comment out the construct field in the yaml file. This will help avoid errors related to database connection issues.

✨ If you need to construct your Knowledge Graph (KG) with your Triple Extraction result, you can refer to this example for guidance. Mimic this example and add the construct field. Just update the field with your own database parameters.

construct: # (Optional) If you want to construct a Knowledge Graph, you need to set the construct field, or you must delete this field.
  database: Neo4j # your database type.
  url: neo4j://localhost:7687 # your database URL，Neo4j's default port is 7687.
  username: your_username # your database username.
  password: "your_password" # your database password.

Once your database is set up, you can access your graph database through a browser. For Neo4j, the web interface connection URL is usually:

http://localhost:7474/browser

For additional information regarding the Neo4j database, please refer to it's documentation.

⚠️ Warning Again: If you do not intend to build a Knowledge Graph, make sure to remove or comment out the construct field in the yaml file. This will help avoid errors related to database connection issues.

This type of task is represented as Base in the code, signifying any other user-defined open-domain extraction tasks.

We refer to the example above for guidance.

In the context of customized Web News Extraction, we first set the extraction instruction to Extract key information from the given text, and provide the file path to extract content from the file. We specify the output schema from the schema repository as the predefined NewsReport, and then proceed with the extraction.

Next, follow the steps below to complete this task:

• Complete ./examples/config/NewsExtraction.yaml : configure the necessary model and extraction settings
• Run the shell script below:

  config_file=./examples/config/NewsExtraction.yaml
  python src/run.py --config $config_file

  ( Refer to issues for any network issues. )

Here is an excerpt of the extracted content:

Click here to obtain the raw results in json format.

Note: The actual extraction results may not exactly match this due to LLM randomness.

In contrast to eariler tasks, the Base-Type Task requires you to provide an explicit Instruction that clearly defines your extraction task, while not allowing the setting of constraint values.

You can choose source texts of various lengths and forms for extraction.

In practice, you can use the YAML file configuration to handle different types of text input:

• Plain Text: Set use_file to false and enter the text to be extracted in the text field. For example:

  use_file: false
  text: Finally , every other year , ELRA organizes a major conference LREC , the International Language Resources and Evaluation Conference .

• File Content: Set use_file to true and specify the file path in file_path for the text to be extracted. For example:

  use_file: true
  file_path: ./data/input_files/Tulsi_Gabbard_News.html

You can choose from various open-source or proprietary model APIs to perform information extraction tasks.

Note: For complex IE tasks, we recommend using powerful models like OpenAI's or or large-scale open-source LLMs.

Note: We recommend deploying the DeepSeek-R1 models with VLLM.

In practice, you can use the YAML file configuration to employ various LLMs:

• API Service: Set the model_name_or_path to the available model name provided by the company, and enter your api_key as well as the base_url. For exmaple:

  model:
    category: DeepSeek # model category, chosen from ChatGPT and  DeepSeek
    model_name_or_path: deepseek-chat # model name, chosen from deepseek-chat and deepseek-reasoner. Choose deepseek-chat to use DeepSeek-V3 or choose deepseek-reasoner to use DeepSeek-R1.
    api_key: your_api_key # your API key for the model with API service.
    base_url: https://api.deepseek.com # base URL for the API service. No need for open-source models.

• Local Deploy: Set the model_name_or_path to either the model name on Hugging Face or the path to the local model. We support using either Transformer or vllm to access the models.
  • Transformer Example:

    model:
      category: LLaMA # model category, chosen from LLaMA, Qwen, ChatGLM, MiniCPM, OneKE.
      model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct # model name to download from huggingface or use the local model path.
      vllm_serve: false # whether to use the vllm. Default set to false.

    Note that the category of deployment model must be chosen from LLaMA, Qwen, ChatGLM, MiniCPM, OneKE.
  • VLLM Example:

    model:
      category: DeepSeek # model category
      model_name_or_path: meta-llama/Meta-Llama-3-8B-Instruct # model name to download from huggingface or use the local model path.
      vllm_serve: true # whether to use the vllm. Default set to false.

    Note that the DeepSeek-R1 series models only support VLLM deployment. Remember to start the VLLM service before running the extraction task. The reference code is as follows:

    config_file=your_yaml_file_path # REMEMBER to set vllm_serve to TRUE!
    python src/models/vllm_serve.py --config $config_file # deploy local model via vllm, executed in the OneKE directory

    You can also run the command vllm serve model_name_or_path directly to start the VLLM service. See the official documents for more details.

You can freely combine different extraction methods to complete the information extraction task.

The configuration for detail extraction methods and mode information can be found in src/config.yaml. You can customize the extraction methods by modifying the customized within this file and set the mode to customize in an external configuration file.

For example, first configure the src/config.yaml as follows:

# src/config.yaml
customized:
    schema_agent:  get_deduced_schema
    extraction_agent: extract_information_direct
    reflection_agent: reflect_with_case

Then, set the mode of your custom extraction task in examples/customized.yaml to customized:

# examples/customized.yaml
mode: customized

This allows you to experience the customized extraction methods.

Tips:

• For longer text extraction tasks, we recommend using the direct mode to avoid issues like attention dispersion and increased processing time.
• For shorter tasks requiring high accuracy, you can try the standard mode to ensure precision.

You can view the predefined schemas within the src/modules/knowledge_base/schema_repository.py file. The Schema Repository is designed to be easily extendable. You just need to define your output schema in the form of a pydantic class following the format defined in the file, and it can be directly used in subsequent extractions.

For example, add a new schema in the schema repository:

# src/modules/knowledge_base/schema_repository.py
class ChemicalSubstance(BaseModel):
    name: str = Field(description="Name of the chemical substance")
    formula: str = Field(description="Molecular formula")
    appearance: str = Field(description="Physical appearance")
    uses: List[str] = Field(description="Primary uses")
    hazards: str = Field(description="Hazard classification")

class ChemicalList(BaseModel):
  chemicals: List[ChemicalSubstance] = Field(description="List of chemicals")

Then, set the method for schema_agent under customized to get_retrieved_schema in src/config.yaml. Finally, set the mode to customized in the external configuration file to enable custom schema extraction.

In this example, the extraction results will be a list of chemical substances that strictly adhere to the defined schema, ensuring a high level of accuracy and flexibility in the extraction results.

Note that the names of newly created objects should not conflict with existing ones.

You can directly view the case storage in the src/modules/knowledge_base/case_repository.json file, but we do not recommend modifying it directly.

The Case Repository is automatically updated with each extraction process once setting update_repository to True in the configuration file.

When updating the Case Repository, you must provide external feedback to generate case information, either by including truth answer in the configuration file or during the extraction process.

Here is an example:

  # examples/config/RE.yaml
  truth: {"relation_list": [{"head": "Guinea", "tail": "Conakry", "relation": "country capital"}]} # Truth data for the relation
  update_case: true

After extraction, OneKE compares results with the truth answer, generates analysis, and finally stores the case in the repository.

Here are some network issues you might encounter and the corresponding solutions.

• Pip Installation Failure: Use mirror websites, run the command as pip install -i [mirror-source] ....
• Docker Image Pull Failure: Configure the docker daemon to add repository mirrors.
• Nltk Download Failure: Manually download the nltk package and place it in the proper directory.
• Model Dowload Failure: Use the Hugging Face Mirror site or ModelScope to download model, and specify the local path to the model when using it.

  Note: We use all-MiniLM-L6-v2 model by default for case matching, so it needs to be downloaded during execution. If network issues occur, manually download the model, and update the embedding_model to its local path in the src/config.yaml file.

Ningyu Zhang, Haofen Wang, Yujie Luo, Xiangyuan Ru, Kangwei Liu, Lin Yuan, Mengshu Sun, Lei Liang, Zhiqiang Zhang, Jun Zhou, Lanning Wei, Da Zheng, Huajun Chen.

We deeply appreciate the collaborative efforts of everyone involved. We will continue to enhance and maintain this repository over the long term. If you encounter any issues, feel free to submit them to us!

We reference itext2kg to aid in building the schema repository and utilize tools from LangChain for file parsing. The experimental datasets we use are curated from the IEPile repository. We appreciate their valuable contributions!