DeepKE

https://github.com/zjunlp/deepke
Admin
[EMNLP 2022] An Open Toolkit for Knowledge Graph Extraction and Construction
Tokens:95,185
Snippets:342
Trust Score:9.6
License:MIT
Update:1 year ago
Show doc for...
Tokens:
Raw
### Install Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/ee/standard/degree/README.md

Installs project dependencies using pip. It assumes a `requirements.txt` file is present in the parent directory for the installation process.

```bash
cd ..
pip install -r requirements
```

--------------------------------

### Install Project Dependencies with Pip

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/ASP/README.md

Installs all required Python packages listed in the 'requirements.txt' file using pip. This ensures all necessary libraries are available for the project to run.

```Shell
pip install -r requirements.txt
```

--------------------------------

### Install Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/ae/standard/README.md

Installs all required Python packages listed in the 'requirements.txt' file. It is recommended to do this within a Python virtual environment.

```bash
pip install -r requirements.txt
```

--------------------------------

### Download DeepKE Code

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/few-shot/README.md

Clones the DeepKE GitHub repository and navigates into the specific example directory for few-shot NER.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/ner/few-shot
```

--------------------------------

### Download DeepKE Code

Source: https://github.com/zjunlp/deepke/blob/main/example/ae/standard/README.md

Clones the DeepKE GitHub repository and navigates into the standard AE example directory. This is the first step to obtain the project files.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/ae/standard
```

--------------------------------

### Clone DeepKE Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/standard/README.md

Clones the DeepKE GitHub repository and navigates into the standard NER example directory. This is the first step to access the project's code and examples.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/ner/standard
```

--------------------------------

### Download DeepKE Project

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/ASP/README.md

Clones the DeepKE project repository from GitHub and navigates into the ASP example directory. This is the initial step to obtain the project's source code.

```Shell
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/triple/ASP
```

--------------------------------

### Install DeepKE Package Locally

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/ASP/README.md

Builds and installs the DeepKE package from its source directory. This step is necessary if installing DeepKE directly rather than via pip.

```Shell
cd ~/DeepKE
python setup.py build
python setup.py install
```

--------------------------------

### Setup Environment and Download Data

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/example.rst

Commands to navigate to the example directory, download the necessary dataset archive, and extract it for use in the project.

```bash
cd example/ae/regular
```

```bash
wget 121.41.117.246:8080/Data/ae/standard/data.tar.gz
```

```bash
tar -xzvf data.tar.gz
```

--------------------------------

### Clone and Navigate Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/ee/standard/degree/README.md

Clones the DeepKE repository from GitHub and changes the current directory to the specific example path for EE standard degree.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/ee/standard/degree
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/re/few-shot/README_CN.md

Installs all necessary Python dependencies for the project using pip. It is recommended to do this within a Python virtual environment.

```bash
pip install -r requirements.txt
```

--------------------------------

### Download DeepKE Code

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/cross/README.md

Clones the DeepKE repository from GitHub and navigates into the specific example directory for cross-domain NER.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/ner/cross
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/standard/README.md

Installs all necessary Python packages listed in the 'requirements.txt' file. This ensures the project has all its required libraries for execution.

```bash
pip install -r requirements.txt
```

--------------------------------

### Standard RE: Setup and Execution

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/example.rst

This snippet details the process for setting up and running the Standard Relation Extraction (RE) example. It covers navigating to the example directory, downloading and extracting the dataset, and executing the training and prediction scripts.

```bash
cd example/re/standard

wget 121.41.117.246:8080/Data/re/standard/data.tar.gz

tar -xzvf data.tar.gz

python run.py

python predict.py
```

--------------------------------

### Clone and Install Apex Library

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/ASP/README.md

Clones the NVIDIA Apex library repository and installs it with specific build options. This library is a dependency for certain functionalities, requiring a modification to its setup.py.

```Shell
cd ~/DeepKE/example/triple/ASP
git clone https://github.com/NVIDIA/apex
cd apex
# Modify line 32 of setup.py as described in the documentation
pip install -v --disable-pip-version-check --no-cache-dir --global-option="--cpp_ext" --global-option="--cuda_ext" ./
```

--------------------------------

### Install DeepKE via setup.py

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/install.rst

Install DeepKE by building from source using setup.py. This method is suitable for developers who need to modify the source code before installation.

```python
python setup.py install
```

--------------------------------

### Standard NER: Setup and Execution

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/example.rst

This snippet details the steps to set up and run the Standard Named Entity Recognition (NER) example. It includes navigating to the example directory, downloading and extracting the dataset, and executing the training and prediction scripts.

```bash
cd example/ner/standard

wget 121.41.117.246:8080/Data/ner/standard/data.tar.gz

tar -xzvf data.tar.gz

python run.py

python predict.py
```

--------------------------------

### Download DeepKE Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/PRGC/README.md

Clones the DeepKE GitHub repository and navigates into the PRGC example directory. This is the first step to get the project code.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/triple/PRGC
```

--------------------------------

### Multimodal NER: Setup and Execution

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/example.rst

This snippet provides instructions for setting up and running the Multimodal Named Entity Recognition (NER) example. It includes steps for changing directories, downloading and extracting data, and executing the training and prediction scripts.

```bash
cd example/ner/multimodal

wget 121.41.117.246:8080/Data/ner/multimodal/data.tar.gz

tar -xzvf data.tar.gz

python run.py

python predict.py
```

--------------------------------

### Install Project Requirements

Source: https://github.com/zjunlp/deepke/blob/main/example/ee/standard/README.md

Installs project dependencies using pip. This includes installing packages listed in requirements.txt and the hydra-core library, ensuring compatibility for the project.

```bash
pip install -r requirements.txt
pip install hydra-core==1.3.1
```

--------------------------------

### Clone DeepKE Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/re/few-shot/README_CN.md

Clones the DeepKE GitHub repository to your local machine. After cloning, navigate into the specific example directory for few-shot relation extraction.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/re/few-shot
```

--------------------------------

### Few-shot NER: Setup and Execution

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/example.rst

This snippet outlines the procedure for setting up and running the Few-shot Named Entity Recognition (NER) example. It covers directory navigation, data download and extraction, and the execution of training and prediction scripts, including options for few-shot training.

```bash
cd example/ner/few-shot

wget 121.41.117.246:8080/Data/ner/few_shot/data.tar.gz

tar -xzvf data.tar.gz

python run.py

python predict.py
```

--------------------------------

### Create and Activate Pip Virtual Environment

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/ASP/README.md

Creates a new Python virtual environment named 'asp' and activates it. This isolates project dependencies from the system's Python installation.

```Shell
python -m venv <path_to_venv>/asp  
source <path_to_venv>/asp/bin/activate
```

--------------------------------

### Clone DeepKE Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/standard/README_CN.md

Clones the DeepKE GitHub repository and navigates to the standard NER example directory. This is the first step to get started with the project.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/ner/standard
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/llm/CPM-Bee/README_CN.md

Installs all necessary Python packages listed in the requirements.txt file. Ensure Python and Torch versions are compatible as per project guidelines.

```bash
pip install -r requirements.txt
```

--------------------------------

### Train on CoNLL-2003 Dataset

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/few-shot/README.md

Initiates the training process for the model using the CoNLL-2003 dataset.

```bash
python run.py
```

--------------------------------

### Download DeepKE Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/re/multimodal/README.md

Clones the DeepKE GitHub repository to your local machine and navigates into the multimodal example directory.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/re/multimodal
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/PRGC/README.md

Installs all necessary Python packages listed in the requirements.txt file using pip. This ensures the environment is set up correctly.

```bash
pip install -r requirements.txt
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/standard/w2ner/README.md

Installs all required Python packages listed in the 'requirements.txt' file. This command ensures that all necessary libraries and their specific versions are available for the project to run correctly.

```bash
pip install -r requirements.txt
```

--------------------------------

### Few-shot Training for Chinese

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/few-shot/README.md

Starts the few-shot training process specifically for Chinese datasets. Pretrained weights must be provided in the directory specified in 'few_shot_cn.yaml'.

```bash
python run.py +train=few_shot_cn
```

--------------------------------

### Setup mcp project uv environment

Source: https://github.com/zjunlp/deepke/blob/main/mcp-tools/README.md

Installs the 'uv' package manager and sets up a virtual environment for the mcp-tools project. It then activates the environment and installs the necessary Python dependencies, including 'mcp[cli]', 'httpx', 'openai', and 'pyyaml'.

```bash
pip install uv
cd mcp-tools
uv venv
source .venv/bin/activate
uv add "mcp[cli]" httpx openai pyyaml
```

--------------------------------

### Install Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/cross/README_CN.md

Installs project dependencies using pip. It's recommended to create and activate a Python virtual environment before running this command.

```bash
pip install -r requirements.txt
```

--------------------------------

### Start W2NER Training

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/standard/README.md

Executes the training process for the W2NER model. This involves navigating to the `w2ner` directory first. Model hyperparameters are defined in `model.yaml`, and the specific GPU to be used for training can be set via the `device` parameter.

```bash
cd w2ner
python run.py
```

--------------------------------

### Clone DeepKE Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/mt5/README.md

Clones the DeepKE GitHub repository and navigates into the specific example directory for mt5 triple extraction.

```bash
git clone  https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/triple/mt5
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/multimodal/README_CN.md

Install the necessary Python dependencies for the project using pip. It's recommended to create and activate a Python virtual environment before running this command.

```bash
pip install -r requirements.txt
```

--------------------------------

### Download DeepKE Code

Source: https://github.com/zjunlp/deepke/blob/main/example/re/standard/README.md

Clones the DeepKE repository and navigates to the relation extraction example directory.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/re/standard
```

--------------------------------

### Download and Extract Dataset

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/few-shot/README.md

Downloads the required dataset archive and extracts its contents for use in training and evaluation.

```bash
wget 121.41.117.246:8080/Data/ner/few_shot/data.tar.gz
tar -xzvf data.tar.gz
```

--------------------------------

### Build DeepKE Docker Image

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/install.rst

Instructions for building a Docker image for DeepKE. This involves navigating to the docker directory and executing the build command.

```bash
cd docker
docker build -t deepke .
conda activate deepke
```

--------------------------------

### LightNER Citation

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/few-shot/README.md

BibTeX entry for citing the LightNER paper, detailing its contribution to lightweight tuning for low-resource NER.

```bibtex
@inproceedings{DBLP:conf/coling/00160DTXHSCZ22,
  author    = {Xiang Chen and
               Lei Li and
               Shumin Deng and
               Chuanqi Tan and
               Changliang Xu and
               Fei Huang and
               Luo Si and
               Huajun Chen and
               Ningyu Zhang},
  editor    = {Nicoletta Calzolari and
               Chu{-}Ren Huang and
               Hansaem Kim and
               James Pustejovsky and
               Leo Wanner and
               Key{-}Sun Choi and
               Pum{-}Mo Ryu and
               Hsin{-}Hsi Chen and
               Lucia Donatelli and
               Heng Ji and
               Sadao Kurohashi and
               Patrizia Paggio and
               Nianwen Xue and
               Seokhwan Kim and
               Younggyun Hahm and
               Zhong He and
               Tony Kyungil Lee and
               Enrico Santus and
               Francis Bond and
               Seung{-}Hoon Na},
  title     = {LightNER: {A} Lightweight Tuning Paradigm for Low-resource {NER} via
               Pluggable Prompting},
  booktitle = {Proceedings of the 29th International Conference on Computational
               Linguistics, {COLING} 2022, Gyeongju, Republic of Korea, October 12-17,
               2022},
  pages     = {2374--2387},
  publisher = {International Committee on Computational Linguistics},
  year      = {2022},
  url       = {https://aclanthology.org/2022.coling-1.209},
  timestamp = {Mon, 13 Mar 2023 11:20:33 +0100},
  biburl    = {https://dblp.org/rec/conf/coling/00160DTXHSCZ22.bib},
  bibsource = {dblp computer science bibliography, https://dblp.org}
}
```

--------------------------------

### Standard AE Module Setup

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/example.rst

This snippet describes the initial steps for the Standard AE module, which supports various deep learning models. It involves navigating to the module directory and downloading/extracting the dataset. Customization of dataset and parameters is done via 'data' and 'conf' folders.

```bash
cd DeepKE/example/ae/standard

wget 121.41.117.246:8080/Data/ae/standard/data.tar.gz

tar -xzvf data.tar.gz
```

--------------------------------

### Setup Python Environment and Install Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/mt5/README.md

Creates a new Conda virtual environment named 'ccks-mt5' with Python 3.9, activates it, and installs the required Python packages from 'requirements.txt'.

```bash
conda create -n ccks-mt5 python=3.9   
conda activate ccks-mt5
pip install -r requirements.txt
```

--------------------------------

### Download and Navigate DeepKE Project

Source: https://github.com/zjunlp/deepke/blob/main/example/re/document/README.md

Clones the DeepKE repository from GitHub and changes the directory to the example's document processing section. This is the initial step to obtain the project's source code.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/re/document
```

--------------------------------

### Run Finetune with DeepSpeed and Custom Parameters

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/mt5/README.md

Launches the finetuning process using DeepSpeed, specifying multiple GPUs, and configuring various training parameters like epochs, batch size, learning rate, and model paths.

```bash
deepspeed  --include localhost:0,1 run_finetune.py \
    --do_train --do_eval --do_predict \
    --num_train_epochs 10 \
    --per_device_train_batch_size 16 \
    --per_device_eval_batch_size 48 \
    --gradient_accumulation_steps 2 \
    --predict_with_generate \
    --from_checkpoint=True \
    --overwrite_output_dir=False \
    --model_name_or_path google/mt5-base   \
    --output_dir output/ccks_mt5-base_f1_1e-4  \
    --logging_dir output/ccks_mt5-base_f1_1e-4_log \
    --train_file data/train.json \
    --test_file data/valid.json \
    --save_total_limit 1 \
    --load_best_model_at_end \
    --save_strategy "epoch" \
    --evaluation_strategy "epoch" \
    --metric_for_best_model "overall-score" \
    --learning_rate 1e-4 \
    --use_fast_tokenizer=True \
    --preprocessing_num_workers 4 \
    --generation_max_length 256 \
    --generation_num_beams 1 \
    --gradient_checkpointing=True \
    --deepspeed "configs/ds_mt5_z3_config_bf16.json" \
    --seed 42 \
    --bf16=True \

```

--------------------------------

### Configure Language for English Datasets

Source: https://github.com/zjunlp/deepke/blob/main/example/ner/standard/README.md

Specifies the configuration change required when using English datasets. The 'lan' parameter in 'config.yaml' needs to be updated, and 'nltk' must be installed with the 'punkt' tokenizer.

```yaml
# Example configuration snippet (conceptual, actual path in config.yaml)
# lan: en
```

--------------------------------

### Clone DeepKE Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/ee/standard/README.md

Clones the DeepKE GitHub repository and navigates into the example directory for event extraction tasks. This step downloads the necessary project files.

```bash
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/ee/standard
```

--------------------------------

### Install Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/re/standard/README.md

Installs project dependencies using pip from a requirements file.

```bash
pip install -r requirements.txt
```

--------------------------------

### Run Training

Source: https://github.com/zjunlp/deepke/blob/main/example/ae/standard/README.md

Executes the main training script. Configuration parameters for training, such as multi-GPU usage and logging, can be adjusted in the 'conf' folder before running.

```python
python run.py
```

--------------------------------

### Install DeepKE via Pip

Source: https://github.com/zjunlp/deepke/blob/main/docs/source/install.rst

Install the DeepKE package directly from PyPI. This is the recommended method for users who want to use the library without modifying source code.

```python
pip install deepke
```

--------------------------------

### Custom Few-Shot Example Instruction

Source: https://github.com/zjunlp/deepke/blob/main/example/llm/OneKE.md

Example of an instruction format that includes few-shot examples to guide the model. This approach enhances extraction accuracy by providing concrete input-output pairs for the model to learn from.

```json
{
    "instruction": "你是专门进行实体抽取的专家。请从input中抽取出符合schema定义的实体，不存在的实体类型返回空列表。请按照JSON字符串的格式回答。你可以参考example进行抽取。",
    "schema": [
        "检查指标"
    ],
    "example": [
        {
            "input": "CKD诊断标准：1.以下任意一项指标持续超过3个月；且至少满足1项。(1)肾损伤标志：白蛋白尿[尿白蛋白排泄率(AER)≥30mg／24h；尿白蛋白肌酐比值(ACR)≥3mg／mmol]；尿沉渣异常；肾小管相关病变；组织学异常；影像学所见结构异常；肾移植病史。(2)肾小球滤过率下降：eGFR≤60ml·min-1·1.73m-2 ",
            "output": {
                "检查指标": [
                    "尿白蛋白排泄率(AER)",
                    "尿白蛋白肌酐比值(ACR)",
                    "肾小球滤过率",
                    "eGFR"
                ]
            }
        },
        {
            "input": "DPP-4抑制剂在特殊人群中的应用",
            "output": {
                "检查指标": []
            }
        }
    ],
    "input": "目前所有磺脲类药物说明书均将重度肝功能不全列为禁忌证。丙氨酸氨基转移酶(alanine transaminase，ALT)>3倍参考值上限可作为肝损害的敏感而特异的指标，若ALT>8~10倍参考值上限或者ALT>3倍参考值上限且血清总胆红素(total bilirubin，TBIL)>2倍参考值上限则是预测重度肝损害的特异指标，表明肝脏实质细胞受到损害，此时应禁用磺脲类药物。在临床使用中，伴有肝性脑病、腹水或凝血障碍的失代偿肝硬化患者应禁用该类药物以防发生低血糖。"
}
```

--------------------------------

### Install Project Dependencies

Source: https://github.com/zjunlp/deepke/blob/main/example/re/multimodal/README.md

Installs the required Python packages listed in the requirements.txt file. It's recommended to use a virtual environment.

```bash
pip install -r requirements.txt
```

```text
python == 3.8
torch == 1.5
transformers == 3.4.0
hydra-core == 1.0.6
deepke
```

--------------------------------

### Run Finetune Script

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/mt5/README.md

Executes the main finetuning process using a provided bash script.

```bash
bash run_finetene_ds.bash
```

--------------------------------

### Install DeepKE Package

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/ASP/README_CN.md

Installs the DeepKE package from source after building it. Requires modifying numpy version in requirements.txt beforehand.

```Shell
cd ~/DeepKE
python setup.py build
python setup.py install
```

--------------------------------

### Train Model

Source: https://github.com/zjunlp/deepke/blob/main/example/ee/standard/degree/README.md

Initiates the model training process by executing the `run.py` script. Training parameters, model paths, and other configurations are specified in the `.conf/config.yaml` file.

```python
python run.py
```

--------------------------------

### Clone DeepKE Repository

Source: https://github.com/zjunlp/deepke/blob/main/example/triple/ASP/README_CN.md

Clones the DeepKE GitHub repository and navigates into the ASP example directory. This is the initial step to obtain the project code.

```Shell
git clone https://github.com/zjunlp/DeepKE.git
cd DeepKE/example/triple/ASP
```
Tokens:
Raw