Real-ESRGAN

https://github.com/xinntao/real-esrgan
Admin
Real-ESRGAN develops practical algorithms for general image and video restoration, extending ESRGAN...
Tokens:19,480
Snippets:123
Trust Score:8.4
License:BSD-3-Clause
Update:2 weeks ago
Show doc for...
Tokens:
Raw
### Install dependencies and setup Real-ESRGAN

Source: https://github.com/xinntao/real-esrgan/blob/master/README.md

Installs the required Python packages (basicsr, facexlib, gfpgan) and the project dependencies, then runs setup.py develop to install Real-ESRGAN in development mode. Requires Python >= 3.7 and PyTorch >= 1.7.

```bash
# Install basicsr - https://github.com/xinntao/BasicSR
# We use BasicSR for both training and inference
pip install basicsr
# facexlib and gfpgan are for face enhancement
pip install facexlib
pip install gfpgan
pip install -r requirements.txt
python setup.py develop
```

--------------------------------

### Example meta_info txt content

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Example lines from a meta_info txt file that lists image paths. This file is used to prepare training data and must be generated for your own dataset.

```txt
DF2K_HR_sub/000001_s001.png
DF2K_HR_sub/000001_s002.png
DF2K_HR_sub/000001_s003.png
...
```

--------------------------------

### inference_realesrgan.py usage and options

Source: https://github.com/xinntao/real-esrgan/blob/master/README.md

Command-line help for inference_realesrgan.py. Shows all options, defaults, and a common usage example with --outscale 3.5 and --face_enhance.

```console
Usage: python inference_realesrgan.py -n RealESRGAN_x4plus -i infile -o outfile [options]...

A common command: python inference_realesrgan.py -n RealESRGAN_x4plus -i infile --outscale 3.5 --face_enhance

  -h                   show this help
  -i --input           Input image or folder. Default: inputs
  -o --output          Output folder. Default: results
  -n --model_name      Model name. Default: RealESRGAN_x4plus
  -s, --outscale       The final upsampling scale of the image. Default: 4
  --suffix             Suffix of the restored image. Default: out
  -t, --tile           Tile size, 0 for no tile during testing. Default: 0
  --face_enhance       Whether to use GFPGAN to enhance face. Default: False
  --fp32               Use fp32 precision during inference. Default: fp16 (half precision).
  --ext                Image extension. Options: auto | jpg | png, auto means using the same extension as inputs. Default: auto
```

--------------------------------

### Train Real-ESRGAN formally (multi-GPU and single-GPU)

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Start formal Real-ESRGAN training with automatic resume. The first command uses four GPUs; the second uses a single GPU.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --auto_resume
```

```bash
python realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --auto_resume
```

--------------------------------

### Train Real-ESRNet formally (multi-GPU and single-GPU)

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Start formal Real-ESRNet training with automatic resume. The first command uses four GPUs; the second uses a single GPU.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --auto_resume
```

```bash
python realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --auto_resume
```

--------------------------------

### Debug Real-ESRGAN training (multi-GPU and single-GPU)

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Run Real-ESRGAN training in debug mode to verify setup. The first command uses four GPUs; the second uses a single GPU.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --launcher pytorch --debug
```

```bash
python realesrgan/train.py -opt options/train_realesrgan_x4plus.yml --debug
```

--------------------------------

### Clone Real-ESRGAN repository

Source: https://github.com/xinntao/real-esrgan/blob/master/README.md

Clones the Real-ESRGAN repository and changes into its directory. This is the first step of the installation process.

```bash
git clone https://github.com/xinntao/Real-ESRGAN.git
cd Real-ESRGAN
```

--------------------------------

### Debug Real-ESRNet training (multi-GPU and single-GPU)

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Run Real-ESRNet training in debug mode to verify setup. The first command uses four GPUs with distributed launch; the second uses a single GPU.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --launcher pytorch --debug
```

```bash
python realesrgan/train.py -opt options/train_realesrnet_x4plus.yml --debug
```

--------------------------------

### Run Real-ESRGAN NCNN executable on frames

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_video_model.md

Run the Real-ESRGAN NCNN Vulkan executable on extracted frames. Create the out_frames folder before running. This example uses the Windows executable.

```bash
./realesrgan-ncnn-vulkan.exe -i tmp_frames -o out_frames -n realesr-animevideov3 -s 2 -f jpg
```

--------------------------------

### Get input video fps with ffmpeg

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_video_model.md

Get the fps of the input video by running ffmpeg without an output. The console block documents the -i flag as the input video path.

```bash
ffmpeg -i onepiece_demo.mp4
```

```console
Usage:
-i                   input video path
```

--------------------------------

### Finetune Real-ESRGAN with four GPUs (on-the-fly degradation)

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Launches distributed training with four GPUs using the finetune_realesrgan_x4plus.yml configuration. The --auto_resume flag automatically resumes training if interrupted.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/finetune_realesrgan_x4plus.yml --launcher pytorch --auto_resume
```

--------------------------------

### Finetune Real-ESRGAN with four GPUs (paired data)

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Launches distributed training with four GPUs using the finetune_realesrgan_x4plus_pairdata.yml configuration. The --auto_resume flag automatically resumes training if interrupted.

```bash
CUDA_VISIBLE_DEVICES=0,1,2,3 \
python -m torch.distributed.launch --nproc_per_node=4 --master_port=4321 realesrgan/train.py -opt options/finetune_realesrgan_x4plus_pairdata.yml --launcher pytorch --auto_resume
```

--------------------------------

### Finetune Real-ESRGAN with a single GPU (on-the-fly degradation)

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Runs fine-tuning on a single GPU using the finetune_realesrgan_x4plus.yml configuration. The --auto_resume flag automatically resumes training if interrupted.

```bash
python realesrgan/train.py -opt options/finetune_realesrgan_x4plus.yml --auto_resume
```

--------------------------------

### Finetune Real-ESRGAN with a single GPU (paired data)

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Runs fine-tuning on a single GPU using the finetune_realesrgan_x4plus_pairdata.yml configuration. The --auto_resume flag automatically resumes training if interrupted.

```bash
python realesrgan/train.py -opt options/finetune_realesrgan_x4plus_pairdata.yml --auto_resume
```

--------------------------------

### Download and run PyTorch inference with RealESRGAN_x4plus_anime_6B

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_model.md

Downloads the RealESRGAN_x4plus_anime_6B model into the weights directory and runs inference with the provided script, specifying the model name and input directory.

```bash
# download model
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.4/RealESRGAN_x4plus_anime_6B.pth -P weights
# inference
python inference_realesrgan.py -n RealESRGAN_x4plus_anime_6B -i inputs
```

--------------------------------

### Run PyTorch video inference with realesr-animevideov3

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_video_model.md

Download the realesr-animevideov3 model and run PyTorch video inference with single GPU, single GPU multi-process, or multi-GPU multi-process configurations. The --num_process_per_gpu flag increases GPU utilization when IO is the bottleneck.

```bash
# download model
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.5.0/realesr-animevideov3.pth -P weights
# single gpu and single process inference
CUDA_VISIBLE_DEVICES=0 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2
# single gpu and multi process inference (you can use multi-processing to improve GPU utilization)
CUDA_VISIBLE_DEVICES=0 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2 --num_process_per_gpu 2
# multi gpu and multi process inference
CUDA_VISIBLE_DEVICES=0,1,2,3 python inference_realesrgan_video.py -i inputs/video/onepiece_demo.mp4 -n realesr-animevideov3 -s 2 --suffix outx2 --num_process_per_gpu 2
```

--------------------------------

### Download RealESRGAN_x4plus model

Source: https://github.com/xinntao/real-esrgan/blob/master/README.md

Downloads the RealESRGAN_x4plus.pth pre-trained model into the weights folder.

```bash
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P weights
```

--------------------------------

### Download RealESRGAN_x4plus.pth pre-trained model

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Downloads the RealESRGAN_x4plus.pth pre-trained model into the experiments/pretrained_models directory. This model is required for fine-tuning.

```bash
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.0/RealESRGAN_x4plus.pth -P experiments/pretrained_models
```

--------------------------------

### Download ESRGAN pre-trained model

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Downloads the pre-trained ESRGAN model into the experiments/pretrained_models directory as required before training Real-ESRNet.

```bash
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.1.1/ESRGAN_SRx4_DF2KOST_official-ff704c30.pth -P experiments/pretrained_models
```

--------------------------------

### Generate meta_info txt with script

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Command to generate a meta_info txt file using the generate_meta_info.py script. Merge multiple folders into one meta_info file.

```bash
 python scripts/generate_meta_info.py --input datasets/DF2K/DF2K_HR datasets/DF2K/DF2K_multiscale --root datasets/DF2K datasets/DF2K --meta_info datasets/DF2K/meta_info/meta_info_DF2Kmultiscale.txt
```

--------------------------------

### Download RealESRGAN_x4plus_netD.pth pre-trained model

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Downloads the RealESRGAN_x4plus_netD.pth pre-trained discriminator model into the experiments/pretrained_models directory. This model is required for fine-tuning.

```bash
wget https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.2.3/RealESRGAN_x4plus_netD.pth -P experiments/pretrained_models
```

--------------------------------

### Run Real-ESRGAN NCNN executable

Source: https://github.com/xinntao/real-esrgan/blob/master/README.md

Runs the portable NCNN executable for Real-ESRGAN on Windows. The -n argument selects the model; available models include realesrgan-x4plus (default), realesrnet-x4plus, realesrgan-x4plus-anime, and realesr-animevideov3. No CUDA or PyTorch environment is needed.

```bash
./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n model_name
```

--------------------------------

### Convert ONNX model to NCNN with onnx2ncnn.exe

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/ncnn_conversion.md

Navigate to the onnx2ncnn tool directory and convert the ONNX model to NCNN format, producing raw .param and .bin files.

```shell
cd ncnn-master\ncnn\build\tools\onnx
```

```shell
onnx2ncnn.exe realesrgan-x4.onnx realesrgan-x4-raw.param realesrgan-x4-raw.bin
```

--------------------------------

### NCNN executable usage options

Source: https://github.com/xinntao/real-esrgan/blob/master/README.md

Shows the command-line options for the portable realesrgan-ncnn-vulkan executable. Note that it does not support all functions (such as outscale) compared to the Python script inference_realesrgan.py. It may introduce block inconsistency because it processes the image in tiles.

```console
Usage: realesrgan-ncnn-vulkan.exe -i infile -o outfile [options]...

  -h                   show this help
  -i input-path        input image path (jpg/png/webp) or directory
  -o output-path       output image path (jpg/png/webp) or directory
  -s scale             upscale ratio (can be 2, 3, 4. default=4)
  -t tile-size         tile size (>=32/0=auto, default=0) can be 0,0,0 for multi-gpu
  -m model-path        folder path to the pre-trained models. default=models
  -n model-name        model name (default=realesr-animevideov3, can be realesr-animevideov3 | realesrgan-x4plus | realesrgan-x4plus-anime | realesrnet-x4plus)
  -g gpu-id            gpu device to use (default=auto) can be 0,1,2 for multi-gpu
  -j load:proc:save    thread count for load/proc/save (default=1:2:2) can be 1:2,2,2:2 for multi-gpu
  -x                   enable tta mode"
  -f format            output image format (jpg/png/webp, default=ext/png)
  -v                   verbose output
```

--------------------------------

### Optimize NCNN model with ncnnoptimize.exe in fp16 mode

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/ncnn_conversion.md

Navigate to the ncnn tools directory and optimize the raw NCNN model in fp16 mode (flag 1), producing the final .param and .bin files.

```shell
cd ncnn-master\ncnn\build\tools
```

```shell
ncnnoptimize.exe realesrgan-x4-raw.param realesrgan-x4-raw.bin realesrgan-x4.param realesrgan-x4.bin 1
```

--------------------------------

### Configure Real-ESRNet training dataset

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Configuration snippet for the training dataset in options/train_realesrnet_x4plus.yml. Modify dataroot_gt and meta_info to match your dataset paths.

```yaml
train:
    name: DF2K+OST
    type: RealESRGANDataset
    dataroot_gt: datasets/DF2K  # modify to the root path of your folder
    meta_info: realesrgan/meta_info/meta_info_DF2Kmultiscale+OST_sub.txt  # modify to your own generate meta info txt
    io_backend:
        type: disk
```

--------------------------------

### Configure finetune_realesrgan_x4plus_pairdata.yml for paired data

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

YAML configuration for the training dataset when using paired data. Modify the dataroot_gt, dataroot_lq, and meta_info paths to match your dataset.

```yaml
train:
    name: DIV2K
    type: RealESRGANPairedDataset
    dataroot_gt: datasets/DF2K  # modify to the root path of your folder
    dataroot_lq: datasets/DF2K  # modify to the root path of your folder
    meta_info: datasets/DF2K/meta_info/meta_info_DIV2K_sub_pair.txt  # modify to your own generate meta info txt
    io_backend:
        type: disk
```

--------------------------------

### Inference general images with RealESRGAN_x4plus

Source: https://github.com/xinntao/real-esrgan/blob/master/README.md

Runs inference on general images with the RealESRGAN_x4plus model and face enhancement enabled. Results are saved in the results folder.

```bash
python inference_realesrgan.py -n RealESRGAN_x4plus -i inputs --face_enhance
```

--------------------------------

### Run Real-ESRGAN inference with --fp32

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/FAQ.md

Use this command to run Real-ESRGAN inference with fp32 precision, which is required when the CPU mode does not support half precision (fp16) operators. The --fp32 option avoids the 'slow_conv2d_cpu' not implemented for 'Half' error.

```bash
python inference_realesrgan.py -n RealESRGAN_x4plus.pth -i inputs --fp32
```

--------------------------------

### Run ncnn executable for anime model on Windows

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_model.md

Runs the ncnn executable for Windows to upscale an input image using the anime model, specifying input, output, and model name.

```bash
./realesrgan-ncnn-vulkan.exe -i input.jpg -o output.png -n realesrgan-x4plus-anime
```

--------------------------------

### Generate meta info for paired data

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Generates a meta_info txt file for paired training data using the generate_meta_info_pairdata.py script. Provide the ground-truth and low-quality folders as input.

```bash
python scripts/generate_meta_info_pairdata.py --input datasets/DF2K/DIV2K_train_HR_sub datasets/DF2K/DIV2K_train_LR_bicubic_X4_sub --meta_info datasets/DF2K/meta_info/meta_info_DIV2K_sub_pair.txt
```

--------------------------------

### Enable validation in Real-ESRNet training

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Commented-out validation configuration for Real-ESRNet training. Uncomment and adjust paths and metrics to enable validation during training.

```yaml
  # Uncomment these for validation
  # val:
  #   name: validation
  #   type: PairedImageDataset
  #   dataroot_gt: path_to_gt
  #   dataroot_lq: path_to_lq
  #   io_backend:
  #     type: disk

...

  # Uncomment these for validation
  # validation settings
  # val:
  #   val_freq: !!float 5e3
  #   save_img: True

  #   metrics:
  #     psnr: # metric name, can be arbitrary
  #       type: calculate_psnr
  #       crop_border: 4
  #       test_y_channel: false
```

--------------------------------

### Extract video frames with ffmpeg for NCNN inference

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_video_model.md

Extract frames from a video using ffmpeg for the NCNN workflow. Create the tmp_frames folder before running.

```bash
ffmpeg -i onepiece_demo.mp4 -qscale:v 1 -qmin 1 -qmax 1 -vsync 0 tmp_frames/frame%08d.png
```

--------------------------------

### Generate multi-scale DF2K images

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Optional step to downsample HR images to multiple scales for the DF2K dataset. Use this script to create multi-scale ground-truth images.

```bash
python scripts/generate_multiscale_DF2K.py --input datasets/DF2K/DF2K_HR --output datasets/DF2K/DF2K_multiscale
```

--------------------------------

### Merge frames and copy audio with ffmpeg

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_video_model.md

Merge enhanced frames into a video and copy audio from the original input. The console block documents the -i, -c:v, -r, and -pix_fmt flags, noting that -i uses two input streams.

```bash
ffmpeg -r 23.98 -i out_frames/frame%08d.jpg -i onepiece_demo.mp4 -map 0:v:0 -map 1:a:0 -c:a copy -c:v libx264 -r 23.98 -pix_fmt yuv420p output_w_audio.mp4
```

```console
Usage:
-i                   input video path, here we use two input streams
-c:v                 video encoder (usually we use libx264)
-r                   fps, remember to modify it to meet your needs
-pix_fmt             pixel format in video
```

--------------------------------

### Crop DF2K images to sub-images

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/Training.md

Optional step to crop DF2K images into smaller sub-images for faster IO and processing. Use the extract_subimages.py script with specified crop size and step.

```bash
 python scripts/extract_subimages.py --input datasets/DF2K/DF2K_multiscale --output datasets/DF2K/DF2K_multiscale_sub --crop_size 400 --step 200
```

--------------------------------

### Merge enhanced frames into video with ffmpeg

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_video_model.md

Merge enhanced frames into a video with ffmpeg. The console block documents the -i, -c:v, -r, and -pix_fmt flags.

```bash
ffmpeg -r 23.98 -i out_frames/frame%08d.jpg -c:v libx264 -r 23.98 -pix_fmt yuv420p output.mp4
```

```console
Usage:
-i                   input video path
-c:v                 video encoder (usually we use libx264)
-r                   fps, remember to modify it to meet your needs
-pix_fmt             pixel format in video
```

--------------------------------

### PyTorch inference script usage for --num_process_per_gpu and --extract_frame_first

Source: https://github.com/xinntao/real-esrgan/blob/master/docs/anime_video_model.md

Usage notes for the PyTorch inference script. --num_process_per_gpu controls the total process count (num_gpu * num_process_per_gpu) to improve GPU utilization when IO is the bottleneck; --extract_frame_first can be enabled if ffmpeg errors occur with multi-processing.

```console
Usage:
--num_process_per_gpu    The total number of process is num_gpu * num_process_per_gpu. The bottleneck of
                         the program lies on the IO, so the GPUs are usually not fully utilized. To alleviate
                         this issue, you can use multi-processing by setting this parameter. As long as it
                         does not exceed the CUDA memory
--extract_frame_first    If you encounter ffmpeg error when using multi-processing, you can turn this option on.
```
Tokens:
Raw