Zero123++

https://github.com/sudo-ai-3d/zero123plus
Admin
Zero123++ is a single image to consistent multi-view diffusion base model that generates multi-view...
Tokens:2,479
Snippets:23
Trust Score:6.6
License:Apache-2.0
Update:3 months ago
Show doc for...
Tokens:
Raw
### Install Project Dependencies

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

Installs all necessary dependencies for the project from the requirements.txt file. This is a prerequisite for running the local demo.

```bash
pip install -r requirements.txt
```

--------------------------------

### Run Streamlit Demo Locally

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

Launches the Streamlit-based interactive demo application. Ensure dependencies are installed first.

```bash
streamlit run app.py
```

--------------------------------

### Run Gradio Demo Locally

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

Launches the Gradio-based interactive demo application. Ensure dependencies are installed first.

```python
python gradio_app.py
```

--------------------------------

### Generate Multi-View Images with Zero123++

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

Use this pipeline to generate multi-view images from a single input image. Ensure you have torch, diffusers, and transformers installed. The recommended diffusers version is 0.20.2. This example requires approximately 5GB of VRAM. Input images should be square with a resolution of 320x320 or higher.

```python
import torch
import requests
from PIL import Image
from diffusers import DiffusionPipeline, EulerAncestralDiscreteScheduler

# Load the pipeline
pipeline = DiffusionPipeline.from_pretrained(
    "sudo-ai/zero123plus-v1.1", custom_pipeline="sudo-ai/zero123plus-pipeline",
    torch_dtype=torch.float16
)

# Feel free to tune the scheduler!
# `timestep_spacing` parameter is not supported in older versions of `diffusers`
# so there may be performance degradations
# We recommend using `diffusers==0.20.2`
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(
    pipeline.scheduler.config, timestep_spacing='trailing'
)
pipeline.to('cuda:0')

# Download an example image.
cond = Image.open(requests.get("https://d.skis.ltd/nrp/sample-data/lysol.png", stream=True).raw)

# Run the pipeline!
result = pipeline(cond, num_inference_steps=75).images[0]
# for general real and synthetic images of general objects
# usually it is enough to have around 28 inference steps
# for images with delicate details like faces (real or anime)
# you may need 75-100 steps for the details to construct

result.show()
result.save("output.png")
```

--------------------------------

### Remove Background with rembg

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

After generating an image, you can optionally remove the gray background using the `rembg` library. Install it using `pip install rembg`.

```python
# !pip install rembg
import rembg
result = rembg.remove(result)
result.show()
```

--------------------------------

### Matting Postprocess Example

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

This script shows how to use the generated normal images for matting postprocessing. It helps obtain a more accurate mask compared to SAM-based approaches.

```python
import torch

from diffusers import Zero123plusPipeline

pipe = Zero123plusPipeline.from_pretrained(
    "sudo-ai/zero123plus", torch_dtype=torch.float16
)
pipe.load_lora_weights("sudo-ai/zero123plus-lora-normal", weight_name="zero123plus-lora-normal.safetensors")
pipe.to("cuda")

# Load your image
init_image = load_image("burger.png").convert("RGB")

# Generate normal image
normal_image = pipe(
    init_image,
    num_inference_steps=25,
    guidance_scale=3.0,
    controlnet_conditioning_scale=0.8,
    camera_params={
        "intrinsics": [1000, 1000, 512, 512],
        "extrinsics": [0, 0, 0, 0, 0, 0, 1],
    },
    output_type="pil",
).images[0]

# Use the normal image to generate a mask
# This part would involve a separate matting model or function
# For demonstration, let's assume we have a function `generate_mask_from_normals`
# mask = generate_mask_from_normals(normal_image)

# The actual matting postprocess logic would go here.
# This example snippet focuses on the setup and normal generation part.
# For a complete matting example, refer to the full script in the repository.
print("Normal image generated. Matting postprocess logic to be implemented.")
```

--------------------------------

### Normal Generator Example

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

This script demonstrates the usage of the normal generator ControlNet. It can be used to generate view-space normal images, which can then be used to obtain more accurate masks.

```python
import torch

from diffusers import Zero123plusPipeline

pipe = Zero123plusPipeline.from_pretrained(
    "sudo-ai/zero123plus", torch_dtype=torch.float16
)
pipe.load_lora_weights("sudo-ai/zero123plus-lora-normal", weight_name="zero123plus-lora-normal.safetensors")
pipe.to("cuda")

# Load your image
init_image = load_image("burger.png").convert("RGB")

# Generate normal image
# The normal generator is a ControlNet, so we pass it to the pipeline
# The controlnet_conditioning_scale controls the strength of the controlnet
# The camera_params are the intrinsics and extrinsics of the camera
# The output_type can be "pil" or "np" (numpy array)

output = pipe(
    init_image,
    num_inference_steps=25,
    guidance_scale=3.0,
    controlnet_conditioning_scale=0.8,
    camera_params={
        "intrinsics": [1000, 1000, 512, 512],
        "extrinsics": [0, 0, 0, 0, 0, 0, 1],
    },
    output_type="pil",
).images[0]

output.save("burger-normal.png")
```

--------------------------------

### Generate Images with Depth ControlNet

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

This example demonstrates how to use the depth ControlNet with Zero123++ for image generation. It requires approximately 5.7GB of VRAM. Ensure you have the necessary models loaded and the conditioning scale set appropriately.

```python
import torch
import requests
from PIL import Image
from diffusers import DiffusionPipeline, EulerAncestralDiscreteScheduler, ControlNetModel

# Load the pipeline
pipeline = DiffusionPipeline.from_pretrained(
    "sudo-ai/zero123plus-v1.1", custom_pipeline="sudo-ai/zero123plus-pipeline",
    torch_dtype=torch.float16
)
pipeline.add_controlnet(ControlNetModel.from_pretrained(
    "sudo-ai/controlnet-zp11-depth-v1", torch_dtype=torch.float16
), conditioning_scale=0.75)
# Feel free to tune the scheduler
pipeline.scheduler = EulerAncestralDiscreteScheduler.from_config(
    pipeline.scheduler.config, timestep_spacing='trailing'
)
pipeline.to('cuda:0')
# Run the pipeline
cond = Image.open(requests.get("https://d.skis.ltd/nrp/sample-data/0_cond.png", stream=True).raw)
depth = Image.open(requests.get("https://d.skis.ltd/nrp/sample-data/0_depth.png", stream=True).raw)
result = pipeline(cond, depth_image=depth, num_inference_steps=36).images[0]
result.show()
result.save("output.png")
```

--------------------------------

### Zero123++ Citation

Source: https://github.com/sudo-ai-3d/zero123plus/blob/main/README.md

BibTeX entry for citing the Zero123++ research paper. Include this in academic works that utilize the model.

```bibtex
@misc{shi2023zero123plus,
      title={Zero123++: a Single Image to Consistent Multi-view Diffusion Base Model},
      author={Ruoxi Shi and Hansheng Chen and Zhuoyang Zhang and Minghua Liu and Chao Xu and Xinyue Wei and Linghao Chen and Chong Zeng and Hao Su},
      year={2023},
      eprint={2310.15110},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}
```
Tokens:
Raw