### Create Conda Environment for SemDeDup

Source: https://github.com/facebookresearch/semdedup/blob/main/README.md

Use the provided YAML file to create and activate a conda environment for SemDeDup. This is the recommended setup procedure.

```bash
conda env create -n semdedup --file semdedup_conda_env.yml 
 conda activate semdedup
```

--------------------------------

### Sort Clusters using Python Script

Source: https://github.com/facebookresearch/semdedup/blob/main/README.md

Use this script to sort clusters after modifying the configuration file. It submits a job using submitit.

```bash
python sort_clusters.py --confg-file "/configs/openclip/clustering_configs.yaml" \
                            --partition <partition> \
                            --num-tasks 10 \
                            --cpus-per-task 5 \
                            --timeout 300 \

```

```python
from sort_clusters import assign_and_sort_clusters

assign_and_sort_clusters(
    data=emb_memory,
    paths_list=paths_memory,
    sim_metric=params["sim_metric"],
    keep_hard=params["keep_hard"],
    kmeans_with_cos_dist=params["kmeans_with_cos_dist"],
    save_folder=params["save_folder"],
    sorted_clusters_file_loc=params["sorted_clusters_file_loc"],
    cluster_ids=range(0, params["ncentroids"]),
    logger=logger,
) 

```

--------------------------------

### Run K-Means Clustering via Command Line

Source: https://github.com/facebookresearch/semdedup/blob/main/README.md

Execute the `compute_centroids.py` script to perform K-Means clustering on precomputed embeddings. Modify the configuration file and specify partition, GPU, and CPU resources. This script utilizes `submitit` for job submission.

```bash
python compute_centroids.py --confg-file "/configs/openclip/clustering_configs.yaml" \
                            --partition <partition> \
                            --ngpus 1 \
                            --cpus-per-task 10 \
                            --timeout 300 \

```

--------------------------------

### Run SemDeDup Job using Submitit

Source: https://github.com/facebookresearch/semdedup/blob/main/README.md

This script launches jobs on multiple nodes using submitit for the SemDeDup process. Modify the semdedup_configs.yaml file before running. Each node processes parts of the clusters with parallelized tasks.

```bash
python submit_semdedup_job.py  --config-file "semdedup_configs.yaml" \
                               --eps-list <eps-list> \
                               --partition <partition> \
                               --nodes 8 \
                               --tasks-per-node 20 \
                               --cpus-per-task 4 \

```

--------------------------------

### Run K-Means Clustering via Python Script

Source: https://github.com/facebookresearch/semdedup/blob/main/README.md

This Python code snippet demonstrates how to compute K-Means centroids programmatically. It loads parameters from a YAML configuration file, initializes memmap arrays for embeddings and paths, and then calls the `compute_centroids` function. Ensure the configuration file is correctly set up.

```python
import yaml
import random
import numpy as np
import logging
from clustering.clustering import compute_centroids


logger = logging.getLogger(__name__) 
logger.addHandler(logging.StreamHandler())

confg_file = "configs/openclip/clustering_configs.yaml"
## -- Load kmeans clustering parameters from configs file
with open(confg_file, 'r') as y_file:
    params = yaml.load(y_file, Loader=yaml.FullLoader)

## -- Fix the seed
SEED = params['seed']
random.seed(SEED)
emb_memory_loc = params['emb_memory_loc'] 
paths_memory_loc = params['paths_memory_loc'] 
dataset_size = params['dataset_size'] 
emb_size = params['emb_size'] 
path_str_type = params['path_str_type']

emb_memory = np.memmap(emb_memory_loc, dtype='float32', mode='r', shape=(dataset_size, emb_size))
paths_memory = np.memmap(paths_memory_loc, dtype=path_str_type, mode='r', shape=(dataset_size,))

compute_centroids(
    data=emb_memory,
    ncentroids=params['ncentroids'],
    niter=params['niter'],
    seed=params['seed'],
    Kmeans_with_cos_dist=params['Kmeans_with_cos_dist'],
    save_folder=params['save_folder'],
    logger=logger,
    verbose=True,
)
```

--------------------------------

### Compute and Store Embeddings with get_embeddings

Source: https://github.com/facebookresearch/semdedup/blob/main/README.md

This template code demonstrates how to use the `get_embeddings` function to compute and store embeddings from a pretrained model. Ensure your dataloader returns data, paths, and indices, and that memmap arrays are correctly initialized for embeddings and paths.

```python
from compute_pretrained_embeddings import get_embeddings

model = ...
dataloader = ...

path_str_type = ...
emb_memory_loc = ...
paths_memory_loc = ...
dataset_size = ...
emb_size = ...
emb_array = np.memmap(emb_memory_loc, dtype='float32', mode='w+', shape=(dataset_size, emb_size))
path_array = np.memmap(emb_memory_loc, dtype=path_str_type, mode='w+', shape=(dataset_size,)) # Note: This line seems to have a typo, should likely be paths_memory_loc

get_embeddings(model, dataloader, emd_memmap, paths_memmap)
```

--------------------------------

### Extract Deduplicated Data Paths

Source: https://github.com/facebookresearch/semdedup/blob/main/README.md

Use this function to extract the paths of deduplicated data after running SemDeDup. Ensure the paths to sorted clusters and SemDeDup tables are correct.

```python
from extract_dedup_data import extract_pruned_data

output_txt_path = ...
semdedup_pruning_tables_path = ...
sorted_clusters_path = ...
eps = ...
num_clusters = ...
extract_pruned_data(sorted_clusters_path, semdedup_pruning_tables_path, eps, num_clusters, output_txt_path, retreive_kept_samples=True)

```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.