### Download CodeT5 Checkpoints using gsutil

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Instructions for downloading CodeT5 pretrained and fine-tuned checkpoints using the gsutil command-line tool. Ensure gsutil is installed.

```bash
# pip install gsutil
cd your-cloned-codet5-path

gsutil -m cp -r "gs://sfr-codet5-data-research/pretrained_models" .
gsutil -m cp -r "gs://sfr-codet5-data-research/data" .
gsutil -m cp -r "gs://sfr-codet5-data-research/finetuned_models" .
```

--------------------------------

### Code Summarization with CodeT5-base-multi-sum

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Use this snippet to generate a code summary for a given Python function. Ensure you have the transformers library installed.

```python
from transformers import RobertaTokenizer, T5ForConditionalGeneration

if __name__ == '__main__':
    tokenizer = RobertaTokenizer.from_pretrained('Salesforce/codet5-base')
    model = T5ForConditionalGeneration.from_pretrained('Salesforce/codet5-base-multi-sum')

    text = """def svg_to_image(string, size=None):
    if isinstance(string, unicode):
        string = string.encode('utf-8')
        renderer = QtSvg.QSvgRenderer(QtCore.QByteArray(string))
    if not renderer.isValid():
        raise ValueError('Invalid SVG data.')
    if size is None:
        size = renderer.defaultSize()
        image = QtGui.QImage(size, QtGui.QImage.Format_ARGB32)
        painter = QtGui.QPainter(image)
        renderer.render(painter)
    return image"""

    input_ids = tokenizer(text, return_tensors="pt").input_ids

    generated_ids = model.generate(input_ids, max_length=20)
    print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
    # this prints: "Convert a SVG string to a QImage."
```

--------------------------------

### Load and Use InstructCodeT5+ 16B for Code Generation

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

Demonstrates loading the InstructCodeT5+ 16B model and tokenizer, setting up the device, and performing code generation. Requires `trust_remote_code=True` for larger models.

```python
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
import torch

checkpoint = "Salesforce/instructcodet5p-16b"
device = "cuda" # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(checkpoint,
                                              torch_dtype=torch.float16,
                                              low_cpu_mem_usage=True,
                                              trust_remote_code=True).to(device)

encoding = tokenizer("def print_hello_world():", return_tensors="pt").to(device)
encoding['decoder_input_ids'] = encoding['input_ids'].clone()
outputs = model.generate(**encoding, max_length=15)
print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

--------------------------------

### Run CodeT5 for Multi-Task Training

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Initiate multi-task training for the CodeT5-base model. This command is used when training across multiple tasks simultaneously.

```bash
python run_exp.py --model_tag codet5_base --task multi_task --sub_task none
```

--------------------------------

### Generate Programs for HumanEval Benchmark

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

Generate programs from CodeT5+ models for the HumanEval benchmark using nucleus sampling. Adjust model, temperature, and prediction count as needed.

```bash
model=instructcodet5p-16b
temp=0.2
max_len=800
pred_num=200
num_seqs_per_iter=2 # 25 for 350M and 770M, 10 for 2B, 8 for 6B, 2 for 16B on A100-40G

output_path=preds/${model}_T${temp}_N${pred_num}

mkdir -p ${output_path}
echo 'Output path: '$output_path
echo 'Model to eval: '$model

```

--------------------------------

### Load Finetuned Checkpoint in run_gen.py

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Specify the path to the downloaded finetuned checkpoint in the run_gen.py script. This allows the model to load pre-trained weights for inference.

```python
file = "CodeT5/finetuned_models/summarize_python_codet5_base.bin"
```

--------------------------------

### Run CodeT5 for Python Code Summarization

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Execute the CodeT5-base model for the code summarization task on Python data. Ensure the model tag, task, and sub-task arguments are correctly set.

```bash
python run_exp.py --model_tag codet5_base --task summarize --sub_task python
```

--------------------------------

### Instruction Finetuning CodeT5+ 16B

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

Finetune CodeT5+ 16B on instruction data using DeepSpeed for optimization. Ensure the instruction data path is correctly specified.

```bash
MODEL=Salesforce/codet5p-16b
SAVE_DIR=saved_models/instructcodet5p-16b

deepspeed instruct_tune_codet5p.py \
  --load $MODEL --save-dir $SAVE_DIR --instruct-data-path code_alpaca_20k.json \
  --fp16 --deepspeed deepspeed_config.json
```

--------------------------------

### Reproducing HumanEval Pass@1 Results

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

This command allows you to reproduce the reported Pass@1 results for the InstructCodeT5+ 16B model on the HumanEval dataset.

```bash
evaluate_functional_correctness humaneval/instructcodet5p-16b_T0.2_N200.jsonl
```

--------------------------------

### Perform Code Summarization with CodeT5+ 220M Bimodal Model

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

Illustrates using the CodeT5+ 220M bimodal model for code summarization. The model takes code as input and generates a summary.

```python
from transformers import AutoModel, AutoTokenizer

checkpoint = "Salesforce/codet5p-220m-bimodal"
device = "cuda"  # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModel.from_pretrained(checkpoint, trust_remote_code=True).to(device)

code = """
def svg_to_image(string, size=None):
    if isinstance(string, unicode):
        string = string.encode('utf-8')
        renderer = QtSvg.QSvgRenderer(QtCore.QByteArray(string))
    if not renderer.isValid():
        raise ValueError('Invalid SVG data.')
    if size is None:
        size = renderer.defaultSize()
        image = QtGui.QImage(size, QtGui.QImage.Format_ARGB32)
        painter = QtGui.QPainter(image)
        renderer.render(painter)
    return image"""

input_ids = tokenizer(code, return_tensors="pt").input_ids.to(device)

generated_ids = model.generate(input_ids, max_length=20)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
# Convert a string of SVG data to an image.
```

--------------------------------

### Code Generation with CodeT5-base

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

This snippet demonstrates how to use a CodeT5-base model for generating a code span. It requires the transformers library.

```python
from transformers import RobertaTokenizer, T5ForConditionalGeneration

tokenizer = RobertaTokenizer.from_pretrained('Salesforce/codet5-base')
model = T5ForConditionalGeneration.from_pretrained('Salesforce/codet5-base')

text = "def greet(user): print(f'hello <extra_id_0>!')"
input_ids = tokenizer(text, return_tensors="pt").input_ids

# simply generate one code span
generated_ids = model.generate(input_ids, max_length=8)
print(tokenizer.decode(generated_ids[0], skip_special_tokens=True))
# this prints "{user.username}"
```

--------------------------------

### Reproduce Results with Finetuned Checkpoints

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Modify the run_exp.sh script to disable training and evaluation, enabling only testing. This is used to reproduce results using pre-trained checkpoints.

```bash
# Remove the --do_train --do_eval --do_eval_bleu and reserve only --do_test at [here](https://github.com/salesforce/CodeT5/blob/5b37c34f4bbbfcfd972c24a9dd1f45716568ecb5/sh/exp_with_args.sh#L84)
```

--------------------------------

### Implement Data Reading Function in utils.py

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Implement a function in utils.py to read your custom dataset. This function should be compatible with the existing data loading mechanisms.

```python
# add your data path and the function to read in utils.py ([here](https://github.com/salesforce/CodeT5/blob/5bb41e21b07fee73f310476a91ded00e385290d7/utils.py#L103) and [here](https://github.com/salesforce/CodeT5/blob/5bb41e21b07fee73f310476a91ded00e385290d7/utils.py#L149))
```

--------------------------------

### Implement Read Function in _utils.py

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Implement the data reading function in _utils.py, following the pattern of existing read functions. This is for custom data loading during fine-tuning.

```python
# The read function can be implemented in _utils.py similar to [this one](https://github.com/salesforce/CodeT5/blob/aaf9c4a920c4986abfd54a74f5456b056b6409e0/_utils.py#L213)
```

--------------------------------

### Evaluating Pass@k with HumanEval

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

This section details how to evaluate the generated code using the HumanEval dataset. It involves processing predictions and then running the functional correctness evaluation.

```bash
output_path=preds/instructcodet5p-16b_T0.2_N200

echo 'Output path: '$output_path
python process_preds.py --path ${output_path} --out_path ${output_path}.jsonl

evaluate_functional_correctness ${output_path}.jsonl
```

--------------------------------

### Text-to-Code Retrieval Evaluation (Bimodal Model)

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

This script evaluates the CodeT5+ bimodal model for text-to-code retrieval, including a matching decoder for reranking. Adjust `top_k` to control the number of candidates for reranking.

```bash
# LANG choices: ruby javascript go python java php AdvTest cosqa
LANG=ruby
BS=256
CODE_LEN=360
TEXT_LEN=64
TOPK=32
MODEL_NAME=Salesforce/codet5p-220m-bimodal
DATA_DIR=/path/to/data

TRG_DIR=saved_models/${LANG}/codet5p_220m_bimodal_TL${TEXT_LEN}_CL${CODE_LEN}_top${TOPK}
mkdir -p $TRG_DIR
echo 'Target dir: '$TRG_DIR

python eval_match_retrieval.py --model_name $MODEL_NAME --lang $LANG --output_dir $TRG_DIR \
  --data_dir $DATA_DIR --max_text_len $TEXT_LEN --max_code_len $CODE_LEN --batch_size $BS --top_k $TOPK
```

--------------------------------

### Add Custom Task in configs.py

Source: https://github.com/salesforce/codet5/blob/main/CodeT5/README.md

Define a new task by adding it to the configurations in configs.py. This is a prerequisite for fine-tuning on a custom dataset.

```python
# Add your own task and sub_task in configs.py ([here](https://github.com/salesforce/CodeT5/blob/d27512d23ba6130e089e571d8c3e399760db1c31/configs.py#L11))
```

--------------------------------

### Extract Code Embeddings with CodeT5+ 110M

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

Shows how to load the CodeT5+ 110M embedding model and tokenizer to extract code embeddings. The output embedding is a 256-dimensional vector.

```python
from transformers import AutoModel, AutoTokenizer

checkpoint = "Salesforce/codet5p-110m-embedding"
device = "cuda"  # for GPU usage or "cpu" for CPU usage

tokenizer = AutoTokenizer.from_pretrained(checkpoint, trust_remote_code=True)
model = AutoModel.from_pretrained(checkpoint, trust_remote_code=True).to(device)

inputs = tokenizer.encode("def print_hello_world():\tprint('Hello World!')", return_tensors="pt").to(device)
embedding = model(inputs)[0]
print(f'Dimension of the embedding: {embedding.size()[0]}, with norm={embedding.norm().item()}')
# Dimension of the embedding: 256, with norm=1.0
```

--------------------------------

### Text-to-Code Retrieval Evaluation (Embedding Model)

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

This script evaluates the CodeT5+ embedding model for text-to-code retrieval. Ensure you have downloaded and preprocessed the required datasets.

```bash
# LANG choices: ruby javascript go python java php AdvTest cosqa
LANG=ruby
BS=256
CODE_LEN=360
TEXT_LEN=64
MODEL_NAME=Salesforce/codet5p-110m-embedding
DATA_DIR=/path/to/data

TRG_DIR=saved_models/${LANG}/codet5p_110m_embedding_TL${TEXT_LEN}_CL${CODE_LEN}
mkdir -p $TRG_DIR
echo 'Target dir: '$TRG_DIR

python eval_contrast_retrieval.py --model_name $MODEL_NAME --lang $LANG --output_dir $TRG_DIR \
  --data_dir $DATA_DIR --max_text_len $TEXT_LEN --max_code_len $CODE_LEN --batch_size $BS
```

--------------------------------

### Parallel Code Generation Script

Source: https://github.com/salesforce/codet5/blob/main/CodeT5+/README.md

This script distributes code generation tasks across multiple GPUs to speed up the process. It iterates through a specified number of GPUs, assigning a range of problems to each.

```bash
index=0
gpu_num=8
for ((i = 0; i < $gpu_num; i++)); do
  start_index=$((i * 21))
  end_index=$(((i + 1) * 21))

  gpu=$((i))
  echo 'Running process #' ${i} 'from' $start_index 'to' $end_index 'on GPU' ${gpu}
  ((index++))
  (
    CUDA_VISIBLE_DEVICES=$gpu python generate_codet5p.py --model Salesforce/${model} \
      --start_index ${start_index} --end_index ${end_index} --temperature ${temp} \
      --num_seqs_per_iter ${num_seqs_per_iter} --N ${pred_num} --max_len ${max_len} --output_path ${output_path}
  ) &
  if (($index % $gpu_num == 0)); then wait; fi
done
```

--------------------------------

### CodeRL Citation

Source: https://github.com/salesforce/codet5/blob/main/README.md

Citation details for the CodeRL paper.

```bibtex
@inproceedings{
    le2022coderl,
    title={CodeRL: Mastering Code Generation through Pretrained Models and Deep Reinforcement Learning},
    author={Le, Hung and Wang, Yue and Gotmare, Akhilesh Deepak and Savarese, Silvio and Hoi, Steven C. H.},
    booktitle={NeurIPS},
    year={2022}
}
```

--------------------------------

### CodeT5+ Citation

Source: https://github.com/salesforce/codet5/blob/main/README.md

Citation details for the CodeT5+ paper.

```bibtex
@article{
    wang2023codet5plus,
    title={CodeT5+: Open Code Large Language Models for Code Understanding and Generation},
    author={Wang, Yue and Le, Hung and Gotmare, Akhilesh Deepak and Bui, Nghi D.Q. and Li, Junnan and Hoi, Steven C. H.},
    journal={arXiv preprint},
    year={2023}
}
```

--------------------------------

### CodeT5 Citation

Source: https://github.com/salesforce/codet5/blob/main/README.md

Citation details for the CodeT5 paper.

```bibtex
@inproceedings{
    wang2021codet5,
    title={CodeT5: Identifier-aware Unified Pre-trained Encoder-Decoder Models for Code Understanding and Generation}, 
    author={Yue Wang, Weishi Wang, Shafiq Joty, Steven C.H. Hoi},
    booktitle={EMNLP},
    year={2021},
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.