### Run OpenAI API Inference for Code Completion

Source: https://github.com/amazon-science/cceval/blob/main/README.md

Utilize this script to perform inference using OpenAI models via their API. Ensure the `openai` library is installed and API keys are configured.

```bash
export model=gpt-3.5-turbo-0125 
export language=python
export task=line_completion_rg1_unixcoder_cosine_sim
export output_dir=./tmp/crosscodeeval_openai_testrun/
python scripts/openai_inference.py \
  --task $task \
  --language $language \
  --model $model \
  --output_dir $output_dir \
  --use_crossfile_context 

```

--------------------------------

### Run vLLM Inference for Code Completion

Source: https://github.com/amazon-science/cceval/blob/main/README.md

Use this script for fast and distributed inference with publicly available models like StarCoder via vLLM. Ensure vLLM is installed and necessary environment variables are set.

```bash
export gpus=2
export model=bigcode/starcoder2-3b
export language=python
export task=line_completion_rg1_unixcoder_cosine_sim
export output_dir=./tmp/crosscodeeval_testrun/
python scripts/vllm_inference.py \
  --tp $gpus \
  --task $task \
  --language $language \
  --model $model \
  --output_dir $output_dir \
  --use_crossfile_context 
```

--------------------------------

### Run Non-vLLM Inference with Accelerate

Source: https://github.com/amazon-science/cceval/blob/main/README.md

Execute greedy evaluation using Salesforce/codegen-350M-mono with cross-file context via the `accelerate` library. Configure `accelerate` first and ensure the tree-sitter library is built.

```bash
export model_type=codelm_cfc # or codelm for no cross-file context eval
export model_name=Salesforce/codegen-350M-mono
export language=python
export ts_lib=./build/${language}-lang-parser.so
export dtype=bf16 # or fp16
export prompt_file=./data/crosscodeeval_data/${language}/line_completion_rg1_unixcoder_cosine_sim.jsonl # or other options in the dir, which corresponds to different retrieval methods and/or retrieval settings
export max_seq_length=2048
export cfc_seq_length=512 
export batch_size=16 # reduce for larger models
export output_dir=./tmp/crosscodeeval_testrun/

accelerate launch eval.py \
        --model_type $model_type \
        --model_name_or_path $model_name \
        --cfc_seq_length $cfc_seq_length \
        --prompt_file $prompt_file \
        --gen_length 50 \
        --max_seq_length $max_seq_length \
        --batch_size $batch_size \
        --output_dir $output_dir \
        --dtype $dtype \
        --num_return_sequences 1 \
        --overwrite_cache True \
        --ts_lib $ts_lib \
        --language $language
```

--------------------------------

### Configure Sampling for Non-vLLM Inference

Source: https://github.com/amazon-science/cceval/blob/main/README.md

Add these arguments to the non-vLLM inference script to enable sampling-based generation instead of greedy decoding. Adjust `top_p` and `temperature` for different sampling behaviors.

```bash
--do_sample \
        --top_p 0.95 \
        --temperature 0.2 \
        --num_return_sequences 5 \

```

--------------------------------

### Calculate Metrics for Code Completion

Source: https://github.com/amazon-science/cceval/blob/main/README.md

Compute the final evaluation metrics after generating code completions. This script requires the prompt file, output directory, and language-specific tree-sitter library.

```bash
export language=python
export ts_lib=./build/${language}-lang-parser.so; 
export task=line_completion_oracle_unixcoder_cosine_sim
export prompt_file=./data/${language}/${task}.jsonl 
export output_dir=./tmp/crosscodeeval_testrun/;  
python scripts/eval.py \
  --prompt_file $prompt_file \
  --output_dir $output_dir \
  --ts_lib $ts_lib \
  --language $language \
  --only_compute_metric
```

--------------------------------

### CrossCodeEval Citation

Source: https://github.com/amazon-science/cceval/blob/main/README.md

BibTeX entry for citing the CrossCodeEval paper. Use this in your academic publications.

```bibtex
@inproceedings{ding2023crosscodeeval,
    title={CrossCodeEval: A Diverse and Multilingual Benchmark for Cross-File Code Completion}, 
    author={Yangruibo Ding and Zijian Wang and Wasi Uddin Ahmad and Hantian Ding and Ming Tan and Nihal Jain and Murali Krishna Ramanathan and Ramesh Nallapati and Parminder Bhatia and Dan Roth and Bing Xiang},
    year={2023},
    booktitle={Thirty-seventh Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
    url={https://arxiv.org/pdf/2310.11248.pdf}
}
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.