### Install vLLM for Inference

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Install the vLLM library to enable efficient inference. This command installs version 0.11.0.

```bash
pip install vllm==0.11.0
```

--------------------------------

### Run vLLM-Ascend Docker Container

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Starts a Docker container with necessary devices and volume mounts for vLLM-Ascend. Replace 'your container name' and ensure MODEL_PATH is correctly set.

```plain
NAME=your container name
MODEL_PATH=put your absolute model path here if you already have it locally.

docker run -itd --privileged --name=$NAME --net=host \
 --shm-size=1000g \
   --device /dev/davinci0 \
   --device /dev/davinci1 \
   --device /dev/davinci2\
   --device /dev/davinci3 \
   --device /dev/davinci4 \
   --device /dev/davinci5 \
   --device /dev/davinci6 \
   --device /dev/davinci7 \
   --device=/dev/davinci_manager \
   --device=/dev/hisi_hdc \
   --device /dev/devmm_svm \
   -v /usr/local/dcmi:/usr/local/dcmi \
    -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \
    -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \
    -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \
    -v /etc/ascend_install.info:/etc/ascend_install.info \
   -v /usr/local/sbin:/usr/local/sbin \
   -v /etc/hccn.conf:/etc/hccn.conf \
   -v $MODEL_PATH:$MODEL_PATH \
   quay.io/ascend/vllm-ascend:v0.11.0rc2 \
   bash

docker exec -u root -it $NAME bash
```

--------------------------------

### Install SGLang for Inference

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Install the SGLang library for inference. It is recommended to use the latest version with '-U'. A Docker image is also available.

```bash
pip install sglang -U
```

```bash
docker pull lmsysorg/sglang:latest
```

--------------------------------

### Start vLLM OpenAI API Server

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Launches the vLLM OpenAI-compatible API server for online inference. Configures model, parallelism, and network settings. Ensure taskset is available.

```bash
model_id=MedAIBase/AntAngelMed
taskset -c 0-23 python3 -m vllm.entrypoints.openai.api_server \
  --model $model_id \
  --max-num-seqs=200 \
  --tensor-parallel-size 4 \
  --data-parallel-size 2 \
  --enable_expert_parallel \
  --gpu_memory_utilization 0.97 \
  --served-model-name AntAngelMed \
  --max-model-len 32768 \
  --port 8080 \
  --enable-prefix-caching \
  --block-size 128 \
  --async-scheduling \
  --trust_remote_code
```

--------------------------------

### Pull vLLM-Ascend Docker Image

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Pulls the specified version of the vLLM-Ascend Docker image from Quay.io. Ensure you have Docker installed and configured.

```plain
docker pull quay.io/ascend/vllm-ascend:v0.11.0rc3
```

--------------------------------

### Start SGLang Inference Server

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Launch the SGLang server for inference, supporting BF16 and FP8 models. This command configures the server with ModelScope integration, attention backend, and tensor parallelism.

```bash
SGLANG_USE_MODELSCOPE=true python -m sglang.launch_server \
    --model-path $MODLE_PATH \
    --host 0.0.0.0 --port $PORT \
    --trust-remote-code \
    --attention-backend fa3 \
    --tensor-parallel-size 4 \
    --served-model-name AntAngelMed
```

--------------------------------

### vLLM Inference with AntAngelMed

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Perform inference using vLLM, supporting offline batched inference or an OpenAI-Compatible API Service. This example demonstrates direct inference with specified sampling parameters.

```python
from modelscope import AutoTokenizer
from vllm import LLM, SamplingParams

def main():
    model_path = "MedAIBase/AntAngelMed" # model_id or your_local_model_path
    tokenizer = AutoTokenizer.from_pretrained(model_path, trust_remote_code=True)
    sampling_params = SamplingParams(
        temperature=0.6,
        top_p=0.95,
        top_k=20,
        repetition_penalty=1.05,
        max_tokens=16384,
    )
    llm = LLM(
        model=model_path,
        trust_remote_code=True,
        dtype="bfloat16",
        tensor_parallel_size=4, 
    )

    prompt = "What should I do if I have a headache?" 
    messages = [
        {"role": "system", "content": "You are AntAngelMed, a helpfull medical assistant."},
        {"role": "user", "content": prompt},
    ]
    text = tokenizer.apply_chat_template(
        messages,
        tokenize=False,
        add_generation_prompt=True,
    )
    outputs = llm.generate([text], sampling_params)
    print(outputs[0].outputs[0].text)
  
if __name__ == "__main__":
    main()
```

--------------------------------

### SGLang Client Request for Chat Completions

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Send a chat completion request to the running SGLang server using cURL. This example demonstrates a simple user message.

```bash
curl -s http://localhost:${PORT}/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{"model": "auto", "messages": [{"role": "user", "content": "What should I do if I have a headache?"}]}'
```

--------------------------------

### Use AntAngelMed Chat Model with Hugging Face Transformers

Source: https://github.com/medaibase/antangelmed/blob/main/README.md

This snippet demonstrates how to load the AntAngelMed model and tokenizer from Hugging Face, format a chat prompt, generate a response, and decode it. Ensure you have the transformers library installed.

```plain
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "MedAIBase/AntAngelMed"  # model_id or your_local_model_path
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "What should I do if I have a headache?"
messages = [
    {"role": "system", "content": "You are AntAngelMed, a helpful medical assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt", return_token_type_ids=False).to(model.device)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=16384,
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
```

--------------------------------

### Send Chat Completion Request to API Server

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Example cURL command to send a chat completion request to the running vLLM API server. Adjust the model name and messages as needed.

```bash
curl http://0.0.0.0:8080/v1/chat/completions -d '{
    "model": "AntAngelMed",
    "messages": [
        {
            "role": "system",
            "content": "You are a helpful assistant."
        },
        {
            "role": "user",
            "content": "What should I do if I have a headache?"
        }
    ],
    "temperature": 0.6
}'
```

--------------------------------

### Hugging Face Transformers Chat Model Inference

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Use this snippet to perform chat-based inference with the AntAngelMed model using the Hugging Face Transformers library. Ensure you have the transformers library installed.

```python
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "MedAIBase/AntAngelMed"  # model_id or your_local_model_path
model = AutoModelForCausalLM.from_pretrained(
    model_name,
    device_map="auto",
    trust_remote_code=True,
)
tokenizer = AutoTokenizer.from_pretrained(model_name)

prompt = "What should I do if I have a headache?"
messages = [
    {"role": "system", "content": "You are AntAngelMed, a helpfull medical assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
model_inputs = tokenizer([text], return_tensors="pt", return_token_type_ids=False).to(model.device)
generated_ids = model.generate(
    **model_inputs,
    max_new_tokens=16384,
)
generated_ids = [
    output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
]
response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
print(response)
```

--------------------------------

### Configure vLLM-Ascend Environment Variables

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Sets essential environment variables for vLLM inference on Ascend hardware. These variables control memory allocation, parallelism, and device visibility.

```plain
export PYTORCH_NPU_ALLOC_CONF=expandable_segments:True
export HCCL_OP_EXPANSION_MODE="AIV"
export NPU_MEMORY_FRACTION=0.97
export TASK_QUEUE_ENABLE=1
export OMP_NUM_THREADS=100
export ASCEND_LAUNCH_BLOCKING=0
export ASCEND_RT_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
#You can use ModelScope mirror to speed up download:
export VLLM_USE_MODELSCOPE=true
```

--------------------------------

### Offline Inference with vLLM-Ascend

Source: https://github.com/medaibase/antangelmed/blob/main/Figures/README.md

Performs offline inference using a pre-trained model with vLLM. Requires transformers and vLLM libraries. Adjust model_path and sampling parameters as needed.

```python
from transformers import AutoTokenizer
from vllm import LLM, SamplingParams
model_path = "MedAIBase/AntAngelMed" # model_id or your_local_model_path
tokenizer = AutoTokenizer.from_pretrained(model_path)
sampling_params = SamplingParams(temperature=0.7, top_p=0.8, repetition_penalty=1.05, max_tokens=16384)
llm = LLM(model=model_path, 
    dtype='float16',
    tensor_parallel_size=4,                     
    gpu_memory_utilization=0.97,  
    enable_prefix_caching=True,  
    enable_expert_parallel=True,
    trust_remote_code=True)
prompt = "What should I do if I have a headache?"
messages = [
    {"role": "system", "content": "You are AntAngelMed, a helpful medical assistant."},
    {"role": "user", "content": prompt}
]
text = tokenizer.apply_chat_template(
    messages,
    tokenize=False,
    add_generation_prompt=True
)
outputs = llm.generate([text], sampling_params)
```

=== COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.