### Deploy with vLLM

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blame/0d8fc4f53e39a082b133993000902fbe1588ef36/config.json

Instructions for installing vLLM, starting the server, and querying it via the OpenAI-compatible API.

```bash
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "togethercomputer/RedPajama-INCITE-7B-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/RedPajama-INCITE-7B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'
```

--------------------------------

### Serve Model with vLLM

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base?local-app=vllm

Commands to install vLLM, start the inference server, and query it via the OpenAI-compatible API.

```bash
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "togethercomputer/RedPajama-INCITE-7B-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/RedPajama-INCITE-7B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'
```

```bash
# Install vLLM from pip:
pip install vllm
```

```bash
# Start the vLLM server:
vllm serve "togethercomputer/RedPajama-INCITE-7B-Base"
```

```bash
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/RedPajama-INCITE-7B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'
```

--------------------------------

### Install Transformers Library

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/colab

Install or upgrade the transformers library to ensure compatibility with the model.

```bash
!pip install -U transformers
```

--------------------------------

### Serve RedPajama-INCITE-7B-Base with SGLang

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blame/6ebc83dffdc83eddb0894dc535a453c3235ef45f/README.md

Install SGLang and launch the server to host the model for inference.

```bash
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "togethercomputer/RedPajama-INCITE-7B-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/RedPajama-INCITE-7B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'
```

--------------------------------

### Initialize Torch for GPU Inference

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blob/main/README.md?code=true

Basic import for GPU-based inference setup.

```python
import torch
```

--------------------------------

### Install Dependencies for GPU Inference

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blame/0d8fc4f53e39a082b133993000902fbe1588ef36/README.md

Install the required accelerate and bitsandbytes libraries for Int8 GPU inference.

```bash
pip install accelerate
pip install bitsandbytes
```

--------------------------------

### Initialize GPU Inference

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blob/main/README.md

Setup imports and version check for GPU inference, requiring at least 16GB of VRAM.

```python
import torch
import transformers
from transformers import AutoTokenizer, AutoModelForCausalLM

MIN_TRANSFORMERS_VERSION = '4.25.1'
```

--------------------------------

### Deploy with SGLang

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blame/0d8fc4f53e39a082b133993000902fbe1588ef36/config.json

Instructions for running the model using SGLang via pip or Docker containers.

```bash
# Install SGLang from pip:
pip install sglang
# Start the SGLang server:
python3 -m sglang.launch_server \
    --model-path "togethercomputer/RedPajama-INCITE-7B-Base" \
    --host 0.0.0.0 \
    --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/RedPajama-INCITE-7B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'
```

```bash
docker run --gpus all \
    --shm-size 32g \
    -p 30000:30000 \
    -v ~/.cache/huggingface:/root/.cache/huggingface \
    --env "HF_TOKEN=<secret>" \
    --ipc=host \
    lmsysorg/sglang:latest \
    python3 -m sglang.launch_server \
        --model-path "togethercomputer/RedPajama-INCITE-7B-Base" \
        --host 0.0.0.0 \
        --port 30000
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:30000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/RedPajama-INCITE-7B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'
```

--------------------------------

### Initialize and run inference on GPU

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blob/main/README.md

Uses 8-bit quantization and automatic device mapping for GPU inference. Requires a compatible environment with transformers installed.

```python
# check transformers version
assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'

# init
tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-INCITE-7B-Base")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-INCITE-7B-Base", device_map='auto', torch_dtype=torch.float16, load_in_8bit=True)

# infer
prompt = "Alan Turing is"
inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
input_length = inputs.input_ids.shape[1]
outputs = model.generate(
    **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True
)
token = outputs.sequences[0, input_length:]
output_str = tokenizer.decode(token)
print(output_str)
```

--------------------------------

### Serve Model with vLLM

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blame/3ff92e91a84b4bc88ceaea1fb67f37b1ae160a8f/special_tokens_map.json

Deploy the model using vLLM via pip or Docker and interact with the OpenAI-compatible API.

```bash
# Install vLLM from pip:
pip install vllm
# Start the vLLM server:
vllm serve "togethercomputer/RedPajama-INCITE-7B-Base"
# Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/completions" \
	-H "Content-Type: application/json" \
	--data '{
		"model": "togethercomputer/RedPajama-INCITE-7B-Base",
		"prompt": "Once upon a time,",
		"max_tokens": 512,
		"temperature": 0.5
	}'
```

```bash
docker model run hf.co/togethercomputer/RedPajama-INCITE-7B-Base
```

--------------------------------

### Initialize and Run Inference

Source: https://huggingface.co/togethercomputer/RedPajama-INCITE-7B-Base/blob/main/README.md

Checks the transformers version, loads the model and tokenizer, and performs text generation on a GPU.

```python
# check transformers version
assert transformers.__version__ >= MIN_TRANSFORMERS_VERSION, f'Please upgrade transformers to version {MIN_TRANSFORMERS_VERSION} or higher.'

# init
tokenizer = AutoTokenizer.from_pretrained("togethercomputer/RedPajama-INCITE-7B-Base")
model = AutoModelForCausalLM.from_pretrained("togethercomputer/RedPajama-INCITE-7B-Base", torch_dtype=torch.float16)
model = model.to('cuda:0')
# infer
prompt = "Alan Turing is"
inputs = tokenizer(prompt, return_tensors='pt').to(model.device)
input_length = inputs.input_ids.shape[1]
outputs = model.generate(
    **inputs, max_new_tokens=128, do_sample=True, temperature=0.7, top_p=0.7, top_k=50, return_dict_in_generate=True
)
token = outputs.sequences[0, input_length:]
output_str = tokenizer.decode(token)
print(output_str)
```