### Install vLLM Spark2_5 plugin from source Source: https://github.com/xhtoken/spark-x2.5/wiki/FAQ Alternative installation from the GitHub repository. Clone the repo and run pip install . in the plugin directory. ```bash git clone https://github.com/XHToken/Spark-plugin.git cd ./Spark-plugin && pip install . ``` -------------------------------- ### Install Spark Plugin Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Install the Spark plugin inside the container using uv. ```bash pip install uv uv venv ~/spark2_5 source ~/spark2_5/bin/activate git clone https://github.com/XHToken/Spark-plugin.git cd ./Spark-plugin uv pip install . ``` -------------------------------- ### Start vLLM Server Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Launch the vLLM server with the specified model configuration. ```bash vllm serve "/models/Spark-X2.5-4B" \ --port "30000" \ --trust-remote-code \ --served-model-name spark25 \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.7 \ --enable-prefix-caching \ --chat-template /models/Spark-X2.5-4B/chat_template.jinja ``` -------------------------------- ### Install Spark-MLX-LLM Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Clone and install the Spark-MLX-LLM repository with environment-specific dependencies. ```bash git clone https://github.com/XHToken/Spark-MLX-LLM.git cd Spark-MLX-LLM python3 -m venv .venv source .venv/bin/activate # Apple silicon python -m pip install -e . # Linux CPU python -m pip install -e '.[cpu]' # Linux with CUDA 12 python -m pip install -e '.[cuda12]' # Linux with CUDA 13 python -m pip install -e '.[cuda13]' ``` -------------------------------- ### Deploy vLLM on Ascend A3 Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Setup and run the vLLM container for Ascend A3 hardware. ```bash export IMAGE=quay.io/ascend/vllm-ascend:nightly-main-a3 docker pull "$IMAGE" export DEVICE0=/dev/davinci0 export DEVICE1=/dev/davinci1 export MODEL_CACHE="${HOME}/.cache" mkdir -p "$MODEL_CACHE" docker run --rm \ --name vllm-ascend \ --shm-size=1g \ --device "$DEVICE0" \ --device "$DEVICE1" \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v "$MODEL_CACHE:/root/.cache" \ -p 8000:8000 \ -it "$IMAGE" bash ``` -------------------------------- ### Launch Inference Server on Ascend NPU Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Start the API server on Ascend NPU hardware with the required device and volume mappings. ```bash docker run -it --rm -e ASCEND_USE_FIA=1 --network=host --ipc=host --shm-size=16g \ --device=/dev/davinci0 --device=/dev/davinci1 --device=/dev/davinci2 --device=/dev/davinci3 \ --device=/dev/davinci4 --device=/dev/davinci5 --device=/dev/davinci6 --device=/dev/davinci7 \ --device=/dev/davinci8 --device=/dev/davinci9 --device=/dev/davinci10 --device=/dev/davinci11 \ --device=/dev/davinci12 --device=/dev/davinci13 --device=/dev/davinci14 --device=/dev/davinci15 \ --device=/dev/davinci_manager \ --device=/dev/devmm_svm \ --device=/dev/hisi_hdc \ --volume /usr/local/sbin:/usr/local/sbin \ --volume /usr/local/Ascend/driver:/usr/local/Ascend/driver \ --volume /usr/local/Ascend/firmware:/usr/local/Ascend/firmware \ --volume /etc/ascend_install.info:/etc/ascend_install.info \ --volume /var/queue_schedule:/var/queue_schedule \ --volume ~/.cache/:/root/.cache/ \ --volume "$MODEL_PATH:/root/Spark-X2.5-4B:ro" \ --entrypoint=python \ "$SGLANG_IMAGE" \ -m sglang.launch_server \ --model-path /root/Spark-X2.5-4B \ --served-model-name spark2.5 \ --tool-call-parser spark25 \ --reasoning-parser qwen3 \ --tp-size 1 \ --mem-fraction-static 0.8 \ --context-length 1048576 \ --chat-template /root/Spark-X2.5-4B/chat_template.jinja \ --host 0.0.0.0 \ --port 30000 ``` -------------------------------- ### Deploy vLLM on Ascend 950DT Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Setup and run the vLLM container for Ascend 950DT hardware. ```bash export IMAGE=quay.io/ascend/vllm-ascend:nightly-main-a5 docker pull "$IMAGE" export MODEL_CACHE="${HOME}/.cache" mkdir -p "$MODEL_CACHE" docker run --rm \ --name vllm-ascend \ --net=host \ --shm-size=1g \ --device /dev/davinci0 \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/Ascend/driver/tools/hccn_tool:/usr/local/Ascend/driver/tools/hccn_tool \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v "$MODEL_CACHE:/root/.cache" \ -it "$IMAGE" bash ``` -------------------------------- ### Install vLLM Spark2_5 plugin via pip Source: https://github.com/xhtoken/spark-x2.5/wiki/FAQ Install the vllm-spark2_5-plugin into the same Python environment that runs vLLM. On Ascend, use --no-deps to avoid overwriting the vLLM-Ascend build. The official vllm/vllm-openai Docker image also requires this plugin. ```bash uv pip install vllm-spark2_5-plugin ``` -------------------------------- ### Deploy vLLM on Ascend A2 Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Setup and run the vLLM container for Ascend A2 hardware. ```bash export IMAGE=quay.io/ascend/vllm-ascend:nightly-main docker pull "$IMAGE" export DEVICE=/dev/davinci0 export MODEL_CACHE="${HOME}/.cache" mkdir -p "$MODEL_CACHE" docker run --rm \ --name vllm-ascend \ --shm-size=1g \ --device "$DEVICE" \ --device /dev/davinci_manager \ --device /dev/devmm_svm \ --device /dev/hisi_hdc \ -v /usr/local/dcmi:/usr/local/dcmi \ -v /usr/local/bin/npu-smi:/usr/local/bin/npu-smi \ -v /usr/local/Ascend/driver/lib64/:/usr/local/Ascend/driver/lib64/ \ -v /usr/local/Ascend/driver/version.info:/usr/local/Ascend/driver/version.info \ -v /etc/ascend_install.info:/etc/ascend_install.info \ -v "$MODEL_CACHE:/root/.cache" \ -p 8000:8000 \ -it "$IMAGE" bash ``` -------------------------------- ### Launch Inference Server on NVIDIA GPU Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Start the OpenAI-compatible API server on an NVIDIA GPU with a context length of 1,048,576 tokens. ```bash docker run --rm -it \ --gpus '"device=0"' \ --ipc=host \ -p 30000:30000 \ -v "$MODEL_PATH":/root/Spark-X2.5-4B:ro \ lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1 \ python -m sglang.launch_server \ --model-path /root/Spark-X2.5-4B \ --served-model-name spark2.5 \ --tool-call-parser spark25 \ --reasoning-parser qwen3 \ --tp-size 1 \ --mem-fraction-static 0.8 \ --context-length 1048576 \ --chat-template /root/Spark-X2.5-4B/chat_template.jinja \ --host 0.0.0.0 \ --port 30000 ``` -------------------------------- ### Set Model Path Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Define the absolute path to the Spark-X2.5-4B checkpoint before starting the container. ```bash export MODEL_PATH=/absolute/path/to/Spark-X2.5-4B ``` -------------------------------- ### Run Spark-X2.5 with lms CLI Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Commands to list available models and initiate a chat session with Spark-X2.5 using the lms CLI. ```bash # List available models lms ls # Run chat with Spark-X2.5 lms chat spark-x2.5 ``` -------------------------------- ### Run Spark-X2.5 with Ollama Source: https://github.com/xhtoken/spark-x2.5/wiki/FAQ Requires Ollama v0.34.1 or later. Both models have a single 'latest' tag with unquantized weights; for limited VRAM, pull an official GGUF quantization from Hugging Face instead. ```bash ollama run SparkLLM/Spark-X2.5-1.7B # or ollama run SparkLLM/Spark-X2.5-4B ``` -------------------------------- ### Deploy vLLM on NVIDIA GPU Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Run the official vLLM Docker image for NVIDIA GPU deployment. ```bash docker run --rm --gpus all \ --ipc=host \ -p 30000:30000 \ -v "$MODEL_PATH:/models/Spark-X2.5-4B:ro" \ vllm/vllm-openai:latest \ --model /models/Spark-X2.5-4B \ --port 30000 \ --trust-remote-code \ --served-model-name spark25 \ --tensor-parallel-size 1 \ --gpu-memory-utilization 0.7 \ --enable-prefix-caching \ --chat-template /models/Spark-X2.5-4B/chat_template.jinja ``` -------------------------------- ### Pull SGLang Image for Ascend NPUs Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Select the appropriate image based on your hardware (A3 or A2) and pull it. ```bash # A3 daily build export SGLANG_IMAGE=quay.io/ascend/sglang:main-cann9.0.0-a3 # A2 daily build (use this instead on A2 hardware) export SGLANG_IMAGE=quay.io/ascend/sglang:main-cann9.0.0-910b docker pull "$SGLANG_IMAGE" ``` -------------------------------- ### Run Inference via Client Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Send a chat completion request to the vLLM server using curl. ```bash curl -s http://127.0.0.1:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "spark25", "messages": [{"role": "user", "content": "安徽的省会在哪里？"}], "temperature": 1.0, "top_k": -1, "top_p": 0.95 }' ``` -------------------------------- ### Run Ollama Model Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Execute the Spark-X2.5 model using Ollama. ```bash ollama run SparkLLM/Spark-X2.5-1.7B ``` -------------------------------- ### Pull SGLang Image for NVIDIA GPUs Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Use the pre-built image compatible with the Spark-X2.5 runtime. ```bash docker pull lmsysorg/sglang:nightly-dev-cu13-20260827-20621aa1 ``` -------------------------------- ### Run MLX Inference Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Execute inference using the spark-mlx-generate command. ```bash spark-mlx-generate \ --device gpu \ --dtype bfloat16 \ --model XHToken/Spark-X2.5-1.7B \ --prompt "安徽的省会在哪里？" \ --max-tokens 512 \ --temp 0 ``` -------------------------------- ### Query Inference API Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md Send a chat completion request to the running server using curl. ```bash curl -s http://localhost:30000/v1/chat/completions \ -H "Content-Type: application/json" \ -d '{ "model": "spark2.5", "messages": [ { "role": "user", "content": "安徽的省会在哪里？" } ], "max_tokens": 131072, "temperature": 1, "top_k": -1, "top_p": 0.95, "repetition_penalty": 1, "presence_penalty": 0, "frequency_penalty": 0 }' ``` -------------------------------- ### Cite Spark-X2.5 Source: https://github.com/xhtoken/spark-x2.5/blob/main/README.md BibTeX citation for the Spark-X2.5 model series. ```bibtex @misc{sparkx2.5, title = {Spark-X2.5 4B&1.7B: Pushing the Limits of Agentic Capabilities in On-Device Models}, author = {SparkLLM Team}, year = {2026} } ``` -------------------------------- ### Disable thinking mode per request with chat_template_kwargs Source: https://github.com/xhtoken/spark-x2.5/wiki/FAQ Use this JSON snippet to disable thinking mode for a single request, which can prevent looping and improve stability in everyday chat. Set enable_thinking to false in chat_template_kwargs. ```json "chat_template_kwargs": {"enable_thinking": false} ``` === COMPLETE CONTENT === This response contains all available snippets from this library. No additional content exists. Do not make further requests.