LitServe

https://lightning.ai/docs/litserve

Admin

LitServe is a flexible, high-throughput serving engine for AI models built on FastAPI, featuring...

Tokens:242

Snippets:4

Trust Score:9.8

Update:11 months ago

Show doc for...

Tokens:

Raw

### Install LitServe

Source: https://lightning.ai/docs/litserve/index

Installs the LitServe package using pip. This is the first step to begin using LitServe for serving AI models.

```bash
pip install litserve
```

--------------------------------

### Toy Example: SimpleLitAPI with LitServe

Source: https://lightning.ai/docs/litserve/index

Demonstrates a basic LitServe API implementation. It defines a LitAPI class with setup, request decoding, prediction, and response encoding methods, then runs it using LitServer with GPU acceleration.

```python
import litserve as ls

class SimpleLitAPI(ls.LitAPI):
    def setup(self, device):
        self.model1 = lambda x: x**2
        self.model2 = lambda x: x**3

    def decode_request(self, request):
        return request["input"]

    def predict(self, x):
        squared = self.model1(x)
        cubed = self.model2(x)
        output = squared + cubed
        return {"output": output}

    def encode_response(self, output):
        return {"output": output}

if __name__ == "__main__":
    api = SimpleLitAPI()
    server = ls.LitServer(api, accelerator="gpu")
    server.run(port=8000)
```

Tokens:

Raw