openjev-sglang

# openjev-sglang

A server implementing the [TypeSafe/Jev HTTP API](https://docs.typesafe.ai/api)
with **Qwen3.6-35B-A3B on SGLang**.



Each container one B200 with SGLang **0.5.19's Rust frontend**,
radix caching, and **breakable prefill CUDA graphs**. A separate Python API process
uses FastAPI, uvloop, the Rust-backed HF tokenizer, and pooled asynchronous HTTP
connections to SGLang on localhost. CUDA dependencies stay in SGLang's container;
`uv sync` on your laptop installs only the API, deployment tools, and tests.

## Run on Modal

```sh
uv sync
# Only if you haven't authenticated Modal on this machine:
uv run modal setup

# Start a temporary Server, run actual inference checks, then shut it down:
uv run modal run modal_app.py

# Deploy a stable public endpoint:
uv run modal deploy modal_app.py
```

The deployment prints a `https://...us-west.modal.direct` URL. It uses a
[Modal Server](https://modal.com/docs/guide/servers), `unauthenticated=True`,
`routing_region="us-west"`, and `compute_region=["us-west", "us-central", "us"]`.
Autoscaling has no explicit container cap and scales to zero after five idle minutes.
Set `min_containers=1` in `modal_app.py` to keep a B200 warm.
If SGLang exits unexpectedly, the API exits too. The Modal launcher watches the
API and exits the container so Modal can replace it, rather than leaving a live
HTTP process with a dead inference backend. Normal shutdown disarms both watchers.

Cache warmups also request one unused token probability to avoid SGLang's
[mixed-logprob batch crash](https://github.com/sgl-project/sglang/issues/34719).
This keeps warmups and scoring requests batch-compatible without patching SGLang.

The first build imports a large SGLang image. The first GPU start also downloads
weights and compiles/captures kernels. M