Migrate to vLLM

[AnkushMalaker]

We have configs, tests and code with ollama, but I believe using vLLM is a better option.
It allows more freedom in configuration and deployment.
Ollama, as I understand is "good for starters".
I'd love if someone could link comprehensive comparisons, otherwise happy to get this migration going.

[linear]

CHR-66

[01PrathamS]

can i take this up, don't know how to make around and how to make around without gpu (if needed at all - don't know).

[thestumonkey]

be aware that this has different setup challenges on different hardware

[AnkushMalaker]

Hey @01PrathamS
I'm copying what I said on discord for broader visibility

I would like to know how docker hub models work as suggested by @0xrushi
Then, I would like to hear thoughts on how vllm would be vs docker hub models
We already use docker so it will simplify setup greatly
Any downsides to docker
Then we go from there

Let me know any thoughts.

[0xrushi]

vllm is hard, it won't work on amd/ even some nvidia gpus.
also vllm for what? we can have it decoupled into config.yml with openai base url

[AnkushMalaker]

@0xrushi
I just want to move away from ollama to a more flexible and performant inference engine.

Llama.cpp, vllm, anything works. Yes I agree we should de couple from config and only have the API url there