When I'm writing a script that requires some kind of call to an LLM, I use this: https://github.com/simonw/llm.

This is of course cross-platform and works with both models accessible through an API and local ones.

I'm afraid this might not solve your problem though, as this is not an out of the box solution, it requires the user to either provide their own API key or to install Ollama and wire it up on their own.

I've been working on a more unixy version of his tool I call llcat. Composable, stateless, agnostic, and generic:

https://github.com/day50-dev/llcat

It might help things get closer..

It's under 2 days old and it's already really fundamentally changing how I do things.

Also for edge running look into the LFM 2.5 class of models: https://huggingface.co/LiquidAI/LFM2.5-1.2B-Instruct

I love this concept. Looks great, I will definitely check it out.

Please use it and give me feedback. I'm going to give a lightning talk on it tonight at sfvlug

I think this is definitely a step in the right direction, and is exactly the kind of answer I was looking for. Thank you!

`llm` gives my tool a standard bin to call to invoke completions, and configuring and managing it is the user's responsibility.

If more tools started expecting something like this, it could become a defacto standard. Then maybe the OS would begin to provide it.

Holy shit.