You're doing AI wrong.
Visiting r/LocalLLaMa shows people cutting down model fidelity to get faster and faster inference (tokens per second of output).
You don't need the LLM to output data faster than you can read - because following the model's "thought"-process is how you stop an LLM from making time consuming mistakes (thus negating that high output speed) as well as keeping yourself both in-the-know of what's happening to your system & code, as well as you learning (!) things from the LLM.
A few weeks ago I started using a local LLM to not only aid in software development, but also system maintenance. While giving the LLM a task I also sit and "hand-hold" it like I would a junior. I go in and interrupt if I see that the thought process is going in the wrong direction, and I also learn from the investigations the LLM makes in areas I'm not familiar with myself.
The utility in "one-shotting" (as it's known) work with LLMs seems counter productive. If it generates something that works, you don't know how. If it generates something that doesn't work you've wasted time and resources.
The LLM is your very eager, and very junior, aid. Outsource repetitive and basic tasks, and work together with it on the things that are important.
I have no boss telling me I must use AI. I decide what I do and how I do it (for these projects, I have other ones at customers where I don't) - and I wouldn't use tools that don't have a net-positive contribution to my productivity.
Local LLMs do, at least for all tasks that have a very binary works/doesn't work completion control point.
Just don't treat them as knowledge databases. That they aren't. They're workers, like you, that need to produce and read guidelines, documentations and follow well-written plans.
@troed What system maintenance tasks do you use them successfully for?
@h4890 A typical one would be that I've recently started learning Kubernetes (better) due to setting up the Element ESS CE stack Matrix server. I've run another Matrix server for years, but using Docker Compose. When I've now wanted to use some of the bridges that have published docker compose instructions but with k3s I've asked my local LLM to plan the implementation of the custom charts/deployements needed, when they look I tell it to proceed. When it has brought them up I tell it to check if logs look "sane" etc.
These tasks are very ok/nok and I'm learning k8s at the same time as the LLM "thinks". I also have enough general knowledge in hosting on Linux that I can catch things that seem off (because they do happen) at which point I interrupt the model and nudge it in another direction.
The time saving is substantial - not just because the model is able to "read" data a lot faster than I can, but also that my own loop from starting to learn the subject until I have a good grasp of the relevant technologies becomes a lot shorter. Asking the model to start planning a task also helps in getting over the initial high-friction "oh I need to learn how to do _this_" or even general procrastination.
