Main Area and Open Discussion > Living Room

AI Coding Assistants (Who uses them and which)

<< < (6/8) > >>

Shades:
By the way, in this environment there is is still a i7 CPU 1st generation active as a database server. It has 32 GByte of RAM and SSDs. Yes, it is even more ancient that your second generation i7 system, but it doesn't feel slow in anything it needs to do. So my boss doesn't want to change it, mainly because of (grandfathered) Oracle licensing-costs and Oracle being slowly phased out here.

So, ancient computer or not, as long as you don't game on those computers, they are still quite useful. And energy costs are not that big of a concern here in this part of South America, where the whole country runs on hydro-power.

Deozaan:
I forgot to mention I have 16 GB RAM in this machine.

I tried it on CPU, CUDA 11 and 12 and Vulkan and I get similar results for all of them in a form looking very much like this:


--- Code: Text ---11012.227 s   Starting...←[2m2024-10-19T03:54:38.253368Z←[0m ←[33m WARN←[0m ←[2mllama_cpp_server::supervisor←[0m←[2m:←[0m ←[2mcrates\llama-cpp-server\src\supervisor.rs←[0m←[2m:←[0m←[2m98:←[0m llama-server <embedding> exited with status code -1073741795, args: `Command { std: "D:\\Apps\\tabby_x86_64-windows-msvc\\llama-server.exe" "-m" "C:\\Users\\Deozaan\\.tabby\\models\\TabbyML\\Nomic-Embed-Text\\ggml\\model.gguf" "--cont-batching" "--port" "30888" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--embedding" "--ubatch-size" "4096", kill_on_drop: true }`←[2m2024-10-19T03:54:38.253838Z←[0m ←[33m WARN←[0m ←[2mllama_cpp_server::supervisor←[0m←[2m:←[0m ←[2mcrates\llama-cpp-server\src\supervisor.rs←[0m←[2m:←[0m←[2m110:←[0m <embedding>: warning: not compiled with GPU offload support, --gpu-layers option will be ignored←[2m2024-10-19T03:54:38.255312Z←[0m ←[33m WARN←[0m ←[2mllama_cpp_server::supervisor←[0m←[2m:←[0m ←[2mcrates\llama-cpp-server\src\supervisor.rs←[0m←[2m:←[0m←[2m110:←[0m <embedding>: warning: see main README.md for information on enabling GPU BLAS support
I guess I'll try LM Studio next.

Deozaan:
Hmmm... LM Studio won't work for me either. After poking around in the various tabs of the GUI it seems my CPU is the problem. 😭

Shades:
Hmmm... LM Studio won't work for me either. After poking around in the various tabs of the GUI it seems my CPU is the problem. 😭

[ Invalid Attachment ]
-Deozaan (October 18, 2024, 11:32 PM)
--- End quote ---
Too bad about your CPU.

In all honesty, I did not expect that, as I have used both Tabby and LM studio on a computer with an AMD APU (9700), same GPU and 16 GB RAM. That APU was designed in the 'bulldozer'-era of AMD, but made to fit on the AM4 socket. That era may not be 2nd gen i7, but isn't that far off either. Tabby and LM Studio worked both on that CPU, but it struggled when more than 1 person was using the LLM functionality. Hence I moved it to the next best (or should I say worst) thing, which was the computer with the i3 10100F CPU. Since then, Tabby is "smooth" sailing" with 5 people accessing it.

Here in Paraguay it isn't that hard to find motherboards and CPUs from the 10 gen for cheap. It is also not really a problem to get 13th and 14th gen gear, but that is more expensive than what it costs in the U.S. or Europe. If I remember my parts prices correctly, that i3 computer would have costed about 425 to 450 USD, with the GPU as the most expensive part.

Perhaps it is an option to get/trade your computer (without GPU) for another older computer with a 4th/5th/6th gen CPU for cheap? Refurbished gear or something from a company that dumped their written-off gear at a computer thrift store? For getting your feet wet with LLMs/AI  that could be useful, while also not breaking the bank.

Deozaan:
I don't really have the budget for upgrading my PC. Maybe I should get a $15 microcontroller and play around with TinyML.

But that's not really an LLM, so I guess I'm better off just accessing some free tier of ChatGPT.

Navigation

[0] Message Index

[#] Next page

[*] Previous page