AI Coding Assistants (Who uses them and which)

[Shades]

Sorry, I thought I posted more relevant details.

My machine is ancient. Core i7 2700K. GTX 670 (6GB). 👴

I installed the nVidia tools and it let me install Cuda 12.x but I don't know if my GPU supports that.

The command I'm trying to run is very slightly different from yours (StarCoder-1B instead of 3B), taken from the Windows Installation documentation:

Code: Text [Select]

1. .\tabby.exe serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device cuda

But after counting up for about 10 seconds it starts spitting out messages like the following every few seconds:

Code: Text [Select]

1. 10.104 s   Starting...←[2m2024-10-18T08:32:25.331474Z←[0m ←[33m WARN←[0m ←[2mllama_cpp_server::supervisor←[0m←[2m:←[0m ←[2mcrates\llama-cpp-server\src\supervisor.rs←[0m←[2m:←[0m←[2m98:←[0m llama-server <embedding> exited with status code -1073741795, args: `Command { std: "D:\\Apps\\tabby_x86_64-windows-msvc-cuda122\\llama-server.exe" "-m" "C:\\Users\\Deozaan\\.tabby\\models\\TabbyML\\Nomic-Embed-Text\\ggml\\model.gguf" "--cont-batching" "--port" "30888" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--embedding" "--ubatch-size" "4096", kill_on_drop: true }

I left it running for about 2 hours and it just kept doing that.

More help would be appreciated.

-Deozaan (October 18, 2024, 03:40 AM)

First, it may be handy to list the specifications of the computer I use with Tabby (for reference):
CPU: Intel i3 10100F (bog standard cooling, no tweaks of any kind)
GPU: MSI GeForce 1650 (4 GB of VRAM, 128-bit bus, 75 Watt version (without extra power connectors on the card))
RAM: Kingston 32 GByte (DDR4, 3200 MHz) As of this week, I added another 16 GB RAM stick, just to see if dual channel was an improvement or not. Till now I didn't notice it. 
SSD: SSD via SATA interface (Crucial 500 GByte)
All inside a cheap, no-name "gamer" case

There are 3 things to try:

• Tabby without GPU support
• Tabby with GPU support
• No Tabby

Tabby without GPU support:
You will need to download a (much smaller) Tabby, extract it and make sure that you have as much of the fastest RAM as the motherboard supports in your computer to make this the best experience possible.
Start it with:

[Select]

.\tabby.exe serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct

Less than ideal, but you would still be able to (patiently) see how it works.

Tabby with GPU support:
This overview from NVidia shows which type of CUDA is supported on their GPUs.
My GPU is a 1650, which supports CUDA 7.5
Your GPU is a 670, which supports CUDA 3.0

This overview shows you need NVidia driver 450.xxx for your card if you use CUDA development software 11.x.
You can get v11.7 of the CUDA development tools here. As far as I know, you can go the NVidia website and download a tool that identifies your card and the maximum driver number it supports. If that number isn't 450 or higher, than I'm pretty sure that the Tabby version for CUDA devices won't work.

In case you can download a sufficient driver for your GPU, download the 'tabby_x86_64-windows-msvc-cuda117.zip' archive, extract it and start it with:

[Select]

.\tabby.exe serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device cuda

In case you cannot download a sufficient driver for your GPU, you could still try a Tabby version that supports vulkan devices. Vulkan is supported by NVIdia/AMD/Intel GPUs and is often used to make Windows games work on Linux. As your GPU is from around 2012 and Vulkan was made relatively recent, I don't know how far back Vulkan support goes. You might be lucky though. Anyway, download the 'tabby_x86_64-windows-msvc-vulkan.zip' archive, extract it and start it with:

[Select]

.\tabby.exe serve --model StarCoder-1B --chat-model Qwen2-1.5B-Instruct --device vulkan

No Tabby:
If this also doesn't work, than you have to come to the conclusion that your GPU is simply too old for use with LLMs/AI and that you are relegated to software that provide CPU-only access to LLMs/AI. And in that case, I recommend another free tool: LM Studio. This tool, in combination with the LLM 'bartowski/StableLM Instruct 3B' is my advice. This software is very versatile, takes about 1 GB of RAM and the LLM takes about 4 GB of RAM, so you'll need to have 8 GByte or more in your computer for LM Studio to work halfway decent.

[Shades]

By the way, in this environment there is is still a i7 CPU 1st generation active as a database server. It has 32 GByte of RAM and SSDs. Yes, it is even more ancient that your second generation i7 system, but it doesn't feel slow in anything it needs to do. So my boss doesn't want to change it, mainly because of (grandfathered) Oracle licensing-costs and Oracle being slowly phased out here.

So, ancient computer or not, as long as you don't game on those computers, they are still quite useful. And energy costs are not that big of a concern here in this part of South America, where the whole country runs on hydro-power.

[Deozaan]

I forgot to mention I have 16 GB RAM in this machine.

I tried it on CPU, CUDA 11 and 12 and Vulkan and I get similar results for all of them in a form looking very much like this:

Code: Text [Select]

1. 11012.227 s   Starting...←[2m2024-10-19T03:54:38.253368Z←[0m ←[33m WARN←[0m ←[2mllama_cpp_server::supervisor←[0m←[2m:←[0m ←[2mcrates\llama-cpp-server\src\supervisor.rs←[0m←[2m:←[0m←[2m98:←[0m llama-server <embedding> exited with status code -1073741795, args: `Command { std: "D:\\Apps\\tabby_x86_64-windows-msvc\\llama-server.exe" "-m" "C:\\Users\\Deozaan\\.tabby\\models\\TabbyML\\Nomic-Embed-Text\\ggml\\model.gguf" "--cont-batching" "--port" "30888" "-np" "1" "--log-disable" "--ctx-size" "4096" "-ngl" "9999" "--embedding" "--ubatch-size" "4096", kill_on_drop: true }`
2. ←[2m2024-10-19T03:54:38.253838Z←[0m ←[33m WARN←[0m ←[2mllama_cpp_server::supervisor←[0m←[2m:←[0m ←[2mcrates\llama-cpp-server\src\supervisor.rs←[0m←[2m:←[0m←[2m110:←[0m <embedding>: warning: not compiled with GPU offload support, --gpu-layers option will be ignored
3. ←[2m2024-10-19T03:54:38.255312Z←[0m ←[33m WARN←[0m ←[2mllama_cpp_server::supervisor←[0m←[2m:←[0m ←[2mcrates\llama-cpp-server\src\supervisor.rs←[0m←[2m:←[0m←[2m110:←[0m <embedding>: warning: see main README.md for information on enabling GPU BLAS support

I guess I'll try LM Studio next.

[Deozaan]

Hmmm... LM Studio won't work for me either. After poking around in the various tabs of the GUI it seems my CPU is the problem. 😭

[Shades]

Hmmm... LM Studio won't work for me either. After poking around in the various tabs of the GUI it seems my CPU is the problem. 😭

[ Invalid Attachment ]

-Deozaan (October 18, 2024, 11:32 PM)

Too bad about your CPU.

In all honesty, I did not expect that, as I have used both Tabby and LM studio on a computer with an AMD APU (9700), same GPU and 16 GB RAM. That APU was designed in the 'bulldozer'-era of AMD, but made to fit on the AM4 socket. That era may not be 2nd gen i7, but isn't that far off either. Tabby and LM Studio worked both on that CPU, but it struggled when more than 1 person was using the LLM functionality. Hence I moved it to the next best (or should I say worst) thing, which was the computer with the i3 10100F CPU. Since then, Tabby is "smooth" sailing" with 5 people accessing it.

Here in Paraguay it isn't that hard to find motherboards and CPUs from the 10 gen for cheap. It is also not really a problem to get 13th and 14th gen gear, but that is more expensive than what it costs in the U.S. or Europe. If I remember my parts prices correctly, that i3 computer would have costed about 425 to 450 USD, with the GPU as the most expensive part.

Perhaps it is an option to get/trade your computer (without GPU) for another older computer with a 4th/5th/6th gen CPU for cheap? Refurbished gear or something from a company that dumped their written-off gear at a computer thrift store? For getting your feet wet with LLMs/AI  that could be useful, while also not breaking the bank.

[Deozaan]

I don't really have the budget for upgrading my PC. Maybe I should get a $15 microcontroller and play around with TinyML.

But that's not really an LLM, so I guess I'm better off just accessing some free tier of ChatGPT.

[Shades]

Don't stare yourself too blind on ChatGPT, because it isn't the "all-in-one" solution for everyone or every use-case.

So here are some (mainstream) alternatives:
- ChatGPT (pretty much on par with Gemini for my use-cases).
- PHind (did well with Powershell requests and Oracle requests I made).
- v0 (relatively new player, that looks promising. Bit of a nag about getting a subscription with them, but that is it).
- Gemini (pretty much on par with ChatGPT for my use-cases).
- CoPilot (no experience with, besides unconsciously activating it by misclicking in the Edge browser).

- Replicate (site that lets you try out many different models, some free, some at costs).

[Deozaan]

I forgot to thank you once again for being so helpful.

So thanks! I had no idea about most of those. I'll check them out.

[KynloStephen66515]

For API Usage:
Mistral have an extremely generous free tier (1 billion tokens per month, 500,000/minute)
Gemini is free with a huge context window (2 million on pro, 1 million on flash models)
Groq (not Grok) is also free (but has heavy usage limits that might not suit everybody, but has EXTREMELY fast output speed)

Also try OpenWebUI, as it's capable of "plugging in" any model from the above list.  Another really good web-ui with a lot of extra features, that can also run locally, is LobeChat (They also have a hosted version)


I don't wanna shamelessly promote the company I work for (We have a very generous free tier for our VSCode/JetBrains/WebUI, AI coding assistant) so if you wanna know about that, drop me a DM here or discord and I'll let ya know (and can hook you up with a month of free Pro tier, too).

[Deozaan]

Thanks Kynlo. While I was adding these to my bookmarks, I decided to try out LobeChat's hosted version. It went something like this (highly paraphrased):


Q: Does the onboard LED of the Raspberry Pi Pico W support PWM to adjust its brightness?

A: Yes, it does.

Q: Can you show me some example code that will fade the LED?

A: This code will do that for you.

[ 15 minutes trying to get the code to work ]

Q: The code isn't working. What might be wrong?

A: Try these troubleshooting steps.

[ 15 minutes trying all the troubleshooting steps ]

Q: Are you absolutely sure the onboard LED supports PWM?

A: My apologies. The onboard LED does not support PWM.

Q: Can you show me example code to simulate PWM to adjust the LED's brightness in software rather than hardware?

A: This code will do that for you.

[ It works! ]

Pretty helpful when it doesn't lead you on a wild goose chase with wrong information.


I'm surprised there hasn't been a movie made yet about a group of people with a robot/AI companion who all end up stuck in a cavern and they ask the robot/AI
to help them find a way out and it unintentionally states incorrect information as fact and based on that information (and future queries to the robot/AI that don't challenge the initial assumption) the humans travel through a treacherous path where many people are injured or die until finally someone asks "Are you sure this is the best course of action?" and then the robot/AI says "My apologies. We can just take the ladder near where we started."

[Shades]

You could watch a TV-series, called 'Beacon 23'. Which plays in distant future, where humankind uses manned beacons for interstellar travel. And you get a hint of what life is like in such a beacon., which belong to corporations that have AI on those beacons themselves, and tend to provide a personal AI to the person manning the beacon.

Might sound boring for a sci-fi series, but it really isn't. Quite a bit more serious than your movie idea. Which I think would go over well 

[KynloStephen66515]

Thanks Kynlo. While I was adding these to my bookmarks, I decided to try out LobeChat's hosted version. It went something like this (highly paraphrased):


Q: Does the onboard LED of the Raspberry Pi Pico W support PWM to adjust its brightness?

A: Yes, it does.

Q: Can you show me some example code that will fade the LED?

A: This code will do that for you.

[ 15 minutes trying to get the code to work ]

Q: The code isn't working. What might be wrong?

A: Try these troubleshooting steps.

[ 15 minutes trying all the troubleshooting steps ]

Q: Are you absolutely sure the onboard LED supports PWM?

A: My apologies. The onboard LED does not support PWM.

Q: Can you show me example code to simulate PWM to adjust the LED's brightness in software rather than hardware?

A: This code will do that for you.

[ It works! ]

Pretty helpful when it doesn't lead you on a wild goose chase with wrong information.


I'm surprised there hasn't been a movie made yet about a group of people with a robot/AI companion who all end up stuck in a cavern and they ask the robot/AI
to help them find a way out and it unintentionally states incorrect information as fact and based on that information (and future queries to the robot/AI that don't challenge the initial assumption) the humans travel through a treacherous path where many people are injured or die until finally someone asks "Are you sure this is the best course of action?" and then the robot/AI says "My apologies. We can just take the ladder near where we started."

-Deozaan (October 26, 2024, 05:55 PM)

What model did you use for this?

I'd highly suggest only using Claude-3.5-Sonnet-Latest, for coding-related work, as it's the only one that's actually any use.  GPT4o isn't terrible, but given that it has a knowledge cutoff in 2023, it has absolutely no idea about anything more recent.


This is the output I got using Claude (albeit, with a tuned system prompt for coding):

Yes, the Raspberry Pi Pico W's onboard LED (GPIO 25/"LED" pin) does support PWM! I'll show you both hardware PWM and software PWM approaches.

1. Hardware PWM Solution: Here's a complete example that fades the LED up and down using hardware PWM:

[Select]

from machine import PWM, Pin
import time

# Setup the onboard LED with PWM
led_pwm = PWM(Pin("LED"))  # GPIO 25
led_pwm.freq(1000)  # Set PWM frequency to 1000Hz

def pulse_led():
    while True:
        # Fade in
        for duty in range(0, 65535, 1000):
            led_pwm.duty_u16(duty)
            time.sleep(0.01)
           
        # Fade out   
        for duty in range(65535, 0, -1000):
            led_pwm.duty_u16(duty)
            time.sleep(0.01)

try:
    pulse_led()
except:
    # Clean up PWM on exit
    led_pwm.deinit()

2. Software PWM Solution (if hardware PWM isn't working): This simulates PWM using software timing:

[Select]

from machine import Pin
import time

led = Pin("LED", Pin.OUT)

def set_brightness(brightness):
    # brightness should be between 0 and 100
    cycle_time = 0.001  # 1ms cycle
    on_time = (brightness / 100.0) * cycle_time
    off_time = cycle_time - on_time
   
    led.value(1)
    time.sleep(on_time)
    led.value(0)
    time.sleep(off_time)

def pulse_led():
    while True:
        # Fade in
        for brightness in range(0, 101, 5):
            for _ in range(10):  # Multiple cycles per brightness level
                set_brightness(brightness)
       
        # Fade out
        for brightness in range(100, -1, -5):
            for _ in range(10):  # Multiple cycles per brightness level
                set_brightness(brightness)

try:
    pulse_led()
except:
    led.value(0)  # Turn off LED on exit

Key differences between the approaches:

Hardware PWM:

• More efficient
• Smoother fading
• No CPU overhead
• Uses dedicated hardware

Software PWM:

• More CPU intensive
• May show slight flickering
• Works on any GPIO pin
• Useful when hardware PWM isn't available

To run either example:

[Select]

mpremote cp main.py :main.py

[Select]

mpremote reset

The LED should start fading in and out in a continuous loop. The hardware PWM version will give you the smoothest results, but both approaches work to demonstrate LED brightness control.

[Deozaan]

What model did you use for this?

-KynloStephen66515 (October 27, 2024, 08:20 AM)

When I created my account it asked me what I was interested in and I selected the option that appeared to be more about coding and development. It then suggested (and I accepted) a bunch of "assistants" which I assumed it would query all of or pick the best one for me based on my question. So I just started typing away (in the "Just Chat" section).

Looking more into it, it seems to have created a variety of "assistants" for me on various topics but they are all set to use gpt-4o-mini, except for one which is set to use gpt-3.5-turbo-16k.

I'll try adjusting or creating one using Claude and see if that gives better results, but judging on your answer, it still gives you incorrect information. As far as I can tell the onboard LED of the Pico W does not actually support (hardware) PWM, because it's controlled by the WiFi module instead of the RP2040. But there's a big enough knowledge gap for me that I may be wrong about that. Maybe the fact that I'm using Arduino IDE (which I think is just C or C++) rather than MicroPython makes a difference in what hardware features the libraries I'm including use.

[KynloStephen66515]

Raspberry Pi Pico W's onboard LED

-Deozaan (October 27, 2024, 03:32 PM)

Not to stray too far off-topic, but you seem to be correct, as per: https://forums.raspb...ewtopic.php?t=358964 and https://forums.raspb...ewtopic.php?t=336836