Show Posts
1
Mini-Reviews by Members / Re: Horse Browser Review
« on: May 31, 2025, 07:22 PM »
You can run a (smaller) local AI/LLM easily enough. Depending on your GPU hardware. Or lack thereof. From this and other posts, I gathered that you stored all of your previous research already. That data could be put into a vector database (RAG) and this vector database can then be coupled to this locally running AI/LLM. Once that is done, you will see that even these smaller local LLM's are pretty good for helping you out finding what you need, collect this data and "feed" that into an external genealogy database.
You could even find out which research paths were a dead end, or maybe less of a dead end than envisioned, with a few simple prompts. Or tell the AI/LLM that those paths were already marked as a dead end, so not to be investigated (in an much more automated) way.
Smaller models do tend to hallucinate more than online ones, but if the data in your RAG solution is solid, you'll find there will be no to hardly any hallucination. The "garbage in, garbage out"-concept is very much a thing with AI/LLM. The very large online versions are usually filled with better/more coherent data, making those look good in comparison with smaller models.
But you will be very pleasantly surprised how well those small models perform, when you can let it go loose with your own proper data. And those will not rob you blind with subscription fees, token consumption limitations and possible overcharge fees.
Just get a free tool like 'LM Studio" (GUI tool for Windows, Linux and Mac) and/or "Msty" (GUI tool for Windows, Linux and Mac) or even "Ollama" (PowerShell/terminal-based text tool for Windows, Linux and Mac). All of these also have a server-like function. Meaning you can connect LLM web-interfaces (such as 'Open-WebUI') to these tools. Then you can use your local AI/LLM with any device in your LAN (computers, laptops, tablets, phones, even a smart TV if it has a decent enough browser).
Personally, I went the "LM Studio"-way, because it also has an excellent LLM model search function build-in. Where I discovered model 'ui-tars-1.5-7b', which is surprisingly sound of logic (without giving it a system prompt to tweak it) given it's size. And even manages to output between 4 to 5 tokens per second on a desktop with a 10th generation Intel i3 CPU (5 years old by now), no GPU of any kind, a small and simple 2,5" SATA SSD drive and 16 GByte of 3200 MHz RAM.
Fit such an old PC with a GPU that contains 6 GByte of VRAM and this model can be loaded into VRAM instead. The 4 to 5 tokens/sec output is too slow for person who reads. When the same model is loaded in VRAM, the output goes to around 12 to 15 tokens/sec. And that is fast enough for adept readers. Maybe not for speed readers, but given the state of today, there aren't that many persons anymore that have and/or use that skill.
Sorry for ranting on and on about this. Thought I mention all of the above, because you already did a lot of legwork and have the data. And in this case, I expect (local)AI/LLM to be a big boon for you. Just need to figure out the RAG solution for your collected data. Tools like 'Rlama' and 'LlamaIndex' are likely to be a great help in finding the right solution and/or help you build your RAG solution, as both can deal with PDFs, images, images in PDF, word and excel documents, text and MarkDown, etc.
You could even find out which research paths were a dead end, or maybe less of a dead end than envisioned, with a few simple prompts. Or tell the AI/LLM that those paths were already marked as a dead end, so not to be investigated (in an much more automated) way.
Smaller models do tend to hallucinate more than online ones, but if the data in your RAG solution is solid, you'll find there will be no to hardly any hallucination. The "garbage in, garbage out"-concept is very much a thing with AI/LLM. The very large online versions are usually filled with better/more coherent data, making those look good in comparison with smaller models.
But you will be very pleasantly surprised how well those small models perform, when you can let it go loose with your own proper data. And those will not rob you blind with subscription fees, token consumption limitations and possible overcharge fees.
Just get a free tool like 'LM Studio" (GUI tool for Windows, Linux and Mac) and/or "Msty" (GUI tool for Windows, Linux and Mac) or even "Ollama" (PowerShell/terminal-based text tool for Windows, Linux and Mac). All of these also have a server-like function. Meaning you can connect LLM web-interfaces (such as 'Open-WebUI') to these tools. Then you can use your local AI/LLM with any device in your LAN (computers, laptops, tablets, phones, even a smart TV if it has a decent enough browser).
Personally, I went the "LM Studio"-way, because it also has an excellent LLM model search function build-in. Where I discovered model 'ui-tars-1.5-7b', which is surprisingly sound of logic (without giving it a system prompt to tweak it) given it's size. And even manages to output between 4 to 5 tokens per second on a desktop with a 10th generation Intel i3 CPU (5 years old by now), no GPU of any kind, a small and simple 2,5" SATA SSD drive and 16 GByte of 3200 MHz RAM.
Fit such an old PC with a GPU that contains 6 GByte of VRAM and this model can be loaded into VRAM instead. The 4 to 5 tokens/sec output is too slow for person who reads. When the same model is loaded in VRAM, the output goes to around 12 to 15 tokens/sec. And that is fast enough for adept readers. Maybe not for speed readers, but given the state of today, there aren't that many persons anymore that have and/or use that skill.
Sorry for ranting on and on about this. Thought I mention all of the above, because you already did a lot of legwork and have the data. And in this case, I expect (local)AI/LLM to be a big boon for you. Just need to figure out the RAG solution for your collected data. Tools like 'Rlama' and 'LlamaIndex' are likely to be a great help in finding the right solution and/or help you build your RAG solution, as both can deal with PDFs, images, images in PDF, word and excel documents, text and MarkDown, etc.
