Looking for lightweight, simple, portable/integrated database for Go

[Deozaan]

Hi all,

I've been thinking about writing a little utility for myself that could probably benefit from using a database of some kind. My needs will honestly be simple enough that I can probably just write everything out to a text (XML or JSON) or binary file, but on the chance that this is something that gets released and used by others who have a lot more data to wade through than I will, I figured I'd look into options of some sort of simple database system that might help keep things relatively speedy compared to parsing and iterating through plaintext to get all the relevant details.

The hard part about answering my request is that I'm not sure what language I'll be using for this utility. I'm most familiar with C# (but by no means an expert), so my inclination is to use that, but lately I've been tinkering with Python and Go and am considering using one of those if it would suit this project better. Also, cross-platform compatibility is somewhat important to me, at least between Windows and Linux. Ideally I'd like for my utility to have a GUI, but I'm pretty inexperienced with GUI frameworks and they tend to not always be very cross-platform friendly, and I'm not looking to make this a huge project that will take over my life for the next year. So for now I'd be satisfied with a command-line utility that can read, write, search, and display the pertinent information on demand.

Oh, and I'd like to be able to include the database with the utility in a portable manner, so the end-user can just run everything on a thumb drive or whatever without having to worry about installing things.

Maybe that was more information than needed. Anyway, can anyone recommend a simple and easy to integrate/use, portable database-type system that would be more efficient for parsing and looking things up than a plaintext (JSON/XML) alternative?

Thanks in advance

[wraith808]

SQLite has been the go-to for that kind of database, though I utilized VistaDB (though got that for free from a review- it's actually a bit pricey).

https://system.data....trunk/www/index.wiki

https://www.nuget.or...s/System.Data.SQLite

[wraith808]

Also, I use this to manage my DBs.

http://sqlitebrowser.org/

There are others... though I haven't really tried any of them as that was enough.

http://www.ginktage....te-management-tools/

[Ath]

+1 for SQLite, it's used by default on Android and highly endorsed when using Xamarin, has support for nearly all platforms in use and bindings/libraries for many programming languages.

[KodeZwerg]

A NoSql Variant could be MongoDB. It can be used by many programming Languages.

[Ath]

A NoSql Variant could be MongoDB. It can be used by many programming Languages.

-KodeZwerg (August 31, 2018, 06:00 AM)

Hm, but it doesn't quite match the OP's requirements of being lightweight, simple and portable, AFAICS.

[Shades]

Besides all of the above, MongoDB is intended more for document management.  The strength of NoSQL databases lies also in their ability to have nodes (read: separate computers) for reading/storing data at speed. All of that does not indicate that it even can (or should) be used as "portable" solution.

SQL Lite is indeed a good option. MySQL has a lot of free tools to manage that database. Even web-based it that tickles your fancy. There is a tool called: XAMPP which you can use in a portable sense to create a full blown web development environment. there is a tool in it that allows you to start/stop each individual section of that environment separately (Apache, MySQL, PHP and Python). So you could only start the database from it and you could use the command line to create databases and users. Or use the included PHPMyAdmin web interface (which requires you to enable Apache only for as long as you need it to). And if manually starting MySQL sounds like a bit of a drag, the start/stop tool also allows you to enable it as a self starting service too.

XAMPP is an archive of about 75MByte (7-zip) to download and if you extract it in a root folder (for example:  C:\xampp) than you don't need to do anything to make MySQL work. If you don't want to use the root folder, there is an executable batch script in that archive that shows you the steps to reconfigure it for any desirable storage location.

If your tool becomes popular you might want the extra features of a more mature database, which can be easily implemented on (cloud-based) web servers as well. In that case MySQL is a decent bet. Quite a feature set, smallish resource footprint, a lot of free (web-based) management tools and lots of code examples on how to work with it programmatically in whatever language.

[Deozaan]

I decided to give SQLite a try with Go, but all my Googling seems to suggest I should use https://github.com/mattn/go-sqlite3 to interface with SQLite, but I can't get that project to build. It requires GCC, which I can't find anywhere without it being bundled with a billion other development tools. And after installing a few different development packages (mingw, msys, windows-build)

After working on it for a couple of hours I'm ready to consider alternative ideas. I was hoping I could find something that would just be "drop in" for myself as a developer, and also "just work" invisibly behind the scenes for the end-user.

I've just discovered Bolt DB while pausing from writing this post to look further into the issues I was having with go-sqlite3, and it seems to be fairly drop in and simple. I'll give that a try and see how it works.

[KodeZwerg]

A NoSql Variant could be MongoDB. It can be used by many programming Languages.

-KodeZwerg (August 31, 2018, 06:00 AM)

Hm, but it doesn't quite match the OP's requirements of being lightweight, simple and portable, AFAICS.

-Ath (August 31, 2018, 06:10 AM)

In one of my applications i've included a few lines php, more not needed to interact with mDB, MongoDB works over Cloud, how much more portabilty exists? Anyway, OP goes SQLite, so thread is done i guess 

Best wishes! (i did not wanted to tell that sql is bad, no misunderstanding please, i just wanted to give another option a slight chance of being known.)

[Ath]

Thank you @KodeZwerg for your nosql alternative, but the biggest objection to me is one of the main features of MongoDB: It's in the cloud. That means that my data is not really my data... and not really portable, as it requires an internet connection.

@Deozaan: As with any project, you have to choose the right tools for the job. If the combo Go/SQLite isn't working as expected/desired, it most likely isn't meant to be. You might need other tools...

[Deozaan]

@Deozaan: As with any project, you have to choose the right tools for the job. If the combo Go/SQLite isn't working as expected/desired, it most likely isn't meant to be. You might need other tools...

-Ath (August 31, 2018, 02:48 PM)

As I said, my intent as of this moment is to try out Go with Bolt DB. But I haven't had a chance to get to the DB stuff yet because I'm still trying to figure out how to parse JSON from an HTTP GET request into an easy to use data structure.

I need a Go cheat sheet or something because I'm such a noob.

[wraith808]

A NoSql Variant could be MongoDB. It can be used by many programming Languages.

-KodeZwerg (August 31, 2018, 06:00 AM)

Hm, but it doesn't quite match the OP's requirements of being lightweight, simple and portable, AFAICS.

-Ath (August 31, 2018, 06:10 AM)

In one of my applications i've included a few lines php, more not needed to interact with mDB, MongoDB works over Cloud, how much more portabilty exists? Anyway, OP goes SQLite, so thread is done i guess 

Best wishes! (i did not wanted to tell that sql is bad, no misunderstanding please, i just wanted to give another option a slight chance of being known.)

-KodeZwerg (August 31, 2018, 02:01 PM)

Portable usually means that the software can be used anywhere, without needing anything external to the reference.  If you're in a location with no internet, or with internet but the address you're accessing your db behind blocked, you're sort of out of luck with a cloud-based solution.

[Deozaan]

I've made some progress. Since it seems I'll be sticking with Go for this project, I've modified the topic subject to reflect that.

Also of note for anyone else who may be interested:

I've found GJSON to be just what I needed to easily parse JSON files.

And here are a couple of good tutorials explaining how to get started with BoltDB:

Intro to BoltDB: Painless Performant Persistence
Bolt -- an embedded key/value database for Go