Extremely delayed THANK YOU + a little request

[puzzle]

Hello and greetings from Germany!

MPQ is actually my one and only screensaver since 2013 - I absolutely adore the almost endless combinations of text and image it conjures. And yes, I use the video frame capture feature as well, it really adds to the diversity and the fun.

Thank you so much for this gem!


...but... (and here's the request part)

Over the years, I put together a bunch of pretty large text files to quote from. As MPQ needs to scan in all lines from all text files on startup, it now already takes way more than 1 minute for the first screen activity to show up for me, and that's quite annoying. Especially because I know this issue will become worse in the future, as I'm not willing to stop enhancing my quote collection 

It doesn't look like MPQ is still being worked on, so in order to keep the MPQ code untouched, I came up with the following thoughts (I hope they make sense):

I could prepare each text file with the exact number of following quote entries in the very first line, like:

line 1: 12345
line 2: quote line 1
line 3: quote line 2
.
.
line 12346: quote line 12345

A small application (say, "grabquote.exe") compatible with MPQ's $$$appcap command then reads that first line, randomly chooses a quote line from that file (based on the given number of entries) and returns it to %lastappcap%. This way, I could use a very small quote book with $$$appcap commands only, each pointing to a different text file containing large numbers of lines to quote from, and without MPQ needing to cache each single line from each file itself.

Something like: $$$appcap %appdir%\grabquote.exe %quotedir%\file.txt;;;%lastappcap%;;;$$$setimagedir %imagedir%

I believe this would cut down the startup time of MPQ to an absolute minimum, regardless of the total amount of quote entries I make available throughout the various text files.


My problem here: I have no idea how to write an MPQ compatible app like this myself.

Is there a kind community member out there who could help me out on this please (or point me to an already existing and working solution that does the trick)?

That'd be great! 



PS: My long overdue donation for MPQ will be out this November, and of course I'd love to PayPal something extra if custom coding work has to be done to solve my "problem" 

[mouser]

MPQ is one my apps that just never got much attention (though I use it myself).
I'm glad to hear that someone is enjoying it.

The MPQ codebase is alive and well and I'm happy to add features and release a new version.

---

There are a few ways I could improve the speed of parsing quote files.. The easiest might be to support an alternate quote file format that it could read and cache.. wouldn't be too hard.

But before I do it would be nice to figure out where the slowdown is really happening and why.. MPQ is C++ code that should be able to parse a pretty huge file pretty quickly, so a minute is a VERY long time for even huge quote file.. It may be there is some inefficient code that I've written that I could clean up..

Can you email me your quote files so I can reproduce the same scenario? Or similarly large files..? ([email protected])

[puzzle]

Wow, mouser, thanks for your fast reply!

And thank you as well for willing to rework parts of MPQ to solve my issue, but I don't think you have written "some inefficient code"...

...maybe I should have been more specific about the amount of data I want to use with MPQ - my current text folder is about 250 MB (!) in total. I guess that's where the extremely long startup time comes from, and that's why I came up with the idea of using an external app. Seems to be the easiest way rather than you trying to tweak MPQ itself solely for the strange needs of a not-average user.

"WTF! 250 MB of text?!" - "You should see my image and video folder!"

Well, for a short explanation (in case you're curious): I'm a crazy person and I love chaos, randomness and diversity (hence I love MPQ). I have many files with text collections of all sorts and a great number of own writings. And s**tloads of movie subtitles on top of that. No end in sight.

When the startup delay started to get annoying, my first thought was there may be too many files, so I bundled them to a couple of large files (~ 10 to 50 MB) for testing, but there's no difference (obviously). Of course I can assemble and send you a sample file of similar size, but I guess that's obsolete now.

I also guess MPQ caches my huge pile of text to physical memory while running, so an external solution seems desirable for that reason as well, but I'm not quite sure if my above ideas are even feasible.

[mouser]

Draw god 250mb of text is a lot of text...  Though I respect the craziness

One idea would be that if MPQ is putting up a random quote every so often, you're only going to see a small subset of these potential quotes in your lifetime.. Perhaps one solution would be a utility that processed all your quote files once and chose a random set of 10,000 and saved those to a new file that MPQ will read from..

The problem with your proposed solution is that i don't think it would solve the speed problem.  It sort of depends.. There is no fast way to jump to line 50,000 in a text file -- the program still has to read every line.  It *would* save you memory, but not time.  In fact you'd start to be much slower if the program had to retrieve multiple quotes between slowing down..

Now you COULD drastically speed up the process if we built a binary "index file" for each of the quote files that stored an index of where each quote was located in the file, so it could read it directly with random access instead of loading the entire file.  This wouldn't be TOO hard to do, though it would probably take a day of coding.. One could use a standalone utility to do it, or it could be built into MPQ to build such binary index files whenever it encounters a new (or changed) quote file.

It's a fun idea, just a little hard to justify for a single person's needs, since there is probably no one else needing it..

Part of why this slowdown is problematic for MPQ is because MPQ is not a program that stays in memory.. So another potential solution would be to add a mode to MPQ that basically didn't exit but rather went into the background and "woke up" when triggered as a screensaver, etc.  That way you would see the slowdown on the first startup of the program, but then not again until reboot..

[puzzle]

Happy Halloween!

You're already putting so much time and effort into this niche matter, awesome.

First of all, the binary index file idea does indeed sound very tempting, but no - as you said, I'm probably the only person who will ever use that feature, so this would be overkill.
But this made me read some stuff about how text files are structured to better understand the potential problems with the method I had in mind.

Just to be sure I got it right: my approach would mean that the external app has to read in all lines of the respective target file until it reaches the randomly chosen line number. So I save memory and get no delay on MPQ startup, but instead I get a delay each time MPQ changes the quote. And the length of each single delay depends on how high the respective random line number is, i.e. rnd line # 50 = no delay, rnd line # 50,000 = long delay.

Well, in that case I probably have figured out a way this could work for me (still pondering, but I'm almost sure).
But first I'd like to ask, how difficult would it be for you to code an app that works exactly like I described above?

After taking all the potential solutions (and their disadvantages) into consideration, this seems to be the most satisfying compromise for me.

By the way, I started my digital text/image collections in the early nineties, so discovering MPQ was like a godsend for me. It's not like I amassed all those text files just because of MPQ - I am crazy, but not *this* crazy

[puzzle]

I probably have figured out a way this could work for me

I'm definitely sure about this now. It will need a little bit of extra work on my larger files, but I found some freeware tools (text splitter/generator) to minimize the necessary efforts.

how difficult/time-consuming would it be for you to code an external app that works exactly like I described above?

After thinking about it, it seems to be quite easy to do...

Now the main question is: Would you do that for me, please? I'd be really happy