DonationCoder.com Software > Post New Requests Here

DONE: Delete every N files, or keep only every N files

<< < (2/4) > >>

Deozaan:
Sorry I wasn't clear. I took your advice about being paranoid about file deletions, made a backup of the 80GB directory of screenshots, then I used Skwire's script to delete things. I figured if I wasn't happy with the results while it ran over night I could just restore the backup and try something else.

Starting at almost 80,000 files (~78k to be more precise) I let it run for about half an hour before I went to bed, at which point it had reduced the number of files to just below 60,000 files. So, about 20k files per half hour meant it should have finished in about 2 hours.

When I came back to my computer this morning just before 10AM, my PC was completely locked up and unresponsive. The clock still said 8:23AM, the mouse cursor wouldn't move, and ctrl-alt-delete or ctrl-shift-escape would not bring up the task manager.

After resetting my computer, I see that the AHK script didn't finish the job. There are still about 23,000 files left and there should be roughly only 13,000 when it finishes. So it still had about 10k to go before things locked up.

I'm guessing that the reason my PC locked up was due to the script, as I can't recall ever having such a total lockup on this system, and the script was putting my Core i7 2600K to near 100% usage on all cores/threads.

MilesAhead:
I used Skwires script to delete things.
-Deozaan (April 22, 2015, 11:15 AM)
--- End quote ---

When I last posted Skwire's script wasn't visible to me.  Sorry for the confusion.  :)

Deozaan:
Yeah, that script definitely caused issues. I moved all the images that were already kept into another directory so it could just resume from where it left off and started the script up again. Here are some screenshots of behavior I saw.

As expected, it immediately maxed out my CPU usage to 100% and made my computer somewhat sluggish (occasional "hiccups" of unresponsiveness). I opened the task manager to take a look at the CPU usage and noticed the RAM usage climbing sky high:

DONE: Delete every N files, or keep only every N files

DONE: Delete every N files, or keep only every N files

RAM usage kept climbing, so I closed out Chrome, Firefox, and other misc. programs, anything using more than 100MB of RAM got shut down.

DONE: Delete every N files, or keep only every N files

The RAM kept climbing up, but soon peaked around 13-14GB of RAM(!), and then started alternating between going down and climbing back up, like so:

DONE: Delete every N files, or keep only every N files

And to get an idea of how fast/slow the files were being deleted, here's an animated gif:



I lowered the process priority so it wouldn't slow down my PC and left it running for a few hours while I was away from my PC again. It eventually finished the job, though I'm not sure how long it took.

Final results of the cull: ~80,000 files totaling ~80GB reduced to ~13,000 files totaling ~13GB. I guess that means the average filesize is about 1MB per image.

Ath:
Be aware that having a (1) directory with ~80,000 files in it will bring Explorer, and other tools looking at that file-list, to a crawl :'(

MilesAhead:
Be aware that having a (1) directory with ~80,000 files in it will bring Explorer, and other tools looking at that file-list, to a crawl :'(
-Ath (April 23, 2015, 01:59 AM)
--- End quote ---

It may have been easier to do a move of the Nths files to an outside folder on the same drive.  Then just delete the original folder and recreate it.  Not sure if AHK would be faster that way(depends how FileRemoveDir is implemented) but it would be fewer file operations until right at the end.  A Move on the same drive should be quite fast.

Edit:  It is difficult when dealing with a large number of objects with untested code.  For one thing, it's not trivial to set up a test case generating scads of fake data.  Nobody wants to do that unless they have a guinea pig PC lying around.  Plus it is very time consuming.  Perhaps a mechanism to lower the AHK task priority(I have never used SetBatchLines.  I'm not sure if it's designed for this situation) would help to make it more of a background task.

Navigation

[0] Message Index

[#] Next page

[*] Previous page