Well, I'm still working on some requests for
Duplicate Photo Finder, but I'm encountering some odd memory problems.
I'm getting "Out of memory" exceptions, and in particular, "A generic error occurred in GDI+".
If the above sounds interesting, read on, otherwise don't because it gets ugly.
I'm running through photos and processing them in threads in a threadpool. I basically queue up several hundred, then let it go.
What happens is that it very happily rips through the photos, then all of a sudden, memory usage spikes real quickly (over say 1~5 seconds), and then I get the GDI+ error. What's happening there is that I run out of memory. The application sucks up 200~500 MB while running (I have 8 GB of RAM), but when it spikes, it dies at 1 GB to 1.4 GB of RAM (from Task Manager).
During those few seconds when the program begins to blow up, the number of threads that it's working on triples out of the blue. It goes from 6~8 threads to BOOM! 24, 26, 28, 30, 31 CRASH~! It hits the GDI+ error.
I've run "memprofiler" and cannot locate any memory leaks. It tells me:
Disposed instance (Ignore...)
5 types have instances that have been disposed but not GCed.
Investigate the types below for more information.
System.Drawing.Bitmap, System.Drawing.Graphics, System.Drawing.Region, DeviceContext, WindowsFont
Instance queued for finalization (Ignore...)
One type has instances that are queued for finalization. This can indicate that a Finalizer method is stuck, which will prevent instances from being finalized and cause memory leaks.
Investigate the type below for more information.
DeviceContext
Pinned instance (Ignore...)
3 types have instances that are pinned in memory.
Investigate the types below for more information.
System.Object, System.Object[], System.String
Instance indirectly rooted by finalizer queue (Ignore...)
2 types have instances that are indirectly rooted by the finalizer queue. This can indicate that a Finalizer method is stuck, which will prevent instances from being finalized and cause memory leaks.
Investigate the types below for more information.
System.Collections.Stack, System.Object[]
Large instance (Ignore...)
2 types have instances that are located in the large object heap.
Investigate the types below for more information.
System.Byte[], System.Object[]
Undisposed instances (unclassified) (Ignore...)
16 types have instances that have been garbage collected without being properly disposed.
Investigate the types below for more information.
SafeTokenHandle, SafeWaitHandle, System.Drawing.Bitmap, System.Drawing.Font, System.IO.BinaryReader, System.IO.MemoryStream, UnmanagedMemoryStream, ResourceReader, RuntimeResourceSet, ExecutionContext, (...)
Here's enough code to see what's going on. I've edited it somewhat to make it shorter (e.g. remove Console.WriteLine & debugging stuff.)
private void QueueHashWorkItem(DirectoryInfo di, bool isOriginal, int count, bool exactImageDataComparison)
{
int indexer = -1;
foreach (FileInfo fi in di.GetFiles())
{
if (fi.Extension.ToLower() == ".jpg")
{
indexer++;
// HashWorkItem is a struct to hold info
HashWorkItem hwi
= new HashWorkItem
(di
.FullName, fi
.FullName, isOriginal, count, exactImageDataComparison
); hwi.DirectoryName = di.FullName;
hwi.FileName = fi.FullName;
hwi.Index = indexer;
hwi.IsOriginal = isOriginal;
// get hashes in threads
ThreadPool
.QueueUserWorkItem(new WaitCallback
(GetHashForItem
), hwi
); // see below }
}
}
private void GetHashForItem(object state)
{
// do the work
HashWorkItem hwi = (HashWorkItem)state; // This is a struct to hold information.
ImageConverter converter
= new ImageConverter
(); long pressure
= new FileInfo
(hwi
.FileName).Length; GC.AddMemoryPressure(pressure);
Bitmap bmp
= new Bitmap
(1,
1); // I actually release the bmp below with bmp.Dispose(), so init here
#region TRY block
try
{
bmp = (Bitmap)Bitmap.FromFile(hwi.FileName); // zero problems here
byte[] bytes
= new byte[1];
if (!hwi.CompareExactImageData)
{
// ****************************************************************
bytes = (byte[])converter.ConvertTo(bmp, bytes.GetType()); // #0 THIS IS WHERE I GET THE GDI+ ERROR
// ****************************************************************
}
else if (hwi.CompareExactImageData)
{
// ****************************************************************
bytes = GetBytesFromBitmap(bmp); // #1 THIS IS WHERE I GET THE GDI+ ERROR -- See method below
// ****************************************************************
}
SuperFastHashUnsafe sfh
= new SuperFastHashUnsafe
(); hwi.ItemHash = sfh.Hash(bytes);
try
{
// invoke the delegate to store the results
this.Invoke(new StoreHashResultsDelegate
(StoreHashResults
), hwi
); }
catch
{
// NEVER happens
}
}
catch (OutOfMemoryException memEx)
{
Console.Write("***** Out of memory: " + hwi.FileName + " *** " + memEx.Message);
Thread.Sleep(2000);
// queue the item again...
ThreadPool
.QueueUserWorkItem(new WaitCallback
(GetHashForItem
), hwi
); }
finally
{
if (bmp != null)
{
bmp.Dispose();
bmp = null;
GC.Collect(); // this makes no difference it seems
}
}
#endregion
GC.RemoveMemoryPressure(pressure);
}
public delegate void StoreHashResultsDelegate(HashWorkItem hwi);
public void StoreHashResults(HashWorkItem hwi)
{
// blah - here for reference - zero problems here
}
/// <summary>
/// Get a byte array of pixel data from a Bitmap.
/// </summary>
/// <param name="bmp">The bitmap to get the pixel data from.</param>
/// <returns>A byte array of pixel data from a bitmap.</returns>
private static byte[] GetBytesFromBitmap(Bitmap bmp)
{
byte[] result = null;
if (bmp == null)
return result;
// Create a memory stream to hold the bitmap.
MemoryStream ms
= new MemoryStream
(); // ****************************************************************
bmp.Save(ms, ImageFormat.Bmp); // #2 THIS IS WHERE I GET THE GDI+ ERROR
// ****************************************************************
result = ms.ToArray();
return result;
}
Does anyone have any ideas how to troubleshoot that?
I've spent about 2 days on this and I'm not getting anywhere.
I've run around looking into everything I can, including LHO (Large Heap Objects), but so far no joy.
BTW - I'm running through directories of photos that are around 4~7 MB each (10 MP camera JPGs) with around 500 photos per directory. The program craps out randomly, sometimes ripping through 200, or 300 or 400 or 500 before dying. Directories are around 3 GB.
Any ideas, pointers or whatever would be appreciated.
Hey, and just for fun, I'll put a $10.00 donation bounty on it for the first person to point me to a solution~!
