Other Software > Developer's Corner
C# GDI+ Problem with byte[] and Bitmap - Memory Issues
Renegade:
Well, I'm still working on some requests for Duplicate Photo Finder, but I'm encountering some odd memory problems.
I'm getting "Out of memory" exceptions, and in particular, "A generic error occurred in GDI+".
If the above sounds interesting, read on, otherwise don't because it gets ugly.
I'm running through photos and processing them in threads in a threadpool. I basically queue up several hundred, then let it go.
What happens is that it very happily rips through the photos, then all of a sudden, memory usage spikes real quickly (over say 1~5 seconds), and then I get the GDI+ error. What's happening there is that I run out of memory. The application sucks up 200~500 MB while running (I have 8 GB of RAM), but when it spikes, it dies at 1 GB to 1.4 GB of RAM (from Task Manager).
During those few seconds when the program begins to blow up, the number of threads that it's working on triples out of the blue. It goes from 6~8 threads to BOOM! 24, 26, 28, 30, 31 CRASH~! It hits the GDI+ error.
I've run "memprofiler" and cannot locate any memory leaks. It tells me:
Disposed instance (Ignore...)
5 types have instances that have been disposed but not GCed.
Investigate the types below for more information.
System.Drawing.Bitmap, System.Drawing.Graphics, System.Drawing.Region, DeviceContext, WindowsFont
Instance queued for finalization (Ignore...)
One type has instances that are queued for finalization. This can indicate that a Finalizer method is stuck, which will prevent instances from being finalized and cause memory leaks.
Investigate the type below for more information.
DeviceContext
Pinned instance (Ignore...)
3 types have instances that are pinned in memory.
Investigate the types below for more information.
System.Object, System.Object[], System.String
Instance indirectly rooted by finalizer queue (Ignore...)
2 types have instances that are indirectly rooted by the finalizer queue. This can indicate that a Finalizer method is stuck, which will prevent instances from being finalized and cause memory leaks.
Investigate the types below for more information.
System.Collections.Stack, System.Object[]
Large instance (Ignore...)
2 types have instances that are located in the large object heap.
Investigate the types below for more information.
System.Byte[], System.Object[]
Undisposed instances (unclassified) (Ignore...)
16 types have instances that have been garbage collected without being properly disposed.
Investigate the types below for more information.
SafeTokenHandle, SafeWaitHandle, System.Drawing.Bitmap, System.Drawing.Font, System.IO.BinaryReader, System.IO.MemoryStream, UnmanagedMemoryStream, ResourceReader, RuntimeResourceSet, ExecutionContext, (...)
--- End quote ---
Here's enough code to see what's going on. I've edited it somewhat to make it shorter (e.g. remove Console.WriteLine & debugging stuff.)
--- Code: C# ---private void QueueHashWorkItem(DirectoryInfo di, bool isOriginal, int count, bool exactImageDataComparison){ int indexer = -1; foreach (FileInfo fi in di.GetFiles()) { if (fi.Extension.ToLower() == ".jpg") { indexer++; // HashWorkItem is a struct to hold info HashWorkItem hwi = new HashWorkItem(di.FullName, fi.FullName, isOriginal, count, exactImageDataComparison); hwi.DirectoryName = di.FullName; hwi.FileName = fi.FullName; hwi.Index = indexer; hwi.IsOriginal = isOriginal; // get hashes in threads ThreadPool.QueueUserWorkItem(new WaitCallback(GetHashForItem), hwi); // see below } }} private void GetHashForItem(object state){ // do the work HashWorkItem hwi = (HashWorkItem)state; // This is a struct to hold information. ImageConverter converter = new ImageConverter(); long pressure = new FileInfo(hwi.FileName).Length; GC.AddMemoryPressure(pressure); Bitmap bmp = new Bitmap(1, 1); // I actually release the bmp below with bmp.Dispose(), so init here #region TRY block try { bmp = (Bitmap)Bitmap.FromFile(hwi.FileName); // zero problems here byte[] bytes = new byte[1]; if (!hwi.CompareExactImageData) { // **************************************************************** bytes = (byte[])converter.ConvertTo(bmp, bytes.GetType()); // #0 THIS IS WHERE I GET THE GDI+ ERROR // **************************************************************** } else if (hwi.CompareExactImageData) { // **************************************************************** bytes = GetBytesFromBitmap(bmp); // #1 THIS IS WHERE I GET THE GDI+ ERROR -- See method below // **************************************************************** } SuperFastHashUnsafe sfh = new SuperFastHashUnsafe(); hwi.ItemHash = sfh.Hash(bytes); try { // invoke the delegate to store the results this.Invoke(new StoreHashResultsDelegate(StoreHashResults), hwi); } catch { // NEVER happens } } catch (OutOfMemoryException memEx) { Console.Write("***** Out of memory: " + hwi.FileName + " *** " + memEx.Message); Thread.Sleep(2000); // queue the item again... ThreadPool.QueueUserWorkItem(new WaitCallback(GetHashForItem), hwi); } finally { if (bmp != null) { bmp.Dispose(); bmp = null; GC.Collect(); // this makes no difference it seems } } #endregion GC.RemoveMemoryPressure(pressure);} public delegate void StoreHashResultsDelegate(HashWorkItem hwi); public void StoreHashResults(HashWorkItem hwi){ // blah - here for reference - zero problems here} /// <summary>/// Get a byte array of pixel data from a Bitmap./// </summary>/// <param name="bmp">The bitmap to get the pixel data from.</param>/// <returns>A byte array of pixel data from a bitmap.</returns>private static byte[] GetBytesFromBitmap(Bitmap bmp){ byte[] result = null; if (bmp == null) return result; // Create a memory stream to hold the bitmap. MemoryStream ms = new MemoryStream(); // **************************************************************** bmp.Save(ms, ImageFormat.Bmp); // #2 THIS IS WHERE I GET THE GDI+ ERROR // **************************************************************** result = ms.ToArray(); return result;}
Does anyone have any ideas how to troubleshoot that?
I've spent about 2 days on this and I'm not getting anywhere.
I've run around looking into everything I can, including LHO (Large Heap Objects), but so far no joy.
BTW - I'm running through directories of photos that are around 4~7 MB each (10 MP camera JPGs) with around 500 photos per directory. The program craps out randomly, sometimes ripping through 200, or 300 or 400 or 500 before dying. Directories are around 3 GB.
Any ideas, pointers or whatever would be appreciated.
Hey, and just for fun, I'll put a $10.00 donation bounty on it for the first person to point me to a solution~! :D
Renegade:
I think I'm closing in on a solution...
http://stackoverflow.com/questions/686950/large-object-heap-fragmentation
worstje:
I can't really test it, but assuming fragmentation is your problem, I think I see the problem you are dealing with. Mind you, I do not usually use GDI+, especially not in .NET, but I assume you have a good reason for using GDI+ as opposed to the System.Drawing classes. Hell, I have no clue how the GDI+ stuff is defined in that context.
Anyhow, on to what I think causes your problem. On line 29, you create a new Bitmap of 1x1 dimensions. The moment you go into the try {} block, you immediately call Bitmap.FromFile() and re-assign the value. In other words: you are creating an object you never use, and the garbage collector only collects that stuff when it feels like it - but at the same time extra objects mean less available continuous blocks. You do not need to create a Bitmap object in order to call FromFile as it is a static function. (See MSDN.) So, the solution ought to be as simple as initializing bmp to null.
I am not sure what all of your GC calls and the likes do. However, GDI+ is not managed, so I doubt tuning/prodding/abusing the GC will do very much for you in this case. But I could be wrong - I'm no .NET expert. :)
Renegade:
I can't really test it, but assuming fragmentation is your problem, I think I see the problem you are dealing with. Mind you, I do not usually use GDI+, especially not in .NET, but I assume you have a good reason for using GDI+ as opposed to the System.Drawing classes. Hell, I have no clue how the GDI+ stuff is defined in that context.
Anyhow, on to what I think causes your problem. On line 29, you create a new Bitmap of 1x1 dimensions. The moment you go into the try {} block, you immediately call Bitmap.FromFile() and re-assign the value. In other words: you are creating an object you never use, and the garbage collector only collects that stuff when it feels like it - but at the same time extra objects mean less available continuous blocks. You do not need to create a Bitmap object in order to call FromFile as it is a static function. (See MSDN.) So, the solution ought to be as simple as initializing bmp to null.
I am not sure what all of your GC calls and the likes do. However, GDI+ is not managed, so I doubt tuning/prodding/abusing the GC will do very much for you in this case. But I could be wrong - I'm no .NET expert. :)
-worstje (January 20, 2011, 03:40 AM)
--- End quote ---
Thanks for taking a stab at it.
Underneath the hood, the System.Drawing.Bitmap uses GDI+... ;( So that's why it's there... ;(
The new Bitmap(1, 1) makes no difference. I put that in so that I could use the try... catch... finally... block because I'm having such a miserable time with this (normally I'd do it like you mention). i.e. In order to to dispose of bmp, it must be constructed first, and not just assigned -- see line 75 there.
I'm still no closer to solving this.
It plugs along fine, then all of a sudden just spins out of control and memory goes boom boom bang bang while I go boo hoo cry cry~! ;(
worstje:
That makes no sense to me, sorry. FromFile() constructs a new bitmap object and returns it for as far I can see. If it weren't ever constructed, then how could you ever use the image you loaded? Unless I am a really big noob at .NET, there is no such thing as a reference to an object that isn't constructed. There's just 'null'.
Navigation
[0] Message Index
[#] Next page
Go to full version