topbanner_forum
  *

avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Thursday March 28, 2024, 1:37 pm
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Last post Author Topic: C# GDI+ Problem with byte[] and Bitmap - Memory Issues  (Read 32135 times)

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« on: January 20, 2011, 12:58 AM »
Well, I'm still working on some requests for Duplicate Photo Finder, but I'm encountering some odd memory problems.

I'm getting "Out of memory" exceptions, and in particular, "A generic error occurred in GDI+".

If the above sounds interesting, read on, otherwise don't because it gets ugly.

I'm running through photos and processing them in threads in a threadpool. I basically queue up several hundred, then let it go.

What happens is that it very happily rips through the photos, then all of a sudden, memory usage spikes real quickly (over say 1~5 seconds), and then I get the GDI+ error. What's happening there is that I run out of memory. The application sucks up 200~500 MB while running (I have 8 GB of RAM), but when it spikes, it dies at 1 GB to 1.4 GB of RAM (from Task Manager).

During those few seconds when the program begins to blow up, the number of threads that it's working on triples out of the blue. It goes from 6~8 threads to BOOM! 24, 26, 28, 30, 31 CRASH~! It hits the GDI+ error.

I've run "memprofiler" and cannot locate any memory leaks. It tells me:

Disposed instance (Ignore...)
5 types have instances that have been disposed but not GCed.
Investigate the types below for more information.
System.Drawing.Bitmap, System.Drawing.Graphics, System.Drawing.Region, DeviceContext, WindowsFont

Instance queued for finalization (Ignore...)
One type has instances that are queued for finalization. This can indicate that a Finalizer method is stuck, which will prevent instances from being finalized and cause memory leaks.
Investigate the type below for more information.
DeviceContext

Pinned instance (Ignore...)
3 types have instances that are pinned in memory.
Investigate the types below for more information.
System.Object, System.Object[], System.String

Instance indirectly rooted by finalizer queue (Ignore...)
2 types have instances that are indirectly rooted by the finalizer queue. This can indicate that a Finalizer method is stuck, which will prevent instances from being finalized and cause memory leaks.
Investigate the types below for more information.
System.Collections.Stack, System.Object[]

Large instance (Ignore...)
2 types have instances that are located in the large object heap.
Investigate the types below for more information.
System.Byte[], System.Object[]

Undisposed instances (unclassified) (Ignore...)
16 types have instances that have been garbage collected without being properly disposed.
Investigate the types below for more information.
SafeTokenHandle, SafeWaitHandle, System.Drawing.Bitmap, System.Drawing.Font, System.IO.BinaryReader, System.IO.MemoryStream, UnmanagedMemoryStream, ResourceReader, RuntimeResourceSet, ExecutionContext, (...)


Here's enough code to see what's going on. I've edited it somewhat to make it shorter (e.g. remove Console.WriteLine & debugging stuff.)

Code: C# [Select]
  1. private void QueueHashWorkItem(DirectoryInfo di, bool isOriginal, int count, bool exactImageDataComparison)
  2. {
  3.         int indexer = -1;
  4.         foreach (FileInfo fi in di.GetFiles())
  5.         {
  6.                 if (fi.Extension.ToLower() == ".jpg")
  7.                 {
  8.                         indexer++;
  9.                         // HashWorkItem is a struct to hold info
  10.                         HashWorkItem hwi = new HashWorkItem(di.FullName, fi.FullName, isOriginal, count, exactImageDataComparison);
  11.                         hwi.DirectoryName = di.FullName;
  12.                         hwi.FileName = fi.FullName;
  13.                         hwi.Index = indexer;
  14.                         hwi.IsOriginal = isOriginal;
  15.  
  16.                         // get hashes in threads
  17.                         ThreadPool.QueueUserWorkItem(new WaitCallback(GetHashForItem), hwi); // see below
  18.                 }
  19.         }
  20. }
  21.  
  22. private void GetHashForItem(object state)
  23. {
  24.         // do the work
  25.         HashWorkItem hwi = (HashWorkItem)state; // This is a struct to hold information.
  26.         ImageConverter converter = new ImageConverter();
  27.         long pressure = new FileInfo(hwi.FileName).Length;
  28.         GC.AddMemoryPressure(pressure);
  29.         Bitmap bmp = new Bitmap(1, 1); // I actually release the bmp below with bmp.Dispose(), so init here
  30.  
  31.         #region TRY block
  32.         try
  33.         {
  34.                 bmp = (Bitmap)Bitmap.FromFile(hwi.FileName); // zero problems here
  35.                 byte[] bytes = new byte[1];
  36.  
  37.                 if (!hwi.CompareExactImageData)
  38.                 {
  39.                         // ****************************************************************
  40.                         bytes = (byte[])converter.ConvertTo(bmp, bytes.GetType()); // #0 THIS IS WHERE I GET THE GDI+ ERROR
  41.                         // ****************************************************************
  42.                 }
  43.                 else if (hwi.CompareExactImageData)
  44.                 {
  45.                         // ****************************************************************
  46.                         bytes = GetBytesFromBitmap(bmp); // #1 THIS IS WHERE I GET THE GDI+ ERROR -- See method below
  47.                         // ****************************************************************
  48.  
  49.                 }
  50.  
  51.                 SuperFastHashUnsafe sfh = new SuperFastHashUnsafe();
  52.                 hwi.ItemHash = sfh.Hash(bytes);
  53.  
  54.                 try
  55.                 {
  56.                         // invoke the delegate to store the results
  57.                         this.Invoke(new StoreHashResultsDelegate(StoreHashResults), hwi);
  58.                 }
  59.                 catch
  60.                 {
  61.                         // NEVER happens
  62.                 }
  63.         }
  64.         catch (OutOfMemoryException memEx)
  65.         {
  66.                 Console.Write("***** Out of memory: " + hwi.FileName + " *** " + memEx.Message);
  67.                 Thread.Sleep(2000);
  68.                 // queue the item again...
  69.                 ThreadPool.QueueUserWorkItem(new WaitCallback(GetHashForItem), hwi);
  70.         }
  71.         finally
  72.         {
  73.                 if (bmp != null)
  74.                 {
  75.                         bmp.Dispose();
  76.                         bmp = null;
  77.                         GC.Collect(); // this makes no difference it seems
  78.                 }
  79.         }
  80.         #endregion
  81.  
  82.         GC.RemoveMemoryPressure(pressure);
  83. }
  84.  
  85.  
  86. public delegate void StoreHashResultsDelegate(HashWorkItem hwi);
  87.  
  88. public void StoreHashResults(HashWorkItem hwi)
  89. {
  90.         // blah - here for reference - zero problems here
  91. }
  92.  
  93. /// <summary>
  94. /// Get a byte array of pixel data from a Bitmap.
  95. /// </summary>
  96. /// <param name="bmp">The bitmap to get the pixel data from.</param>
  97. /// <returns>A byte array of pixel data from a bitmap.</returns>
  98. private static byte[] GetBytesFromBitmap(Bitmap bmp)
  99. {
  100.         byte[] result = null;
  101.         if (bmp == null)
  102.                 return result;
  103.         // Create a memory stream to hold the bitmap.
  104.         MemoryStream ms = new MemoryStream();
  105.         // ****************************************************************
  106.         bmp.Save(ms, ImageFormat.Bmp);  // #2 THIS IS WHERE I GET THE GDI+ ERROR
  107.         // ****************************************************************
  108.         result = ms.ToArray();
  109.  
  110.         return result;
  111. }

Does anyone have any ideas how to troubleshoot that?

I've spent about 2 days on this and I'm not getting anywhere.

I've run around looking into everything I can, including LHO (Large Heap Objects), but so far no joy.

BTW - I'm running through directories of photos that are around 4~7 MB each (10 MP camera JPGs) with around 500 photos per directory. The program craps out randomly, sometimes ripping through 200, or 300 or 400 or 500 before dying. Directories are around 3 GB.

Any ideas, pointers or whatever would be appreciated.

Hey, and just for fun, I'll put a $10.00 donation bounty on it for the first person to point me to a solution~! :D

Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker
« Last Edit: January 20, 2011, 01:00 AM by Renegade »

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #1 on: January 20, 2011, 02:41 AM »
I think I'm closing in on a solution...

http://stackoverflow...t-heap-fragmentation

Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

worstje

  • Honorary Member
  • Joined in 2009
  • **
  • Posts: 588
  • The Gent with the White Hat
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #2 on: January 20, 2011, 03:40 AM »
I can't really test it, but assuming fragmentation is your problem, I think I see the problem you are dealing with. Mind you, I do not usually use GDI+, especially not in .NET, but I assume you have a good reason for using GDI+ as opposed to the System.Drawing classes. Hell, I have no clue how the GDI+ stuff is defined in that context.

Anyhow, on to what I think causes your problem. On line 29, you create a new Bitmap of 1x1 dimensions. The moment you go into the try {} block, you immediately call Bitmap.FromFile() and re-assign the value. In other words: you are creating an object you never use, and the garbage collector only collects that stuff when it feels like it - but at the same time extra objects mean less available continuous blocks. You do not need to create a Bitmap object in order to call FromFile as it is a static function. (See MSDN.) So, the solution ought to be as simple as initializing bmp to null.

I am not sure what all of your GC calls and the likes do. However, GDI+ is not managed, so I doubt tuning/prodding/abusing the GC will do very much for you in this case. But I could be wrong - I'm no .NET expert. :)
« Last Edit: January 20, 2011, 03:42 AM by worstje »

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #3 on: January 20, 2011, 05:20 AM »
I can't really test it, but assuming fragmentation is your problem, I think I see the problem you are dealing with. Mind you, I do not usually use GDI+, especially not in .NET, but I assume you have a good reason for using GDI+ as opposed to the System.Drawing classes. Hell, I have no clue how the GDI+ stuff is defined in that context.

Anyhow, on to what I think causes your problem. On line 29, you create a new Bitmap of 1x1 dimensions. The moment you go into the try {} block, you immediately call Bitmap.FromFile() and re-assign the value. In other words: you are creating an object you never use, and the garbage collector only collects that stuff when it feels like it - but at the same time extra objects mean less available continuous blocks. You do not need to create a Bitmap object in order to call FromFile as it is a static function. (See MSDN.) So, the solution ought to be as simple as initializing bmp to null.

I am not sure what all of your GC calls and the likes do. However, GDI+ is not managed, so I doubt tuning/prodding/abusing the GC will do very much for you in this case. But I could be wrong - I'm no .NET expert. :)

Thanks for taking a stab at it.

Underneath the hood, the System.Drawing.Bitmap uses GDI+... ;( So that's why it's there... ;(

The new Bitmap(1, 1) makes no difference. I put that in so that I could use the try... catch... finally... block because I'm having such a miserable time with this (normally I'd do it like you mention). i.e. In order to to dispose of bmp, it must be constructed first, and not just assigned -- see line 75 there.

I'm still no closer to solving this.

It plugs along fine, then all of a sudden just spins out of control and memory goes boom boom bang bang while I go boo hoo cry cry~! ;(

Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

worstje

  • Honorary Member
  • Joined in 2009
  • **
  • Posts: 588
  • The Gent with the White Hat
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #4 on: January 20, 2011, 05:28 AM »
That makes no sense to me, sorry. FromFile() constructs a new bitmap object and returns it for as far I can see. If it weren't ever constructed, then how could you ever use the image you loaded? Unless I am a really big noob at .NET, there is no such thing as a reference to an object that isn't constructed. There's just 'null'.

Stoic Joker

  • Honorary Member
  • Joined in 2008
  • **
  • Posts: 6,646
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #5 on: January 20, 2011, 07:22 AM »
Mind you common sense is telling me to shutup ... But I just gotta ask...

In lines 37 - 43, why is hwi.CompareExactImageData being evaluated (basically) twice?

I'm reading it as:
if(condition = false) do bla
else if(same condition is not false) do bla

Wouldn't that tend to spike the work load?

Also being that there is no trailing else, then if condition is neither true or false (which granted should be impossible) Just ignore it. < I'm guessing that part was cut out for clarity)

worstje

  • Honorary Member
  • Joined in 2009
  • **
  • Posts: 588
  • The Gent with the White Hat
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #6 on: January 20, 2011, 07:28 AM »
Mind you common sense is telling me to shutup ... But I just gotta ask...

In lines 37 - 43, why is hwi.CompareExactImageData being evaluated (basically) twice?

...

Wouldn't that tend to spike the work load?

It's unclear code. Unless it is a property with very expensive processing behind it, it will do next to nothing except cost a few more processor cycles. And I somehow expect the CLR to be smart enough to recognise that case.

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #7 on: January 20, 2011, 07:50 AM »
GetBytesFromBitmap() doesn't dispose of the MemoryStream?

I agree with Worstje you shouldn't allocate the 1x1 bitmap, just set it to null - you already have a null-check in your finally-block before disposing, anyway.

Also, iirc GDI+ is not properly re-entrant, so you'll have to be pretty careful there; dunno if it has to run on the UI thread, though.
- carpe noctem

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #8 on: January 20, 2011, 07:51 AM »
That makes no sense to me, sorry. FromFile() constructs a new bitmap object and returns it for as far I can see. If it weren't ever constructed, then how could you ever use the image you loaded? Unless I am a really big noob at .NET, there is no such thing as a reference to an object that isn't constructed. There's just 'null'.

(I think I messed up that post above a bit.)

Take a quick look back. You'll see the original bitmap is defined outside the try catch. In order to dispose of it, it must not be null, and must be constructed.

Bitmap bmp; only allocates memory, which at that point is just allocated memory and null. So I have to use "Bitmap bmp = new Bitmap(1, 1);" so that it's not null.

Here, like this:


Code: C# [Select]
  1. Bitmap bmp = new Bitmap(1, 1); // I actually release the bmp below with bmp.Dispose(), so init here
  2.  
  3. #region TRY block
  4. try
  5. {
  6.         bmp = (Bitmap)Bitmap.FromFile(hwi.FileName); // zero problems here
  7.         //
  8. }
  9. catch (OutOfMemoryException memEx)
  10. {
  11.         //
  12. }
  13. finally
  14. {
  15.         if (bmp != null)
  16.         {
  17.                 bmp.Dispose(); // ********* can't dispose of null
  18.                 bmp = null;
  19.                 GC.Collect(); // this makes no difference it seems
  20.         }
  21. }


But you're right - Bitmap.FromFile() returns a fully constructed Bitmap object.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #9 on: January 20, 2011, 07:52 AM »
Mind you common sense is telling me to shutup ... But I just gotta ask...

In lines 37 - 43, why is hwi.CompareExactImageData being evaluated (basically) twice?

I'm reading it as:
if(condition = false) do bla
else if(same condition is not false) do bla

Wouldn't that tend to spike the work load?

Also being that there is no trailing else, then if condition is neither true or false (which granted should be impossible) Just ignore it. < I'm guessing that part was cut out for clarity)


It's a boolean, and doesn't make much difference. I simply put it there so that I could just type an else if I needed. The additional overhead is effectively nothing.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #10 on: January 20, 2011, 07:58 AM »
GetBytesFromBitmap() doesn't dispose of the MemoryStream?

I agree with Worstje you shouldn't allocate the 1x1 bitmap, just set it to null - you already have a null-check in your finally-block before disposing, anyway.

Also, iirc GDI+ is not properly re-entrant, so you'll have to be pretty careful there; dunno if it has to run on the UI thread, though.

The actual code right now is this:

Code: C# [Select]
  1. /// <summary>
  2. /// Get a byte array of pixel data from a Bitmap.
  3. /// </summary>
  4. /// <param name="bmp">The bitmap to get the pixel data from.</param>
  5. /// <returns>A byte array of pixel data from a bitmap.</returns>
  6. private static byte[] GetBytesFromBitmap(Bitmap bmp)
  7. {
  8.         byte[] result = null;
  9.         if (bmp == null)
  10.                 return result;
  11.         // Create a memory stream to hold the bitmap.
  12.         Thread.Sleep(500);
  13.         MemoryStream ms = new MemoryStream();
  14.         bmp.Save(ms, ImageFormat.Bmp);
  15.  
  16.         if (true)
  17.         {
  18.                 byte[] tmp = ms.ToArray();
  19.                 result = new byte[tmp.Length];
  20.                 tmp.CopyTo(result, 0);
  21.                 ms.Close();
  22.                 ms.Dispose();
  23.                 ms = null;
  24.                 bmp.Dispose();
  25.                 bmp = null;
  26.                 //GC.Collect();
  27.         }
  28.         else
  29.         {
  30.                 result = ms.ToArray();
  31.         }
  32.  
  33.         return result;
  34. }

I was trying to see if disposing of the MemoryStream worked to help fix things, but it makes no difference. The error still pops up in the same place.

For why the bitmap is constructed, and not just having memory allocated, I have that 2 posts above. (Catching up here...)

Also, iirc GDI+ is not properly re-entrant, so you'll have to be pretty careful there; dunno if it has to run on the UI thread, though.

I'm not sure what you mean.

But it's not running on the UI thread. Everything above is running from a threadpool like you can see in the QueueHashWorkItem method. Loop through and queue work items. Lots of them. Then sit back and wait for them... to crash~!  :'(  :o  :huh:

Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #11 on: January 20, 2011, 07:59 AM »
Mind you common sense is telling me to shutup ... But I just gotta ask...

In lines 37 - 43, why is hwi.CompareExactImageData being evaluated (basically) twice?

...

Wouldn't that tend to spike the work load?

It's unclear code. Unless it is a property with very expensive processing behind it, it will do next to nothing except cost a few more processor cycles. And I somehow expect the CLR to be smart enough to recognise that case.

It's actually a very simple struct:

Code: C# [Select]
  1. public struct HashWorkItem
  2.     {
  3.         public HashWorkItem(string directory, string file, bool isOriginal, int total, bool exactImageDataComparison)
  4.         {
  5.             _itemHash = 0;
  6.             _index = -1;
  7.             _fileName = file;
  8.             _directoryName = directory;
  9.             _isOriginal = isOriginal;
  10.             _total = total;
  11.             _compareExactImageData = exactImageDataComparison;
  12.         }
  13.  
  14.         private UInt32 _itemHash;
  15.  
  16.         public UInt32 ItemHash
  17.         {
  18.             get { return _itemHash; }
  19.             set { _itemHash = value; }
  20.         }
  21.  
  22.         private int _index;
  23.  
  24.         public int Index
  25.         {
  26.             get { return _index; }
  27.             set { _index = value; }
  28.         }
  29.  
  30.         private int _total;
  31.  
  32.         public int Total
  33.         {
  34.             get { return _total; }
  35.             set { _total = value; }
  36.         }
  37.  
  38.         private string _fileName;
  39.  
  40.         public string FileName
  41.         {
  42.             get { return _fileName; }
  43.             set { _fileName = value; }
  44.         }
  45.  
  46.         private string _directoryName;
  47.  
  48.         public string DirectoryName
  49.         {
  50.             get { return _directoryName; }
  51.             set { _directoryName = value; }
  52.         }
  53.  
  54.         private bool _isOriginal;
  55.  
  56.         public bool IsOriginal
  57.         {
  58.             get { return _isOriginal; }
  59.             set { _isOriginal = value; }
  60.         }
  61.  
  62.         private bool _compareExactImageData;
  63.  
  64.         public bool CompareExactImageData
  65.         {
  66.             get { return _compareExactImageData; }
  67.             set { _compareExactImageData = value; }
  68.         }
  69.     }

Nothing special there.
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #12 on: January 20, 2011, 08:05 AM »
For why the bitmap is constructed, and not just having memory allocated, I have that 2 posts above. (Catching up here...)
-Renegade
Yes, and I commented that you don't need to construct the object, since you have a null-check before disposing it... so just set it to null instead of wasting time on constructing a never-used object :)

As for disposing the MemoryStream, I took a look at in in reflector, and it doesn't look like there's apparent memory leaks if you don't dispose it (but there's some async I/O event cancelling in it's base Stream class) - but since it implements IDisposable, not disposing of it is an error, no matter what the current implementation does.

As for the threading: I'm not sure about the re-entrancy of GDI+, but GDI definitely wasn't re-entrant, and I have a recollection of GDI+ not being, either. This means you should never have more than one thread issuing GDI+ code... when you throw background tasks on a Threadpool, multiple tasks may run in parallel.

Wouldn't be surprised if your issues are because of two threads entering GDI+ at the same time and fucking up some internal state.
- carpe noctem

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #13 on: January 20, 2011, 08:15 AM »
For why the bitmap is constructed, and not just having memory allocated, I have that 2 posts above. (Catching up here...)
-Renegade
Yes, and I commented that you don't need to construct the object, since you have a null-check before disposing it... so just set it to null instead of wasting time on constructing a never-used object :)

As for disposing the MemoryStream, I took a look at in in reflector, and it doesn't look like there's apparent memory leaks if you don't dispose it (but there's some async I/O event cancelling in it's base Stream class) - but since it implements IDisposable, not disposing of it is an error, no matter what the current implementation does.

As for the threading: I'm not sure about the re-entrancy of GDI+, but GDI definitely wasn't re-entrant, and I have a recollection of GDI+ not being, either. This means you should never have more than one thread issuing GDI+ code... when you throw background tasks on a Threadpool, multiple tasks may run in parallel.

Wouldn't be surprised if your issues are because of two threads entering GDI+ at the same time and fucking up some internal state.

Ah... that check should come out actually...

Changing:

Bitmap bmp;// = new Bitmap(1, 1);

Results in this:

if (bmp != null)

Throwing a "Use of unassigned local variable 'bmp'" compiler error (won't compile like that).

If what you're saying about GDI+ is right, I may end up doing it all consecutively in a single thread. I hope not...
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #14 on: January 20, 2011, 08:18 AM »
"Bitmap bmp = null;" :)

If what you're saying about GDI+ is right, I may end up doing it all consecutively in a single thread. I hope not...
-Renegade
Well, I've heard of people resorting to running one process per CPU core in order to parallelize GDI/GDI+ operations - it's per-process re-entrant, but not per-thread-in-process.
- carpe noctem

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #15 on: January 20, 2011, 08:37 AM »
"Bitmap bmp = null;" :)

DOH~!

I've got bada on my brain, and thinking in terms of 2-phase construction...

If what you're saying about GDI+ is right, I may end up doing it all consecutively in a single thread. I hope not...
-Renegade
Well, I've heard of people resorting to running one process per CPU core in order to parallelize GDI/GDI+ operations - it's per-process re-entrant, but not per-thread-in-process.

I'd really like to avoid that as I'm not sure that it's worth the overhead to start a new process for each photo. Though a process per core that ran long and returned a result set might be good...

Still... that error is pissing me off and I'd like to see it go away rather than run away from it and do a work around. I hate working around problems if I can avoid it. In other words, I'd rather work around having to do work arounds. :)


http://www.codeproje...ects____Trouble.aspx
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

worstje

  • Honorary Member
  • Joined in 2009
  • **
  • Posts: 588
  • The Gent with the White Hat
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #16 on: January 20, 2011, 09:41 AM »
Is it indeed Win2003 or WinXP you are devving on? Otherwise, that problem you just linked to wouldn't even be related. (Sorry if you posted it somewhere up there; I didn't notice it.)

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #17 on: January 20, 2011, 09:49 AM »
You definitely don't want to start a process per photo, as process creation (especially for .NET apps) is relatively expensive. If you know you're going to process craploads of images, a process per core is a viable solution; IPC does complicate matters a bit, but it's workable.

I'd suggest you start by testing your current code in a serialized fasion, avoiding the worker pool thread, and see if the memory issue still crops up - if it doesn't, I'd definitely bet my money on a GDI+ reentrancy issue.
- carpe noctem

nharding

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 36
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #18 on: January 20, 2011, 12:15 PM »
I've been using my DCDisplay database which is doing 1 image a second (on average, since it has to extract them from archive), and that has been running for 24 hours. I think you should probably decode the images on 1 thread, and do the checksum on a second thread so you would only ever have 1 image being decoded at once.

I found a problem with Texture2D.FromFile (it failed with CYMK format jpg files) so when that fails I use the code below

if (imageCache == null)
{
  memory.Seek(0, SeekOrigin.Begin);   //Probably safest to do this
  using (System.Drawing.Image image = SD.Bitmap.FromStream(memory))
  using (MemoryStream bitmapStream = new MemoryStream())
  {
     image.Save(bitmapStream, System.Drawing.Imaging.ImageFormat.Bmp);
     bitmapStream.Seek(0, SeekOrigin.Begin);
     imageCache = Texture2D.FromFile(context, bitmapStream);
     bitmapStream.Close();
  }
}                           

But for your case you can just use the bitmap stream to calculate the checksum (BMP is a really simple format, and it means all the exif data has been stripped)

Neil Harding

Renegade

  • Charter Member
  • Joined in 2005
  • ***
  • Posts: 13,288
  • Tell me something you don't know...
    • View Profile
    • Renegade Minds
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #19 on: January 20, 2011, 08:12 PM »
I've been using my DCDisplay database which is doing 1 image a second (on average, since it has to extract them from archive), and that has been running for 24 hours. I think you should probably decode the images on 1 thread, and do the checksum on a second thread so you would only ever have 1 image being decoded at once.

I found a problem with Texture2D.FromFile (it failed with CYMK format jpg files) so when that fails I use the code below

if (imageCache == null)
{
  memory.Seek(0, SeekOrigin.Begin);   //Probably safest to do this
  using (System.Drawing.Image image = SD.Bitmap.FromStream(memory))
  using (MemoryStream bitmapStream = new MemoryStream())
  {
     image.Save(bitmapStream, System.Drawing.Imaging.ImageFormat.Bmp);
     bitmapStream.Seek(0, SeekOrigin.Begin);
     imageCache = Texture2D.FromFile(context, bitmapStream);
     bitmapStream.Close();
  }
}                           

But for your case you can just use the bitmap stream to calculate the checksum (BMP is a really simple format, and it means all the exif data has been stripped)

Neil Harding

The current code (refactored massively for brevity) looks like this:

Code: C# [Select]
  1. private void GetHashForItem(object state)
  2. {
  3.         HashWorkItem hwi = (HashWorkItem)state;
  4.         ImageConverter converter = new ImageConverter();
  5.         Bitmap bmp = (Bitmap)Bitmap.FromFile(hwi.FileName);
  6.         byte[] bytes = new byte[1];
  7.  
  8.         // 1 - Get byte[] data from file or pixels
  9.         if (!hwi.CompareExactImageData) // file
  10.         {
  11.                 bytes = (byte[])converter.ConvertTo(bmp, bytes.GetType());
  12.         }
  13.         else if (hwi.CompareExactImageData) // pixels
  14.         {
  15.                 bytes = GetBytesFromBitmap(bmp);
  16.         }
  17.  
  18.         // 2 - The hashing
  19.         SuperFastHashUnsafe sfh = new SuperFastHashUnsafe();
  20.         hwi.ItemHash = sfh.Hash(bytes);
  21.  
  22.         try
  23.         {
  24.                 // 3 - invoke the delegate to store the results
  25.                 this.Invoke(new StoreHashResultsDelegate(StoreHashResults), hwi);
  26.         }
  27.         catch
  28.         {
  29.         }
  30. }

Which boils down to 3 steps:

1) Get bytes
2) Do hash
3) Queue to store results

The hashing has presented no problems so far. I'd be concerned that there would be a performance hit by spinning that off into another thread as I'd be passing a large byte[] array. While it is a reference type, I just don't really like the idea of throwing references around like that. Not sure why. It just strikes me as "out of scopishly odd". :) The method that it's in would go out of scope while the thread is running. Dunno... It just seems like I could start getting null references there if the byte[] were collected (GC'd) before the hashing had completed. Am I too worried?

The one method there is using the bitmap stream already. It saves to the memorystream, then gets the byte[] from there. It must be a byte[] as the memorystream itself can't be hashed.

But either way, I'm still not much closer to solving the problem. The only things I have on my plate left to try out are:

1) MemoryFailPoint -- Try to determine if there is enough memory available, then handle the situation -- http://www.codeproje...ects____Trouble.aspx
2) Single threading -- get rid of the threadpool and do them all sequentially

I just don't like either. I want things to work as you would expect them to work... I hate work arounds. Well, except for working around work arounds... :) :D  :o
Slow Down Music - Where I commit thought crimes...

Freedom is the right to be wrong, not the right to do wrong. - John Diefenbaker

f0dder

  • Charter Honorary Member
  • Joined in 2005
  • ***
  • Posts: 9,153
  • [Well, THAT escalated quickly!]
    • View Profile
    • f0dder's place
    • Read more about this member.
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #20 on: January 21, 2011, 05:32 AM »
I'd be concerned that there would be a performance hit by spinning that off into another thread as I'd be passing a large byte[] array. While it is a reference type, I just don't really like the idea of throwing references around like that. Not sure why. It just strikes me as "out of scopishly odd". :)
There's no performance problems in passing a reference around, no matter how big the object it refers to is. A reference is just a pointer - so 4 bytes on x86, 8 bytes on x64. Spinning off into a thread has some performance overhead, especially if you create new threads manually instead of using a threadpool.

The method that it's in would go out of scope while the thread is running. Dunno... It just seems like I could start getting null references there if the byte[] were collected (GC'd) before the hashing had completed. Am I too worried?
Scope, as in methods, curly-blocks etc. have pretty much nothing to do with object lifetime in .NET apps, kick that idea out of your head ASAP :). Objects get GC'ed "at some point after there's no longer any references to them", which could mean aeons after they "go out of scope"... but it can even happen within the method where the variable referencing the object was initially allocated.

1) MemoryFailPoint -- Try to determine if there is enough memory available, then handle the situation -- http://www.codeproje...ects____Trouble.aspx
That article specifically mentions that the XP memory manager has trouble, while Vista+ doesn't. If you are getting your issues on non-XP, chasing this is probably a waste of time.

2) Single threading -- get rid of the threadpool and do them all sequentially
This is what I'd investigate first, given that GDI+ isn't too happy about re-entrancy.
- carpe noctem

worstje

  • Honorary Member
  • Joined in 2009
  • **
  • Posts: 588
  • The Gent with the White Hat
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #21 on: January 21, 2011, 05:39 AM »
Random and not very (but still somewhat) relevant: don't use .NET if you are developing an application with XP as a supported OS. You'll regret it and regret it and regret it even more.

There's so many bugs in the .NET implementation for XP (or whatever system components it uses) that you spend crappy oodles of time figuring out why the hell you get random-as-hell crashes with totally useless stacktraces and shit.

I bet that at some point, Vista and the likes will head into the same general direction after W8 comes out. :(

cranioscopical

  • Friend of the Site
  • Supporting Member
  • Joined in 2006
  • **
  • Posts: 4,776
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #22 on: January 21, 2011, 09:02 AM »
Vista and the likes will head into the same general direction after W8 comes out. :(

Just another reason why I can't W8.

worstje

  • Honorary Member
  • Joined in 2009
  • **
  • Posts: 588
  • The Gent with the White Hat
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #23 on: January 21, 2011, 10:48 AM »
Touche. :) I tip my hat to you, good sir.

wraith808

  • Supporting Member
  • Joined in 2006
  • **
  • default avatar
  • Posts: 11,186
    • View Profile
    • Donate to Member
Re: C# GDI+ Problem with byte[] and Bitmap - Memory Issues
« Reply #24 on: January 21, 2011, 10:50 AM »
Random and not very (but still somewhat) relevant: don't use .NET if you are developing an application with XP as a supported OS. You'll regret it and regret it and regret it even more.

There's so many bugs in the .NET implementation for XP (or whatever system components it uses) that you spend crappy oodles of time figuring out why the hell you get random-as-hell crashes with totally useless stacktraces and shit.

I bet that at some point, Vista and the likes will head into the same general direction after W8 comes out. :(

I haven't had that experience... my primary job is developing in .NET for XP... unless you mean .NET4/WPF... I'm doing WinForms for all of my projects (currently developing custom MVVM architecture for WinForms using parts of Prism/Unity)