Other Software > Developer's Corner
C# GDI+ Problem with byte[] and Bitmap - Memory Issues
Renegade:
"Bitmap bmp = null;" :)
-f0dder (January 20, 2011, 08:18 AM)
--- End quote ---
DOH~!
I've got bada on my brain, and thinking in terms of 2-phase construction...
If what you're saying about GDI+ is right, I may end up doing it all consecutively in a single thread. I hope not...-Renegade
--- End quote ---
Well, I've heard of people resorting to running one process per CPU core in order to parallelize GDI/GDI+ operations - it's per-process re-entrant, but not per-thread-in-process.
-f0dder (January 20, 2011, 08:18 AM)
--- End quote ---
I'd really like to avoid that as I'm not sure that it's worth the overhead to start a new process for each photo. Though a process per core that ran long and returned a result set might be good...
Still... that error is pissing me off and I'd like to see it go away rather than run away from it and do a work around. I hate working around problems if I can avoid it. In other words, I'd rather work around having to do work arounds. :)
http://www.codeproject.com/KB/cs/Large_Objects____Trouble.aspx
worstje:
Is it indeed Win2003 or WinXP you are devving on? Otherwise, that problem you just linked to wouldn't even be related. (Sorry if you posted it somewhere up there; I didn't notice it.)
f0dder:
You definitely don't want to start a process per photo, as process creation (especially for .NET apps) is relatively expensive. If you know you're going to process craploads of images, a process per core is a viable solution; IPC does complicate matters a bit, but it's workable.
I'd suggest you start by testing your current code in a serialized fasion, avoiding the worker pool thread, and see if the memory issue still crops up - if it doesn't, I'd definitely bet my money on a GDI+ reentrancy issue.
nharding:
I've been using my DCDisplay database which is doing 1 image a second (on average, since it has to extract them from archive), and that has been running for 24 hours. I think you should probably decode the images on 1 thread, and do the checksum on a second thread so you would only ever have 1 image being decoded at once.
I found a problem with Texture2D.FromFile (it failed with CYMK format jpg files) so when that fails I use the code below
--- ---if (imageCache == null)
{
memory.Seek(0, SeekOrigin.Begin); //Probably safest to do this
using (System.Drawing.Image image = SD.Bitmap.FromStream(memory))
using (MemoryStream bitmapStream = new MemoryStream())
{
image.Save(bitmapStream, System.Drawing.Imaging.ImageFormat.Bmp);
bitmapStream.Seek(0, SeekOrigin.Begin);
imageCache = Texture2D.FromFile(context, bitmapStream);
bitmapStream.Close();
}
}
But for your case you can just use the bitmap stream to calculate the checksum (BMP is a really simple format, and it means all the exif data has been stripped)
Neil Harding
Renegade:
I've been using my DCDisplay database which is doing 1 image a second (on average, since it has to extract them from archive), and that has been running for 24 hours. I think you should probably decode the images on 1 thread, and do the checksum on a second thread so you would only ever have 1 image being decoded at once.
I found a problem with Texture2D.FromFile (it failed with CYMK format jpg files) so when that fails I use the code below
--- ---if (imageCache == null)
{
memory.Seek(0, SeekOrigin.Begin); //Probably safest to do this
using (System.Drawing.Image image = SD.Bitmap.FromStream(memory))
using (MemoryStream bitmapStream = new MemoryStream())
{
image.Save(bitmapStream, System.Drawing.Imaging.ImageFormat.Bmp);
bitmapStream.Seek(0, SeekOrigin.Begin);
imageCache = Texture2D.FromFile(context, bitmapStream);
bitmapStream.Close();
}
}
But for your case you can just use the bitmap stream to calculate the checksum (BMP is a really simple format, and it means all the exif data has been stripped)
Neil Harding
-nharding (January 20, 2011, 12:15 PM)
--- End quote ---
The current code (refactored massively for brevity) looks like this:
--- Code: C# ---private void GetHashForItem(object state){ HashWorkItem hwi = (HashWorkItem)state; ImageConverter converter = new ImageConverter(); Bitmap bmp = (Bitmap)Bitmap.FromFile(hwi.FileName); byte[] bytes = new byte[1]; // 1 - Get byte[] data from file or pixels if (!hwi.CompareExactImageData) // file { bytes = (byte[])converter.ConvertTo(bmp, bytes.GetType()); } else if (hwi.CompareExactImageData) // pixels { bytes = GetBytesFromBitmap(bmp); } // 2 - The hashing SuperFastHashUnsafe sfh = new SuperFastHashUnsafe(); hwi.ItemHash = sfh.Hash(bytes); try { // 3 - invoke the delegate to store the results this.Invoke(new StoreHashResultsDelegate(StoreHashResults), hwi); } catch { }}
Which boils down to 3 steps:
1) Get bytes
2) Do hash
3) Queue to store results
The hashing has presented no problems so far. I'd be concerned that there would be a performance hit by spinning that off into another thread as I'd be passing a large byte[] array. While it is a reference type, I just don't really like the idea of throwing references around like that. Not sure why. It just strikes me as "out of scopishly odd". :) The method that it's in would go out of scope while the thread is running. Dunno... It just seems like I could start getting null references there if the byte[] were collected (GC'd) before the hashing had completed. Am I too worried?
The one method there is using the bitmap stream already. It saves to the memorystream, then gets the byte[] from there. It must be a byte[] as the memorystream itself can't be hashed.
But either way, I'm still not much closer to solving the problem. The only things I have on my plate left to try out are:
1) MemoryFailPoint -- Try to determine if there is enough memory available, then handle the situation -- http://www.codeproject.com/KB/cs/Large_Objects____Trouble.aspx
2) Single threading -- get rid of the threadpool and do them all sequentially
I just don't like either. I want things to work as you would expect them to work... I hate work arounds. Well, except for working around work arounds... :) :D :o
Navigation
[0] Message Index
[#] Next page
[*] Previous page
Go to full version