avatar image

Welcome, Guest. Please login or register.
Did you miss your activation email?

Login with username, password and session length
  • Wednesday December 2, 2020, 4:43 am
  • Proudly celebrating 15+ years online.
  • Donate now to become a lifetime supporting member of the site and get a non-expiring license key for all of our programs.
  • donate

Show Posts

This section allows you to view all posts made by this member. Note that you can only see posts made in areas you currently have access to.

Messages - StCroixSkipper [ switch to compact view ]

Pages: [1] 2next
I've made several changes this last week. First, I did a bad thing calling long running functions in the UI thread. So I've put these long running functions, the ones that collect all the files, folders and USN Journal Entries, in a separate thread. Now you no longer get the "Application not responding" message.

I've also added an event handler for double clicking on the results listbox. If you double click on an entry in the results listbox and it happens to be a file that exists, it will call the Process.Start() function and launch the file in the 'associated' application if an association exists for the file type.

I've attached the code and executable.

Wouldn't you know it!  I found a couple of bugs.  Here is the corrected code.

Options  StCroixSkipper's USN Journal Explorer v1.2
12 May 2010  3 Comments
Posted by StCroixSkipper 
I've added a button, 'List Files' to allow you to list and display all the files on a volume that match a filter string.

I enumerate through the entire Master File Table to find all the files whose extensions match those in the filter. Why would I want to read the Master File Table rather than enumerating through the disk with Directory.FindFiles()? For an active volume it works about 10 faster.

It works much like 'List Folders'. If you click on a specific file, you'll see a popup window that shows the detail. If you double click on the file name or path in the detail window it calls Process.Start() with the file name to start the application associated with the file extension.

I've also changed the 'Update USN Journal State' from a MessageBox to a Window so I could control where it is placed.

In both 'List Folders' and 'List Files' I've sorted the output before displaying it.

The executable and source files are attached.

Here is my task list as of now.  I would welcome more ideas.  Understand that I work on this during my free time.  But I will make progress.

1. Add the code to take care of the condition where, when processing a USN Journal entry, I find that the parent folder no longer exists.

2. Add a function/button that, given a search filter, finds all files that match that filter.

3. Add a function/button that identifies all duplicate files on a volume.

I did forget to mention one item.  There is a slight problem with the code. 

The NTFS filesystem writes entries to the USN Journal as changes are made to a volume.  So, if you modify a file in a directory and then delete the directory you'll have first an entry in the journal about the modification, then an entry about the file delete and then an entry about the directory delete.  Since each entry only has the file's 64-bit File Reference Number and the file's parent's 64-bit File Reference Number it is possible to find a journal entry whose parent has been deleted and, hence, does not exist.  I have code to fix this problem but for this little application I haven't added it yet.

p.s.  I have downloaded the executable and source code and I have been able to unpack it and use it.

Developer's Corner / StCroixSkipper's NTFS USN Journal Explorer
« on: May 10, 2010, 02:15 PM »
I've long been interested in promoting the use of the NTFS USN Journal in Windows applications.  And while .Net doesn't provide access to the USN Journal, I've written the PInvoke code to make it easy to use from C#. I've written a NtfsUsnJournal class in C# that exposes the functionality of the USN Journal but masks most of the complexity required to use it.  I've also written a WPF application I call StCroixSkipper's USN Journal Explore that lets you select a volume, create or delete the USN Journal on that volume, save the USN Journal state, query the USN Journal and view the changes (USN Journal entries) to the volume.  Since accessing the USN Journal is much like reading directly from the NTFS Master File Table, I also let you enumerate all of the directories on the volume and display them in a ListBox.  

I'd like to share the current version with folks to get a discussion going.  I'd like to see ideas on how the USN Journal could be used in applications to speed up things like identifying all the files of a given type on a volume or finding all of the identical files on a volume or finding all the files who attributes meet certain criteria.  

I'd also like to solicit comments on the code.  Out of this process I'd like to bring together a community of folks who have an interest in sharing thier ideas and code relative to the USN Journal.

Remember that to use or access the USN Journal you'll need admin rights.  So right click on the application and select the "Run as administrator" menu option.  If you run without admin rights, you'll get an 'Access denied' message.

Also, the 'View Changes' button reads the USN Journal entries from the saved state to the current state.  So if you have no saved state, you'll get 'No saved state' message.

Here is the code and a copy of the executable attached as an exe file and then the entire project with source code in a zip file. I'll try to download these to make sure I can unzip them.

A little background on the USN Journal:
Jeffrey Cooperstein and Jeffrey Richter have co-authored a great article Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained If you are interested in the USN Journal, this article is well worth your time.

Each entry in the USN Journal is assigned a 64 bit number which is the location with in the actual USN Journal file of the entry. USN Journal entries can be different sizes since at least the filenames are different lengths.

Each file or directory on a volume has a unique 64 bit File Reference Number. Files can have the same names excluding the path but no two files or directories can have the same File Reference Number.
Here is the format of a USN Journal entry.

        /// <summary>
        /// Contains the USN Record Length(32bits), USN(64bits), File Reference Number(64bits), 
        /// Parent File Reference Number(64bits), Reason Code(32bits), File Attributes(32bits),
        /// File Name Length(32bits), the File Name Offset(32bits) and the File Name.
        /// </summary>
        public class USN_RECORD
            private const int FR_OFFSET = 8;
            private const int PFR_OFFSET = 16;
            private const int USN_OFFSET = 24;
            private const int REASON_OFFSET = 40;
            public const int FA_OFFSET = 52;
            private const int FNL_OFFSET = 56;
            private const int FN_OFFSET = 58;
            public UInt32 RecordLength;
            public UInt64 FileReferenceNumber;
            public UInt64 ParentFileReferenceNumber;
            public Int64 Usn;
            public UInt32 Reason;
            public UInt32 FileAttributes;
            public Int32 FileNameLength;
            public Int32 FileNameOffset;
            public string FileName = string.Empty;
            /// <summary>
            /// USN Record Constructor
            /// </summary>
            /// <param name="p">Buffer of bytes representing the USN Record</param>
            public USN_RECORD(IntPtr p)
                this.RecordLength = (UInt32)Marshal.ReadInt32(p);
                this.FileReferenceNumber = (UInt64)Marshal.ReadInt64(p, FR_OFFSET);
                this.ParentFileReferenceNumber = (UInt64)Marshal.ReadInt64(p, PFR_OFFSET);
                this.Usn = Marshal.ReadInt64(p, USN_OFFSET);
                this.Reason = (UInt32)Marshal.ReadInt32(p, REASON_OFFSET);
                this.FileAttributes = (UInt32)Marshal.ReadInt32(p, FA_OFFSET);
                this.FileNameLength = Marshal.ReadInt16(p, FNL_OFFSET);
                this.FileNameOffset = Marshal.ReadInt16(p, FN_OFFSET);
                FileName = Marshal.PtrToStringUni(new IntPtr(p.ToInt32() + this.FileNameOffset), this.FileNameLength / sizeof(char));

In the USN_RECORD class you can easily see what the entry contains. You read a buffer from the USN Journal and parse thru the buffer to find and marshal the values from the buffer into the fields of the USN Journal entry.

So we now have access to the record length, the file reference number, the parent's file reference number (which is the file reference number of the directory which contains the file), the reason why an entry was generated for this
file or directory, the files attributes, the length of the files name in bytes and the offset into the entry of the file name.

As it so happens, the Master File Table is made up of entries that are of the same format so this one class works for both the USN Journal and enumerating all of the files and directories on a volume using the Master File Table. It takes about one tenth of the time to enumerate all of the files and directories on a volume by reading the Master File Table on an active drive than the traditional FindFirst() FindNext() approach.

I've added a help section and expose links to GIMP, GIMP Tutorials, etc.

I developed it on Vista.  It runs on both Vista and Windows 7.

I plan to make some more changes over the holidays.

For instance, I'm going to allow you to drag and drop photos in the preview pane so you can use it like the old 35mm light table sorter.  I'm thinking that if you move pictures around, I'll add a sequence number to the metadata so you can sort on the sequence number or the DateTaken.

I'm looking at and for libraries that can update image metadata in situ.

Alternate streams only work on NTFS partitions, correct.  That is one reason I abandoned alternate streams.

Yes it some of the pictures are of the Mendanhall Glacier...

I agree.  I need an encoder that takes the raw image pixels with the changed metadata and creates a new jpg file without munging the image pixels. 

In the past I've written the metadata to an alternate stream on the jpg so I could update the metadata and not even open the jpg.  But then I had to write my own search engine and database (which I did) or write another filter for the Desktop Search Engine Indexer.  But then I lost the metadata when I uploaded images to the net.

I'm researching ways to modify the metadata and saving the new metadata without recompressing the image data.

I can access the stream of raw image data.  What I would like is an encoder that takes the updated metadata and a stream of the raw image data and creates a new jpg file from the two without recompressing the image data.

How are the other apps solving this problem? 

Another idea I have is to make each thumbnail the thumbnail preview pane like a little window with a 'Maximize' button in the top left corner.  When you hit the maximize button, I would display the fullsize picture with all of the metadata...  Much like double clicking on a thumbnail now.

I also have this idea for the thumbnail preview pane.

Now you can see the thumbnails of the image files you find in a directory.  As the mouse flies over a thumbnail, I bring up a window with a larger better quality image with some of the metadata displayed.

What if I changed it so that when the mouse flies over the thumbnail, I bring up a window that overlays the thumbnail with the metadata with clickable keywords.  The overlay would be partially transparent so you could see the image underneath.  I could also put the context menu items in this window.  It would make the thumbnail preview pane less jumpy.

I could make the thumbnails a little higher quality so that you could enlarge them without the loss of quality.

The activity on this thread has died down some.  I'd like to keep it going.  I've looked at and taken some ideas from them I'd like to incorporate in OverExposure.  Here is what I have in mind...

1. shows the 'tags', keywords in OverExposure with the bookmarks and they are clickable.  In OverExposure's case I'd have to display the keywords with the thumbnails so you could click on a tag of a thumbnail and you get all of the images that have that keyword.

2. I also like the edit tags feature of bookmark.  I think I am going to modify the right click context menu and allow you to edit the keywords from the thumbnail preview pane.

3. also shows you some statistics like how many images have a given tag, the top ten tags, ...

4. When editing tags, it has an auto complete feature.

All of these features I think are interesting.  Any thoughts?  If there is interest, I'll implement them so we can really see how we feel about them.

Per delwoode's suggestion I've created a new topic, "New image organizer feedback wanted" and have made OverExposure available there.

This is the latest copy.  It has Search enabled so it requires Search 4.0 to be installed.

1. It now asks if you want to change the name of the photo file using the directory name as a prefix.  It allows you to check a checkbox to prevent the message from showing up again.  To reset the rename options, right click on a folder.

2. It allows you to right click on a thumbnail in the preview pane and select either edit or delete so you don't have double click on a thumbnail to go to the fullsized photo pane to edit or delete an image file.

3. You can not only move files you can copy files by holding down the control key.

The reason I don't show all the photos in all subfolders is memory.  I have 4000 photos in some folders.  If I had folders several levels deep each with 4000 photos, I'd run out of memory and the response would degrade.  I opted to only show the photos in a folder to keep memory usage down. 

The other feedback I've gotten on Photo Gallery, which does show all the photos in all the subdirectories, is that they have a hard time finding which directory any given file is in.

Sorry, All my machines have Search 4.0 installed.

Here is a copy of OverExposure with Search Disabled.  It should work on XP machines without Search 4.0 installed.  I haven't tried it on XP without Search 4.0, all of my XP machines have XP.

If it doesn't work, let me know and I'll fix it.

I can take out the search code in one version so that OverExposure will work on XP without Windows Search but there won't be any search capability.  But you could provide feedback on the other aspects...

Anyone interested in a version that runs on XP without search?

I'm  interested in the problems you had with Search 4.0. Is anyone else experiencing problems with Search 4.0?

I'll make a new copy available later this week for anyone who is interested.


I took your advice.  I've made the renaming of files optional.  I've added a dialog box that appears when you drop thumbnails onto a folder.  It informs you, by default, OverExposure renames the file using the directory name as the prefix.  If you don't want that behavior, uncheck the 'Rename File(s)' checkbox.

There is also another 'Don't show this dialog box again' checkbox that you can check to avoid answering the question each time.

The reset back to the defaults of Rename File(s) and show the dialog box is in the folder context menu.

Someone else asked that I allow moving and copying of files (when you drag and drop thumbnails on to a folder.)  I've implemented copy too.  Simply hold down the left or right control keys when you drag thumbnails.

Someone else asked that they be able to delete thumbnails from the thumbnail view.  I've implemented that also.  I added a context menu with a 'Delete' menu item to each thumbnail.

I cobbled together a machine and installed XP on it.  I can easily reproduce the error.  Here is what I did to resolve it.

Problem: I use the Desktop search engine thru the Microsoft.Search.Interop.dll.  The problem occurs when I try to load the dll.

Resolution: Install Windows Search 4.0
Here is the link.

I ran the install on my 'new' XP machine, double clicked on OverExposure.exe and it work.

*** Warning ***
OverExposure re-encodes BitmapImage each time it modifies the metadata.   Since I make it really easy to modify the metadata, this occurs alot.

I've actually accessed the pixel bits separately and calculated MD5 digests on just the pixel bits.  After changing the metadata, they are different.  This is a known problem with the jpeg format.

It could be solved by writing another encoder that simply encodes the metadata but preserves the pixel bits.

Also, until i resolve it, OverExposure doesn't appear to work on XP systems.  The error occurs before my code is executed.  (I've put a try/catch block around the code in the constructor and I don't catch the error.)

I have found an XP system I tried to run OverExposure on and was able to reproduce the same error.  It has .Net 3.5 SP 1 installed.  

I thought it might be the WIC functions causing this but according to the documentation, WIC is part of .Net 3.0.  I tried downloading the WIC and installing it but I got the message that "a newer version was already installed."

I put a try/catch block around the code in the constructor and tried it again.  I don't catch this exception.

I think it must be a library I'm using that comes with Vista but not with XP.

Pages: [1] 2next