Topics

1

Developer's Corner / StCroixSkipper's NTFS USN Journal Explorer

« on: May 10, 2010, 02:15 PM »

I've long been interested in promoting the use of the NTFS USN Journal in Windows applications.  And while .Net doesn't provide access to the USN Journal, I've written the PInvoke code to make it easy to use from C#. I've written a NtfsUsnJournal class in C# that exposes the functionality of the USN Journal but masks most of the complexity required to use it.  I've also written a WPF application I call StCroixSkipper's USN Journal Explore that lets you select a volume, create or delete the USN Journal on that volume, save the USN Journal state, query the USN Journal and view the changes (USN Journal entries) to the volume.  Since accessing the USN Journal is much like reading directly from the NTFS Master File Table, I also let you enumerate all of the directories on the volume and display them in a ListBox.  

I'd like to share the current version with folks to get a discussion going.  I'd like to see ideas on how the USN Journal could be used in applications to speed up things like identifying all the files of a given type on a volume or finding all of the identical files on a volume or finding all the files who attributes meet certain criteria.  

I'd also like to solicit comments on the code.  Out of this process I'd like to bring together a community of folks who have an interest in sharing thier ideas and code relative to the USN Journal.

Remember that to use or access the USN Journal you'll need admin rights.  So right click on the application and select the "Run as administrator" menu option.  If you run without admin rights, you'll get an 'Access denied' message.

Also, the 'View Changes' button reads the USN Journal entries from the saved state to the current state.  So if you have no saved state, you'll get 'No saved state' message.


Here is the code and a copy of the executable attached as an exe file and then the entire project with source code in a zip file. I'll try to download these to make sure I can unzip them.

A little background on the USN Journal:
Jeffrey Cooperstein and Jeffrey Richter have co-authored a great article Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained If you are interested in the USN Journal, this article is well worth your time.

Each entry in the USN Journal is assigned a 64 bit number which is the location with in the actual USN Journal file of the entry. USN Journal entries can be different sizes since at least the filenames are different lengths.

Each file or directory on a volume has a unique 64 bit File Reference Number. Files can have the same names excluding the path but no two files or directories can have the same File Reference Number.
Here is the format of a USN Journal entry.

        /// <summary> 
        /// Contains the USN Record Length(32bits), USN(64bits), File Reference Number(64bits),  
        /// Parent File Reference Number(64bits), Reason Code(32bits), File Attributes(32bits), 
        /// File Name Length(32bits), the File Name Offset(32bits) and the File Name. 
        /// </summary> 
        public class USN_RECORD 
        { 
            private const int FR_OFFSET = 8; 
            private const int PFR_OFFSET = 16; 
            private const int USN_OFFSET = 24; 
            private const int REASON_OFFSET = 40; 
            public const int FA_OFFSET = 52; 
            private const int FNL_OFFSET = 56; 
            private const int FN_OFFSET = 58; 
 
            public UInt32 RecordLength; 
            public UInt64 FileReferenceNumber; 
            public UInt64 ParentFileReferenceNumber; 
            public Int64 Usn; 
            public UInt32 Reason; 
            public UInt32 FileAttributes; 
            public Int32 FileNameLength; 
            public Int32 FileNameOffset; 
            public string FileName = string.Empty; 
 
            /// <summary> 
            /// USN Record Constructor 
            /// </summary> 
            /// <param name="p">Buffer of bytes representing the USN Record</param> 
            public USN_RECORD(IntPtr p) 
            { 
                this.RecordLength = (UInt32)Marshal.ReadInt32(p); 
                this.FileReferenceNumber = (UInt64)Marshal.ReadInt64(p, FR_OFFSET); 
                this.ParentFileReferenceNumber = (UInt64)Marshal.ReadInt64(p, PFR_OFFSET); 
                this.Usn = Marshal.ReadInt64(p, USN_OFFSET); 
                this.Reason = (UInt32)Marshal.ReadInt32(p, REASON_OFFSET); 
                this.FileAttributes = (UInt32)Marshal.ReadInt32(p, FA_OFFSET); 
                this.FileNameLength = Marshal.ReadInt16(p, FNL_OFFSET); 
                this.FileNameOffset = Marshal.ReadInt16(p, FN_OFFSET); 
                FileName = Marshal.PtrToStringUni(new IntPtr(p.ToInt32() + this.FileNameOffset), this.FileNameLength / sizeof(char)); 
            } 
        }

In the USN_RECORD class you can easily see what the entry contains. You read a buffer from the USN Journal and parse thru the buffer to find and marshal the values from the buffer into the fields of the USN Journal entry.

So we now have access to the record length, the file reference number, the parent's file reference number (which is the file reference number of the directory which contains the file), the reason why an entry was generated for this
file or directory, the files attributes, the length of the files name in bytes and the offset into the entry of the file name.

As it so happens, the Master File Table is made up of entries that are of the same format so this one class works for both the USN Journal and enumerating all of the files and directories on a volume using the Master File Table. It takes about one tenth of the time to enumerate all of the files and directories on a volume by reading the Master File Table on an active drive than the traditional FindFirst() FindNext() approach.

2

General Software Discussion / "New image organizer feedback wanted"

« on: November 25, 2009, 11:51 AM »

I've read the image organizer reviews and I still haven't found anything that does what I need.  I'm a developer so I've written my own.  I've outlined the major features below and I'd appreciate your feedback.  I know there are more good ideas out there and I'd like to hear them.  I've attached a zip file that contains the .exe and a Microsoft interop dll so you can use it, compare it to others you've tried.

I call it 'OverExposure' because I was intimidated trying to track and find specific images with the number of digital images I was taking and people were sending to me. 

I'd like any feedback you may have from ideas to make it more useful to pesky bugs that you'd like fixed to a better name.  Understand that this is a work in progress.  Not all features have been implemented. You may want to challenge design decisions.  This may become a forum where we can discuss design issues in an open peer review setting.  Here is what it does. I'll try to implement suggested new features or changes.

I like the UI of Microsoft's Photo Gallery UI and have reproduced the parts I like.  The concept is like the light tables we used to sort 35 mm slides updated with all of our digital technology. The left pane is a TreeView with folders where images are stored, keywords, and publishing destinations like iPod, FaceBook, YouTube, and Flickr. The right pane displays the thumbnails of the images in the selected folder.  You can adjust the size of the thumbnails with a slider.  You can select and drag thumbnail images from from the left pane and drop them onto folders, keywords or publishing destinations.  If you drop a thumbnail image on a folder, the image is moved to the new folder.  If you drop a thumbnail image onto a keyword, the keyword is added to the image's metadata.  If you drop a thumbnail image onto a publishing destination, it would 'publish' the image to the site.

If you double click on a thumbnail image, the left pane shows the fullsize picture along with all of the image's metadata.

To search for a specific image or images I use the Desktop Search Engine.  Image files are read by the 'Indexer' which chooses a filter based on the file type to access the metadata. To search for images based on image metadata, you need to ensure that the folders which contain your digital images are included in the folders that the 'indexer' is watching.  You can control the folders the 'indexer' is watching using the 'Indexing Options' member of the 'Control Panel'.

(Note: Publishing images to 'publish' destinations is not yet implemented.  Support for music and video files is not yet implemented. I will make these features available as they become defined and implemented.)

Here are the things I've done differently.

1. The folder icons are graphics that tell you if there are images and/or audio and/or video files in the folder.  You can visually tell if you have a) photos only, b) photos and music, c) photos and videos, d) photos, music and videos, e) music only, f) video only or g) music and video files. (I look into the folders as you open them to see what kinds of files it contains and choose which folder icon to display.)

2. Photo Gallery only lets easily you look in "special" folders like "My Pictures" or "My Videos" by default.  You can look in other folders by "adding" additional folders that reside anywhere on your system. I let you see and navigate to all my drives both fixed and removable along with your special folders.  You can plug your camera into a USB port and see the photos directly from the camera. You can drag and drop thumbnails from your camera directly onto existing or new folders you've created.

3. I hate the P1010485.jpg names my camera gives image files.  When you drag one or more selected thumbnails from the left pane to a folder, it not only moves the image file to the new folder but it renames the file to the folder name with a unique suffix (_1.jpg).  Then it updates the image metadata by removing the keyword that contains the old image file's folder name and adds the new folder name as a keyword to the metadata. If you give folders descriptive names, images are instantly searchable.

4.  If you rename a folder, it will create a new folder with the new folder name. It removes the old folder name from the image metadata keywords.  And it adds the new folder's name as a keyword in the metadata. Renaming folders allows you to instantly change the metadata for all the image files in a folder and updates the Destop Search Index.

5.  Photo Gallery only lets you add keywords (or tags in Photo Gallery), modify Rating and specify a title.  I expose and allow you to edit Title, Keywords, DateTaken, Author, Subject, Rating, Comment and Copyright metadata fields.  I also show you which camera was used to take the picture.

6. I don't like the way Photo Gallery lets you remove tags.  You need to select them and then get a context menu and select the only option: "Remove Tag".  I show you the metadata fields in text.  When you fly over the text with the mouse, I make the text editable so you can quickly make any changes you want.  Keywords are shown together separated by semacolons.  Double click on a word to select the entire word.  'Control-a' selects all the text in the metadata field.  The delete key removes it or you can start typing over it.  Ctrl-z undoes the previous action.   

7. My wife and friends are always sending me pictures taken when their cameras have bad Date and Time values set.  I allow you to change the date and time the images were taken so when you sort by DateTaken you'll see the photos in the order they were taken.

7.  When you drag one or more thumbnails onto a tag, Photo Gallery applies the tag as a keyword to the metadata of all images selected.  I do the same but I've changed the name 'tag' to 'keyword' and added a check box to each tag. You can check one or more keywords and then drag one or more thumbnails onto a keyword.  The keyword onto which the image or images were dropped and all the 'checked' keywords will be added to the metadata of each image file.

8. I've also divided up the keywords or tags into categories.  The initial categories are 'Events', 'People', 'Places' and 'Things'.  But you can create new categories or change the names of categories or delete categories and organize them any way you like.

9. I have trouble find my photos in Photo Gallery.  It gives me the filename but not the full path in many cases.  I have thousands of photos.  I hate struggling to find them.  I display the full path of each file with the metadata.

10. Many times I want to do more than just "Auto Adjust", "Adjust Exposure", "Adjust Color", "Crop Picture", or "Fix Red Eye".  I have an Edit menu option which brings up GIMP if it is installed so I can really manipulate the image.  I could as easily bring up another photo editting application like PhotoShop but it costs hundreds of dollars for much the same functionality. GIMP is OpenSource and free.

11. I use the desktop Search Engine to find images that match a certain criteria just like Photo Gallery.

12. I really want to support video and music files.  I'm working on these now. The folders show whether they contain audio or video but I haven't written the code to display and play them yet.

13.  I want to be able to publish photos to destinations like my FaceBook account or to Flickr or YouTube or to my iPod.  There may be other destinations I need to support.
 

14. I don't organize the photos by date taken yet but I will.  I've done it before and its easy.

In the next post, I'll show some screen shots of the application as it now stands.

Note.  I've used this on my desktop which is a 32-bit quad processor with 3 gb of memory running Vista.  I've also installed this on my laptop which is a 64-bit dual processor with 4 gb of memory running Vista.  To use OverExposure simply copy the OverExposure.exe and the freely redistributable Microsoft.Search.Interop.dll to a folder and double click on OverExposure.exe.

This is the first time I've tried attachments so we'll see if this works.