Welcome Guest.   Make a donation to an author on the site November 23, 2014, 09:40:37 PM  *

Please login or register.
Or did you miss your validation email?


Login with username and password (forgot your password?)
Why not become a lifetime supporting member of the site with a one-time donation of any amount? Your donation entitles you to a ton of additional benefits, including access to exclusive discounts and downloads, the ability to enter monthly free software drawings, and a single non-expiring license key for all of our programs.


You must sign up here before you can post and access some areas of the site. Registration is totally free and confidential.
 
Read the Practical Guide to DonationCoder.com Forum Search Features
   
   Forum Home   Thread Marks Chat! Downloads Search Login Register  
Pages: [1]   Go Down
  Reply  |  New Topic  |  Print  
Author Topic: StCroixSkipper's NTFS USN Journal Explorer  (Read 10227 times)
StCroixSkipper
Supporting Member
**
Posts: 38

View Profile Give some DonationCredits to this forum member
« on: May 10, 2010, 02:15:21 PM »

I've long been interested in promoting the use of the NTFS USN Journal in Windows applications.  And while .Net doesn't provide access to the USN Journal, I've written the PInvoke code to make it easy to use from C#. I've written a NtfsUsnJournal class in C# that exposes the functionality of the USN Journal but masks most of the complexity required to use it.  I've also written a WPF application I call StCroixSkipper's USN Journal Explore that lets you select a volume, create or delete the USN Journal on that volume, save the USN Journal state, query the USN Journal and view the changes (USN Journal entries) to the volume.  Since accessing the USN Journal is much like reading directly from the NTFS Master File Table, I also let you enumerate all of the directories on the volume and display them in a ListBox.  

I'd like to share the current version with folks to get a discussion going.  I'd like to see ideas on how the USN Journal could be used in applications to speed up things like identifying all the files of a given type on a volume or finding all of the identical files on a volume or finding all the files who attributes meet certain criteria.  

I'd also like to solicit comments on the code.  Out of this process I'd like to bring together a community of folks who have an interest in sharing thier ideas and code relative to the USN Journal.

Remember that to use or access the USN Journal you'll need admin rights.  So right click on the application and select the "Run as administrator" menu option.  If you run without admin rights, you'll get an 'Access denied' message.

Also, the 'View Changes' button reads the USN Journal entries from the saved state to the current state.  So if you have no saved state, you'll get 'No saved state' message.


Here is the code and a copy of the executable attached as an exe file and then the entire project with source code in a zip file. I'll try to download these to make sure I can unzip them.

A little background on the USN Journal:
Jeffrey Cooperstein and Jeffrey Richter have co-authored a great article Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained If you are interested in the USN Journal, this article is well worth your time.

Each entry in the USN Journal is assigned a 64 bit number which is the location with in the actual USN Journal file of the entry. USN Journal entries can be different sizes since at least the filenames are different lengths.

Each file or directory on a volume has a unique 64 bit File Reference Number. Files can have the same names excluding the path but no two files or directories can have the same File Reference Number.
Here is the format of a USN Journal entry.

        /// <summary>
        /// Contains the USN Record Length(32bits), USN(64bits), File Reference Number(64bits), 
        /// Parent File Reference Number(64bits), Reason Code(32bits), File Attributes(32bits),
        /// File Name Length(32bits), the File Name Offset(32bits) and the File Name.
        /// </summary>
        public class USN_RECORD
        {
            private const int FR_OFFSET = 8;
            private const int PFR_OFFSET = 16;
            private const int USN_OFFSET = 24;
            private const int REASON_OFFSET = 40;
            public const int FA_OFFSET = 52;
            private const int FNL_OFFSET = 56;
            private const int FN_OFFSET = 58;
 
            public UInt32 RecordLength;
            public UInt64 FileReferenceNumber;
            public UInt64 ParentFileReferenceNumber;
            public Int64 Usn;
            public UInt32 Reason;
            public UInt32 FileAttributes;
            public Int32 FileNameLength;
            public Int32 FileNameOffset;
            public string FileName = string.Empty;
 
            /// <summary>
            /// USN Record Constructor
            /// </summary>
            /// <param name="p">Buffer of bytes representing the USN Record</param>
            public USN_RECORD(IntPtr p)
            {
                this.RecordLength = (UInt32)Marshal.ReadInt32(p);
                this.FileReferenceNumber = (UInt64)Marshal.ReadInt64(p, FR_OFFSET);
                this.ParentFileReferenceNumber = (UInt64)Marshal.ReadInt64(p, PFR_OFFSET);
                this.Usn = Marshal.ReadInt64(p, USN_OFFSET);
                this.Reason = (UInt32)Marshal.ReadInt32(p, REASON_OFFSET);
                this.FileAttributes = (UInt32)Marshal.ReadInt32(p, FA_OFFSET);
                this.FileNameLength = Marshal.ReadInt16(p, FNL_OFFSET);
                this.FileNameOffset = Marshal.ReadInt16(p, FN_OFFSET);
                FileName = Marshal.PtrToStringUni(new IntPtr(p.ToInt32() + this.FileNameOffset), this.FileNameLength / sizeof(char));
            }
        }

In the USN_RECORD class you can easily see what the entry contains. You read a buffer from the USN Journal and parse thru the buffer to find and marshal the values from the buffer into the fields of the USN Journal entry.

So we now have access to the record length, the file reference number, the parent's file reference number (which is the file reference number of the directory which contains the file), the reason why an entry was generated for this
file or directory, the files attributes, the length of the files name in bytes and the offset into the entry of the file name.

As it so happens, the Master File Table is made up of entries that are of the same format so this one class works for both the USN Journal and enumerating all of the files and directories on a volume using the Master File Table. It takes about one tenth of the time to enumerate all of the files and directories on a volume by reading the Master File Table on an active drive than the traditional FindFirst() FindNext() approach.

* UsnJournalProject.exe (44 KB - downloaded 570 times.)
* UsnJournalProject.zip (239.9 KB - downloaded 564 times.)
« Last Edit: May 10, 2010, 02:30:55 PM by StCroixSkipper » Logged
StCroixSkipper
Supporting Member
**
Posts: 38

View Profile Give some DonationCredits to this forum member
« Reply #1 on: May 10, 2010, 02:22:00 PM »

I did forget to mention one item.  There is a slight problem with the code. 

The NTFS filesystem writes entries to the USN Journal as changes are made to a volume.  So, if you modify a file in a directory and then delete the directory you'll have first an entry in the journal about the modification, then an entry about the file delete and then an entry about the directory delete.  Since each entry only has the file's 64-bit File Reference Number and the file's parent's 64-bit File Reference Number it is possible to find a journal entry whose parent has been deleted and, hence, does not exist.  I have code to fix this problem but for this little application I haven't added it yet.

p.s.  I have downloaded the executable and source code and I have been able to unpack it and use it.
Logged
StCroixSkipper
Supporting Member
**
Posts: 38

View Profile Give some DonationCredits to this forum member
« Reply #2 on: May 12, 2010, 09:42:36 AM »

Here is my task list as of now.  I would welcome more ideas.  Understand that I work on this during my free time.  But I will make progress.

1. Add the code to take care of the condition where, when processing a USN Journal entry, I find that the parent folder no longer exists.

2. Add a function/button that, given a search filter, finds all files that match that filter.

3. Add a function/button that identifies all duplicate files on a volume.
Logged
StCroixSkipper
Supporting Member
**
Posts: 38

View Profile Give some DonationCredits to this forum member
« Reply #3 on: May 13, 2010, 08:31:39 AM »

Options  StCroixSkipper's USN Journal Explorer v1.2
12 May 2010  3 Comments
Posted by StCroixSkipper 
I've added a button, 'List Files' to allow you to list and display all the files on a volume that match a filter string.

I enumerate through the entire Master File Table to find all the files whose extensions match those in the filter. Why would I want to read the Master File Table rather than enumerating through the disk with Directory.FindFiles()? For an active volume it works about 10 faster.

It works much like 'List Folders'. If you click on a specific file, you'll see a popup window that shows the detail. If you double click on the file name or path in the detail window it calls Process.Start() with the file name to start the application associated with the file extension.

I've also changed the 'Update USN Journal State' from a MessageBox to a Window so I could control where it is placed.

In both 'List Folders' and 'List Files' I've sorted the output before displaying it.

The executable and source files are attached.

* UsnJournalProject.exe (49 KB - downloaded 206 times.)
* UsnJournalProject.zip (260.58 KB - downloaded 212 times.)
Logged
StCroixSkipper
Supporting Member
**
Posts: 38

View Profile Give some DonationCredits to this forum member
« Reply #4 on: May 13, 2010, 10:03:03 AM »

Wouldn't you know it!  I found a couple of bugs.  Here is the corrected code.

* UsnJournalProject.exe (49 KB - downloaded 216 times.)
* UsnJournalProject.zip (261 KB - downloaded 200 times.)
* UsnJournalProject.exe (49 KB - downloaded 224 times.)
* UsnJournalProject.zip (261 KB - downloaded 265 times.)
Logged
mouser
First Author
Administrator
*****
Posts: 33,692



see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #5 on: May 13, 2010, 10:07:32 AM »

Very cool.
Logged
StCroixSkipper
Supporting Member
**
Posts: 38

View Profile Give some DonationCredits to this forum member
« Reply #6 on: May 16, 2010, 08:28:11 AM »

I've made several changes this last week. First, I did a bad thing calling long running functions in the UI thread. So I've put these long running functions, the ones that collect all the files, folders and USN Journal Entries, in a separate thread. Now you no longer get the "Application not responding" message.

I've also added an event handler for double clicking on the results listbox. If you double click on an entry in the results listbox and it happens to be a file that exists, it will call the Process.Start() function and launch the file in the 'associated' application if an association exists for the file type.

I've attached the code and executable.

* UsnJournalProject.exe (52 KB - downloaded 729 times.)
* UsnJournalProject.zip (280.15 KB - downloaded 727 times.)
Logged
f0dder
Charter Honorary Member
***
Posts: 8,774



[Well, THAT escalated quickly!]

see users location on a map View Profile WWW Read user's biography. Give some DonationCredits to this forum member
« Reply #7 on: July 25, 2010, 07:09:56 PM »

Just wanted to give a nod of approval - USN journal parsing is definitely something a lot of programs could benefit from, especially backup programs that don't run continuously. An app like SpiderOak, for instance, takes ~10 minutes to scan my source/docs partition for changes when I launch it; if it parsed the USN journal, it'd probably be less than a minute.

And the API that deals with USN are, per Microsoft tradition, pretty... raw. Haven't looked at how you've wrapped it, but just the fact that you're doing this work is appreciated.

A thing to keep in mind when working with the USN journal is that it's of limited size - so, theoretically, you risk missing updates if there's "a long timespan" beetween launches of your application. I've got no clue how long this takes for a normal-traffic volume, though, but you'll need to fallback to regular file-traversing code if you want to be entirely safe. Also, while MFT scanning is fast & nice, you should have a graceful fallback to regular traversal (to support non-admin mode, as well as handling possible future NTFS versions).

At any rate, here's a Thmbsup from me smiley
Logged

- carpe noctem
Pages: [1]   Go Up
  Reply  |  New Topic  |  Print  
 
Jump to:  
   Forum Home   Thread Marks Chat! Downloads Search Login Register  

DonationCoder.com | About Us
DonationCoder.com Forum | Powered by SMF
[ Page time: 0.042s | Server load: 0.2 ]