Other Software > Developer's Corner
StCroixSkipper's NTFS USN Journal Explorer
StCroixSkipper:
I've long been interested in promoting the use of the NTFS USN Journal in Windows applications. And while .Net doesn't provide access to the USN Journal, I've written the PInvoke code to make it easy to use from C#. I've written a NtfsUsnJournal class in C# that exposes the functionality of the USN Journal but masks most of the complexity required to use it. I've also written a WPF application I call StCroixSkipper's USN Journal Explore that lets you select a volume, create or delete the USN Journal on that volume, save the USN Journal state, query the USN Journal and view the changes (USN Journal entries) to the volume. Since accessing the USN Journal is much like reading directly from the NTFS Master File Table, I also let you enumerate all of the directories on the volume and display them in a ListBox.
I'd like to share the current version with folks to get a discussion going. I'd like to see ideas on how the USN Journal could be used in applications to speed up things like identifying all the files of a given type on a volume or finding all of the identical files on a volume or finding all the files who attributes meet certain criteria.
I'd also like to solicit comments on the code. Out of this process I'd like to bring together a community of folks who have an interest in sharing thier ideas and code relative to the USN Journal.
Remember that to use or access the USN Journal you'll need admin rights. So right click on the application and select the "Run as administrator" menu option. If you run without admin rights, you'll get an 'Access denied' message.
Also, the 'View Changes' button reads the USN Journal entries from the saved state to the current state. So if you have no saved state, you'll get 'No saved state' message.
Here is the code and a copy of the executable attached as an exe file and then the entire project with source code in a zip file. I'll try to download these to make sure I can unzip them.
A little background on the USN Journal:
Jeffrey Cooperstein and Jeffrey Richter have co-authored a great article Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained If you are interested in the USN Journal, this article is well worth your time.
Each entry in the USN Journal is assigned a 64 bit number which is the location with in the actual USN Journal file of the entry. USN Journal entries can be different sizes since at least the filenames are different lengths.
Each file or directory on a volume has a unique 64 bit File Reference Number. Files can have the same names excluding the path but no two files or directories can have the same File Reference Number.
Here is the format of a USN Journal entry.
/// <summary>
/// Contains the USN Record Length(32bits), USN(64bits), File Reference Number(64bits),
/// Parent File Reference Number(64bits), Reason Code(32bits), File Attributes(32bits),
/// File Name Length(32bits), the File Name Offset(32bits) and the File Name.
/// </summary>
public class USN_RECORD
{
private const int FR_OFFSET = 8;
private const int PFR_OFFSET = 16;
private const int USN_OFFSET = 24;
private const int REASON_OFFSET = 40;
public const int FA_OFFSET = 52;
private const int FNL_OFFSET = 56;
private const int FN_OFFSET = 58;
public UInt32 RecordLength;
public UInt64 FileReferenceNumber;
public UInt64 ParentFileReferenceNumber;
public Int64 Usn;
public UInt32 Reason;
public UInt32 FileAttributes;
public Int32 FileNameLength;
public Int32 FileNameOffset;
public string FileName = string.Empty;
/// <summary>
/// USN Record Constructor
/// </summary>
/// <param name="p">Buffer of bytes representing the USN Record</param>
public USN_RECORD(IntPtr p)
{
this.RecordLength = (UInt32)Marshal.ReadInt32(p);
this.FileReferenceNumber = (UInt64)Marshal.ReadInt64(p, FR_OFFSET);
this.ParentFileReferenceNumber = (UInt64)Marshal.ReadInt64(p, PFR_OFFSET);
this.Usn = Marshal.ReadInt64(p, USN_OFFSET);
this.Reason = (UInt32)Marshal.ReadInt32(p, REASON_OFFSET);
this.FileAttributes = (UInt32)Marshal.ReadInt32(p, FA_OFFSET);
this.FileNameLength = Marshal.ReadInt16(p, FNL_OFFSET);
this.FileNameOffset = Marshal.ReadInt16(p, FN_OFFSET);
FileName = Marshal.PtrToStringUni(new IntPtr(p.ToInt32() + this.FileNameOffset), this.FileNameLength / sizeof(char));
}
}
In the USN_RECORD class you can easily see what the entry contains. You read a buffer from the USN Journal and parse thru the buffer to find and marshal the values from the buffer into the fields of the USN Journal entry.
So we now have access to the record length, the file reference number, the parent's file reference number (which is the file reference number of the directory which contains the file), the reason why an entry was generated for this
file or directory, the files attributes, the length of the files name in bytes and the offset into the entry of the file name.
As it so happens, the Master File Table is made up of entries that are of the same format so this one class works for both the USN Journal and enumerating all of the files and directories on a volume using the Master File Table. It takes about one tenth of the time to enumerate all of the files and directories on a volume by reading the Master File Table on an active drive than the traditional FindFirst() FindNext() approach.
StCroixSkipper:
I did forget to mention one item. There is a slight problem with the code.
The NTFS filesystem writes entries to the USN Journal as changes are made to a volume. So, if you modify a file in a directory and then delete the directory you'll have first an entry in the journal about the modification, then an entry about the file delete and then an entry about the directory delete. Since each entry only has the file's 64-bit File Reference Number and the file's parent's 64-bit File Reference Number it is possible to find a journal entry whose parent has been deleted and, hence, does not exist. I have code to fix this problem but for this little application I haven't added it yet.
p.s. I have downloaded the executable and source code and I have been able to unpack it and use it.
StCroixSkipper:
Here is my task list as of now. I would welcome more ideas. Understand that I work on this during my free time. But I will make progress.
1. Add the code to take care of the condition where, when processing a USN Journal entry, I find that the parent folder no longer exists.
2. Add a function/button that, given a search filter, finds all files that match that filter.
3. Add a function/button that identifies all duplicate files on a volume.
StCroixSkipper:
Options StCroixSkipper's USN Journal Explorer v1.2
12 May 2010 3 Comments
Posted by StCroixSkipper
I've added a button, 'List Files' to allow you to list and display all the files on a volume that match a filter string.
I enumerate through the entire Master File Table to find all the files whose extensions match those in the filter. Why would I want to read the Master File Table rather than enumerating through the disk with Directory.FindFiles()? For an active volume it works about 10 faster.
It works much like 'List Folders'. If you click on a specific file, you'll see a popup window that shows the detail. If you double click on the file name or path in the detail window it calls Process.Start() with the file name to start the application associated with the file extension.
I've also changed the 'Update USN Journal State' from a MessageBox to a Window so I could control where it is placed.
In both 'List Folders' and 'List Files' I've sorted the output before displaying it.
The executable and source files are attached.
StCroixSkipper:
Wouldn't you know it! I found a couple of bugs. Here is the corrected code.
Navigation
[0] Message Index
[#] Next page
Go to full version