Starting with NTFS V 3.0 (Also known as NTFS 5.0) Microsoft introduced several new features in the file system to make it more robust, reliable and efficient. This included Support for disk quotas, EFS, Sparse files, reparse points and Update sequence number (USN) journaling. While NTFS v 3.0 supports all these features the actual implementation is more of an OS function. USN journaling was first used with the release of Windows 2000.
Quick googling the term USN Journal shows the following definition
“The USN Journal (Update Sequence Number Journal), or Change Journal, is a feature of NTFS which maintains a record of changes made to the volume. It is not to be confused with the journal used for the NTFS file system journaling”
Whenever a change in the system volume is made, the USN journal records the type of change made to the particular file/ Directory/ Stream on the volume, thus if one creates a file, then edits it and finally deletes it, all these changes are kept as records in USN journal or change journal along with the date and time of such change.
Why do we need to keep these changes?
Imagine a situation wherein an application needs to know the changes made on the volume. This application might be a backup application for differential backup or may be an antivirus that periodically wants to check the changes made on a volume or some default windows service like Windows file replication service or windows search service. One way of doing it is scanning the entire system and then comparing it with the volume state at an earlier time and finally listing out all the changes. Definitely repeatedly scanning the entire file system is not an efficient manner. An alternative to this is using FindFirstChangeNotification function or ReadDirectoryChangesW function in the operating system as was done in the NTFS versions prior to version 3.0. The biggest problem here is that it requires additional applications to keep running and eat up additional resources on the system. Besides this, these functions have a very limited functionality.
The best way then is by keeping a record of all the changes made to the file system on a volume in a common database which can be consulted by the relevant applications. This gave birth to the USN journal. USN journal keeps the record of minutest of changes on the file system and can be accessed by multiple application through well documented API’s simultaneously.
The physical layout on the NTFS volume:
One can find the USN journal database in the file $Usnjrnl located in the folder $Extend on every volume. Thus, there is one USN journal database for every volume. This folder being a protected file system folder is not accessible with windows explorer or command prompt. If one examines it carefully one can find that the size of this file is 0 byte. This is because by default the size of default $DATA stream is shown which is Zero for the $Usnjrnl file.
The data is stored in an alternate data stream named $J and the related metadata is stored in the alternate data stream $Max.
How does it work?
Initially, the $J stream is empty. When there is a change in the system volume a record is added to the file. This record includes an Update sequence number (USN) which is a 64-bit identifier, time of change and the type of change. Besides this, the file reference pointer to the related entry in $MFT is also recorded along with the MFT sequence number to point to the correct file. The USN number is an incremental number which in fact is the offset in the file where the record starts. As the record length is variable (Due to variable size of the file name) the USN number is not contiguous. Thus, one may find USN number 1 followed by next USN number 136 and then the USN number 192. Every USN record occupies the space in multiple of 8 bytes. To ensure this a padding is added at the end of the record if required. To improve the efficiency, the operating system writes these records in blocks of 4KB which is referred as USN_PAGE_SIZE. The system will not allow the record to spill over this page boundary and fill the leftover space with zeros, and therefore, one may see gaps in USN records. To understand it completely let’s dissect the files manually.
$Max (The USN journal metadata store):
This data stream stores the metadata related to the USN journal. This file carries four data fields:
- Byte 0x00 – 0x07 (Maximum size of the USN journal): This is the maximum size of the USN journal. Whenever USN journal reaches this size, it removes a chunk of the records from the beginning of the $J data stream and adds that much space at the end of $J data stream. The size of this chunk is called Allocation delta which is mentioned in the next field. The size of this file can be increased with FSUTIL.EXE by the system administrator at the time of creation of USN Journal. Ideally, where the number of files is large or there are bound to be regular transactions on the system e.g. an Application server or a File server the size should ideally be increased for forensics preparedness. Besides this where it is not practically possible to bring down the system after incidence the size of USN Journal should ideally be increased.
- Byte 0x08-0x0F (Allocation Delta): This is the size of the chunk that is to be removed from the beginning of $J data stream and added as free space at the end of it. This allocation delta can be specified with fsutil.exe while creating the USN journal.
- Byte 0x10 – 0x17 (USN ID): This is the ID of the USN Journal created. Interestingly it’s just the FILETIME of the creation of the USN Journal. This is an important forensics evidence in case an old USN Journal is deleted and a new USN Journal is created.
- Byte 0x18-0x1F (Lowest valid USN record): This is normally the first USN Record.
A Typical structure of the $Max data stream can be seen in the screenshot below wherein the Maximum size is 32 MB, Allocation delta is 8 MB, Lowest valid USN is 0x00 and USN ID specified time is 29th January 2015 at 1:20:56 AM.
$J (The USN database):
$J data stream carries the actual records of the change. Normally when you open the $J stream in a hex editor you will see a bunch of zeros filling the initial file sectors. The explanation of this goes as follows:
- As the $J data stream reaches its maximum size, the chunk of data equal to allocation delta is to be deleted from the start and equal chunk of space is to added at the bottom of the file for new records.
- USN numbers are always incremental
- USN numbers are always equal to the record offset in the file. This means that the file offsets have to keep increasing too.
- If the maximum size of the file is fixed, how would offset keep on ever increasing?
To handle this the USN Journal is made a sparse file which has a simple method of compression. All the chunks of zeros are not written on the drive and all non-zero chunks are written on the drive. Finally a list of these zero filled chunks is added as metadata to the file. When the stream is opened in a hex editor it’s the uncompressed data that is shown and thus, one sees a lot of zero values at the beginning of the file which does not occupy space on hard drive and is kept as an information in file metadata.
The records in $J has three versions namely, USN_RECORD_V2 (Supported by windows XP and server 2003 or above), USN_RECORD_V3 (Supported by Windows 8 and Server 2012 and above) and USN_RECORD_V4 (Supported by Windows 8.1 and Server 2012 R2 and above) till date. There are minor changes in the version 2 and version 3 whereas the version 4 is still not completely documented by Microsoft. Various experiments have shown that Version 4 is more of an addition to version 2 or 3 wherein it carries additional information in the form of a member called USN_RECORD_EXTENT. There can be one or more USN_RECORD_V4 for a single USN entry and it tracks the changes made to various ranges in the file, in case such tracking has been enabled on the volume.
USN_RECORD_V2 data structure:
The figure below shows the data structure of the USN_RECORD_V2. In general, it is observed that even the latest OS by default uses this data structure, though they support Version 4 as well.
- Byte 0x00-0x03 (4 bytes): This is the size of the entry. In our demo case (In image above) the value is 0x58 the entry starts at the offset 0x6a9b80000 and ends at 0x6a9b8058. This is then immediately followed by the next USN record entry.
- Byte 0x04-0x05 (2 bytes) Major record version: This is the version of USN_RECORD and in our demo case it is V2.
- Byte 0x06-0x07 (2 bytes) Minor record version: It is zero in our case.
- Byte 0x08-0x0D (6 bytes) MFT Entry number: This is the entry number in $MFT. As $MFT entry is frequently reused on file deletions it may be quite confusing. There might be several files pointing to the same entry number, one of which might be on the drive and rest all moved or deleted. As the MFT entry is reused there is another value related to this entry called Sequence number which is incremented by one. Thus, all such multiple files will have different sequence numbers. In our case, it is 0x23 which is entry number 35 in $MFT
- Byte 0x0E-0x0F (2 bytes) MFT entry sequence number: This is the sequence number of the MFT entry. In our case, it is 0x01CE which is 462 in decimal.
- Byte 0x10-0x17 (8 bytes) Parent MFT entry with Sequence number: Just like above two fields first 6 bytes indicates the parent folder MFT entry and 2 bytes indicates its sequence number. This is important as one may find a USN record of a deleted file, once deleted it will be not in $MFT and finding its actual path will be a problem in the absence of parent folder entry.
- Byte 0x18-0x1F (8 Bytes) The USN number and Record offset: This is the USN number as well as the offset to the beginning of this record. In our example it is 0x06A9B80000. This is the USN number and it can be seen that the record starts at exactly the same file offset.
- Byte 0x20-0x27 (8 Bytes) Time-stamp: This stores the timestamp in FILETIME format indicating the exact time when the change was made. In our example, the value is 0x01d1c611119f9943 which when converted to time comes to June 14, 2016, 7:47:58am. This time-value is for UTC and suitable adjustments should be made for related time zone.
- Byte 0x28-0x2B (4 Bytes) Reason flag: This is the reason why this entry was create. This indicates what all changes were made to the file or directory. Once the file or directory is opened there can be multiple changes. USN journal writes these changes in accumulation. The first change will result in first USN record with one change, the second change will result in Second USN record that carries both first and second change, and so on until the final USN record is generated on file close. Next time when the same file is changed, a new USN record is created. Thus, if one Opens a file, Truncates some data, then overwrites some other data, Adds some more data and finally closes it, the following USN entries will be created.
- USN record 1: Reason- Data truncated
- USN record 2: Reason- Data truncated, Data overwritten
- USN record 3: Reason- Data truncated, Data overwritten, Data extended
- USN record 4: Reason- Data truncated, Data overwritten, Data extended, File closed
In our case, the reason value is 0x02 which is for Data extended. The very next entry has the reason as 0x80000002 which is a combination of 0x80000000 (File closed) and 0x02 (Data extended).
Once the file or directory is closed and then reopened the process starts afresh.
This can come in handy as one can differentiate minutely. The USN pattern for a file opened, truncated, closed, reopened, extended and finally closed will be different from the scenario where the file is opened, truncated, extended and finally closed. Despite the final result being same the USN pattern will differ. These patterns can then be used to create signatures for various activities.
The detailed list of Reason flag and its interpretation can be obtained from Microsoft documentation.
- Byte 0x2C-0x2F (4 bytes) Source flag: This flag is used to indicate the source of change to the file. The change might be induced by the operating system or some replication service to match the content of the file with another member in the replication set or cloud. In our case its 0x00 indicating that the change was not made by OS or any file replication. The detailed list of Flags and their meaning can be obtained from Microsoft documentation.
- Byte 0x30-0x33 (4 bytes) Security Identifier: Provides the details of SID related to the file or directory. In our case, this is 0x00 which is null indicating the null authority.
- Byte 0x34-0x37 (4 bytes) File attributes: These bytes provide the information about the attributes of the file or directory. The values can be checked in Table 3. In our case, it is 0x00 which again is indicative of deleted file.
- Byte 0x38-0x39 (2 bytes) Size of the file name: This value indicates the size of the file name in bytes. In our example it is 0x18 and the name of the file is “accasrvc.log” which is 12 bytes long. As the filename is mentioned in Unicode it occupies twice the space thus it is 24 bytes or 0x18 bytes long.
- Byte 0x3A-0x3B (2 bytes) offset to the file name from the start of record: In our case this value is 0x3C. As our record starts at 0x06A9B80000 the file name is located at 0x06A9B8003C.
USN Journal Forensics interpretations:
Pieces of evidence are not analyzed in isolation and the true value of USN journal analysis lies in using it with other pieces of evidence and creating a timeline for analysis which gives the bigger picture and holistic view of the events on the system. These pieces of evidence can come from Logs, Registry, Recycle bin, Prefetch files, Browser histories, and cookies etc.
To get more out of USN Journal analysis one may create signatures of events based on USN journal entries and its sequence. In fact, various packages when performing some task have a peculiar way and sequence of changes they made on the file system. These can be used as signatures to link it with certain events. Let’s get a bit deeper in it.
Using USN journal as activity signature:
From software execution to file download, working with various agents like Dropbox, Snapfish etc. and even file wiping activities, everything effects the file system in a predictable manner. Based on this we can create signatures and get an insight of what exactly has happened on the system. Let’s try out with some of the signatures.
- The file downloaded with google chrome: Google chrome has a peculiar way of downloading a file. There is a well-defined sequence of changes in the file system that happens with the download. Let’s just take an example of USN Journal entries created when a file is downloaded with google chrome. In our case, we downloaded a file named “LogFileParser-master.zip” from the internet. The USN entries related are shown in the following screenshot.
One can see the following changes on the volume related to the downloaded file:
- A file with the name 82ED.tmp was created.
- The file was then closed (Nothing added).
- This file was renamed to a new file.
- The name of the new file is “Unconfirmed 978550.crdownload” (This started a new USN record cycle).
- The file “Unconfirmed 978550.crdownload” closed (The file rename activity finished).
- Data is added/ extended in the file “Unconfirmed 978550.crdownload” (This is where the file is filled with downloaded data).
- The File is closed completing the cycle of download.
- The File initiated a rename.
- The New Name is “LogFileParser-master.zip”.
- The New File “LogFileParser-master.zip” closed.
- The Named data stream changed (Either added, deleted or renamed). In this case, its new stream added in all possibilities.
- The Named Data stream is extended (Data added to it).
- The File “LogFileParser-master.zip” closed.
While all these changes were being made one can observe that the $MFT entry is same throughout (277110-11) where 277110 is MFT entry number and 11 is the sequence number. This means that all the files viz. “82.ED.tmp”, “Unconfirmed 978550.crdownload” and “LogFileParser-master.zip” are one and the same file.
One can easily corroborate this with other pieces of evidence like Chrome history, Zone.identifier data stream of the downloaded file and other filetimes.
- Wiping of a file with file wiper: File wipers wipe off the files completely. Almost all wipers do all or some of the following activities in different orders.
- Overwrite the data (Actual wipe).
- Changes the timestamps (So one cannot get to know when it was accessed or deleted).
- Rename the file (So that it cannot be searched on the drive with the filename).
- Delete the file (Remove the entry from $MFT).
A file wiped off can be identified with the USN record sequences. Let’s take an example of a file named “sg248092.pdf” wiped off with the tool called “File shredder”. The results of USN Journal parser are shown in the screen shot.
The activity can be listed as:
- The data was overwritten (probably zero filled or random filled)
- The data was truncated (Deleted the overwritten data)
- Closed the file
- Renamed the file (To avoid finding with data recovery methods)
- The New name is “16309755.zzz”
- The file was closed
- The file was deleted
Similarly, one can identify the attachments opened from the emails in outlook, Files sent to recycle bin or deleted from recycle bin and so on.
Volume shadow copy and USN Journal:
Every volume shadow copy has its own USN Journal. Whenever a snapshot is created it includes the USN Journal as well. This can come in handy as it keeps safe even very old changes which the default Volume USN Journal may not have or which may have been overwritten. To extract the USN Journal from the volume shadow one has to mount that shadow copy as a drive or mount point and then extract the USN journal and $J data stream.
Tools of the trade:
There are several parsers available either open source or Free for carving out USN journal data. Almost all of them requires three files. $J, $MFT and $Logfile. Though one can carve out just with $J, there would be a couple of information missing. The few tools of my choice are
- NTFS Log Tracker: Can be downloaded from https://sites.google.com/site/forensicnote/ntfs-log-tracker.
- Triforce ANJP free Edition: Can be downloaded from Can be downloaded from https://sites.google.com/site/forensicnote/ntfs-log-tracker.
- USN-Journal-Parser: A python script to carve out USN journal, can be downloaded from https://github.com/PoorBillionaire/USN-Journal-Parser.
Besides this, there are a number of commercial tools available, The detailing of which is beyond the scope of this writeup.
Why do we need to know all those structures when we have tools to parse USN Journals?
As the saying goes, “As a forensic investigator you cannot justify your findings just because it resulted from running a tool”. You ought to know what’s happening behind the running tool and interpret the output of the tool.
The other reason is the slack space. Very often one finds slack space related to a file or a data stream. $J is no exception. As shown in the screenshot below
There is $J data stream as well as $Max data stream but then there is some slack space from $J too. A tool may or may not be able to carve out this though it may have important pieces of evidence. When the forensic investigator understands all these data structure and the way system functions he may be able to carve out the information manually with a hex editor. In example above one can see that the first record in slack space starts with filename. As $J will not have such a thing immediately after chunk of zeros, probably no tool will be able to carve out the entries, whereas manual inspection shows a value that seems to be timestamp pertaining to the file name following it (Marked in Red boxes).
In the course of various investigations, I have come across some smart ways of evading the USN Journals. There are tools that can clean up the USN Journal like “PrivaZer”. The other way is to delete the Journal altogether with the “FSUTIL.EXE” and create a new one. This requires administrative privileges. Fortunately, if the slack space is not wiped one can find the USN slack data. One interesting way I came across recently was rapid file changes induced with various installations and uninstallations spilling the maximum file size of USN journal and thus, getting older records removed.
USN journal can come in handy when it comes to forensic analysis. The only practical problem is handling the evidence. Very often the size of USN Journal is kept at default which ranges from 1 MB to 32 MB. This is a very small value considering the kind of activity that goes on inside a system. Lot many updates, automated tasks, user work, and other activities keep USN journal change very fast. If one creates a new USN journal well in time with the size of 1-5 GB or so, it can help in the hour of need.
Again all the time stamps extracted should be part of a bigger picture coupled with timelines extracted from other artifacts.
- Keeping an Eye on Your NTFS Drives: the Windows 2000 Change Journal Explained by Jeffrey Cooperstein and Jeffrey Richter
- Various documentation from Microsoft/msdn
- SANS Webcast at https://www.youtube.com/watch?v=zKZlXhU2MJQ