What Is an Alternate Data Stream and Why It Is Important

Alternate data streams (ADS) are a feature of the NTFS framework that allows a single file to contain more than one stream of data. ADS are important for many industries, including software developers, digital forensics, and those concerned with data security.

While an alternate data stream is critical to many aspects of software development and function, it can also be applicable to web scraping. Web scraping fuels alternative data, including providing insight into market sentiment, alpha generation by staying ahead of traditional market data, and trend spotting. To understand how it can play a role in what you are doing, consider how alternate data streams work and why they matter.

What Are Alternate Data Streams?

File systems are a core component of how a computer operating system manages and stores data. The New Technology File System (NTFS) framework is a very robust and high-performance file system that Microsoft initially developed years ago. Windows Operating System uses NTFS. The value is that NTFS supports huge volumes and file sizes. In addition to this, it can also provide security features like file encryption and permissions. At its core, it supports advanced data structures that enhance performance and reliability.

NTFS provides the ability to use ADS. Specifically, ADS enables a single file to have more than one stream of data. Each of these streams of data can then store different types of information. This is important because, in traditional field views, these are not visible.

The alternate data stream history is impressive. Many factors play a role in the value of ADS. A good way to observe this is to take a look at its history. The concept of ADS is not new. It can be linked to Apple’s Hierarchical File System, more commonly known as HFS, which was first put into place in 1985. That system was specifically designed for the Macintosh Operating System. Macintosh was a much more complex system in that it required a complex system to store files in a way that would allow both data and resource forks.

The term “data fork” refers to data that contains primary content. The term “resource fork” refers to additional metadata. This may have included things like menu resources or icons. It would also include specific information for applications. By establishing the HFS, it was possible to create a dual fork system that would allow all Macintosh applications to manage these files with more flexibility even as complex as they were. The specific benefit it offered was a way to protect both the primary data and the associated metadata in a seamless manner.

This initial creation was just the start. Other organizations and file systems were developed that could handle multiple data streams in similar ways. Over time, Microsoft developed HTFS. At that time, it introduced the first ADS to ensure compatibility with HFS and support advanced data management features.

What did this do? Ultimately, NTFS’s ADS enabled a single file to contain more than one stream of data. This made it more versatile for applications that were becoming more complex. It also allowed for data storage solutions that could handle a wide range of applications. One of the most important ways that this application was beneficial was in its ability to preserve metadata. It also offered benefits by allowing cross-platform compatibility and supporting better application functionality overall. Over time, it helped to support the trend in file system design that supported rich and multifaceted data structures.

Other Alternative Data Streams in File Systems

As you explore what alternative data streams are, consider that there are a number of other systems and storage technologies that are somewhat similar and have somewhat similar capabilities to handle multiple data streams. Others have extended attributes.

You have likely heard of or even used a number of them. Some examples of these file systems include the following:

• Hierarchical File System Plus: Often referred to as HFS+, it is used by older versions of macOS. This file system supports resource forks, which are somewhat like the forks we have described. A resource fork is different in that it will allow additional metadata and attributes to be stored right with the main data fork of a file.
• Apple File System: APFS is a newer version for macOS and iOS. It allows for extended attributes, which are also somewhat like ADS in terms of how they function.
• Resilient File System: ReFS is a newer system created by Microsoft. It supports extended attributes like others do. However, it does not have the same extensive use of NTFS when it comes to ADS. Additionally, this file system is more focused on data integrity and reducing data corruption. It is also a more viable option for scaling as well.
• B-tree File System: Btrfs is another example, but this one is for the Linux file system. It provides much the same functionality that you would expect from ADS and supports extended attributes as well. To function as ADS, it allows attachments of additional metadata to be added to files.
• Zettabyte File System: ZFS is used in various types of operating systems. Some examples include Solaris and some Linux distributions. It provides a robust framework for storage as well as for data management. It is beneficial for extended attributes as well.
• Extended File Systems: In all versions, Ext2, Ext3, and Ext4, you have a Linux operating system. With xattr, it supports extended attributes. This enables users to store added metadata with files. You can use those attributes for a variety of tasks, including user metadata, system information, and security labels.

There are quite a few options out there. Most of these are similar in terms of the features they offer. However, how you implement them and when you can use them is very different. The cases of multiple data stream usage must be considered carefully. More so, extended attributes will vary. Ultimately, the goal is to manage and store data and meet security requirements, no matter which platform you are using.

Utilizing ADS in NTFS

In this guide, our focus will be on ADS in NTFS, as that is the most common application for web scraping and similar tasks. Let’s break down how ADS works within NTFS to get started.

Within NTFS, every file has a primary data stream. Every file also has several alternate streams. The primary stream will maintain the main content of the field. The alternate streams, then, hold the additional data necessary. These secondary streams are not visible within the standard file listing. To access them, you must use APIs or other tools. Accessing an ADS will require knowing the necessary syntax. This generally involves appending a colon and the stream name to the file path. For example, you may use file.txt.stream to achieve your goal.

The key is that this feature is embedded into NTFS. As a result of that, it enables diverse applications to occur. At the same time, it complicates data management and security tasks.

There are various ways and reasons to use ADS within software as well as in processes for systems. These are legitimate and necessary ways to use it. That includes:

• Metadata storage: A key reason why you may wish to use ADS is because you can store metadata within it. That includes titles, text descriptions, author documentation, or other data. In doing so, you will not alter the main file content.
• Functionality improvement: ADS also offers some key benefits to improving functionality. For example, some applications use it to store configuration data. It may be used to store thumbnails or other information that’s valuable to the specific system data.
• Improves system processes: Operating systems like Windows benefit from ADS as a way to store system-level information. That includes security descriptors as well as indexing attributes. It can also improve how efficiently the system operates overall.

With so many important benefits, it may seem like ADS is an easy way to provide information without interfering with the main content. However, just as there are very good reasons why you should use it, there are also some risks that all users must take into consideration.

For example, in some situations, it can be used to hide data. That includes hiding malware that causes damage to another system. Remember, the information here is not visible within an alternate data stream. That means the malware can hide in alternate data streams because it is not visible in the standard listings.

This creates a risk of malicious activity that can lead to exploiting data or information. It is possible that it can lead to harmful code being placed within the ADS, which can then make it hard to detect. It also does not alter the primary file size or the appearance of that file – which makes it very easy to hide within the ADS file.

It may be possible to find a wide range of malicious data within these files. For example, trojan horse programs may be lurking within the ADS. This would allow that file to get past most traditional antivirus scans, and that means it could become difficult for another person to detect.

In addition, data exfiltration can also occur. Those with nefarious reasons could use alternate data streams to store and then transfer sensitive information. This could be done without any indication or detection. It can also incorporate persistent mechanisms, which would remain hidden and operational. That means that, even after a user engages in a system reboot or security scan, it can lead to a continued presence.

As a result, it can be hard to detect whether ADS exists. The whole point is for it to be hidden. If you are using only traditional file management tools, they will not display ADS in many cases. For that reason, you need to use a specialized tool to determine if ADS is present.

There are numerous forensic tools available that may be helpful, including:

• Streams by Sysinternals: This is a free tool that is designed specifically for listing ADS for files and directories on the NTFS file system.
• Autopsy: This is an open-source digital forensic platform that can provide support for detecting ADS being present and aid in analyzing ADS. It can also provide other digital forensic tools.
• Forensic Toolkit: Provided by AccessData, this is an important forensic tool that can detect and analyze the presence of ADS. This is done as a part of the extensive file system to determine its capabilities.

Numerous other factors and various tools must also be considered. The key for most organizations to remember is that ADS creates risk, and that risk can be exploited in many ways. However, there are strategies to mitigate that risk when using these tools. The more you know about them – and the more you actually teach your team and employees – the less risk there is to your system.

Common Questions You May Have About ADS

There are many questions to consider today about ADS, including when it is present and what to do about it. Let’s consider some of the most commonly asked questions related to this topic.

What is an ADS file? This is not an accurate question. Rather, alternate data streams are file attributes. They are found on the NTFS file system. ADS allows files to contain more than one actual stream of data. While every file has at least one data stream, those with ADS contain more.

How do you convert an ADS file? This question does not pertain to ADS in the way we are referring to it. However, you can convert the ads file format to an app file format if you need to. One of the ways to do that is to load the ADS file and then create a temporary EPMA application. This would allow you to use your computer client to extract the metadata located within the XML.

What are ADS in NTFS? ADS is a stream of data that is written into a file. This is found within the NTFS file system. It allows for sending more information and details about a file but does so without altering the main file. It allows users to send more information about a file than just attributes and properties.

How do you open an ADS file? There are two steps for opening an ADS file. The first one is to create an isolated ADS file. To do that, use the command “echo content > :ads filename”. To open an ADS file, use the command “notepad :ads filename”. You will need to update those codes to reflect the information associated with your file.

How do you detect and then remove ADS files? You may not want anything unknown coming in from the files you select. To do this, you first have to detect the ADS files. Use the dir/r command to do that. This will only detect ADS files that are under the current folder you are in. If your goal is to find ADS files under a subfolder, you will need to open it first using dir ddd, for example. Then, use the command dir ddd/r to get to see the ADS file. You can also use the lads.exe tool to detect these files.

Once you find the files, you can get rid of them. To do that, you will need to follow one of three steps:

• You can delete the entire host file directly. That gets rid of the ADS.
• You can also move the host file to a non-NTFS partition, which would mean the ADS file would not accompany it. Options for this include FAT and FAT32, for example.
• The third option is to use a tool like Streams.exe, which is provided by Microsoft and can be used to delete streams.

Key Best Practices for Managing ADS in Security Audits

Because an alternate data stream can create security risks for companies, it is essential to know when they are present. There are various tasks for this, but putting in place some basic steps and guidelines for your employees and others can help to facilitate a safer experience overall.

Scan for ADS: One of the first steps is to make sure you have talked to and educated your team about how to scan for ADS. Your IT team should be doing this on a routine basis. Use one of the tools provided, such as Sysinternals’ Streams, as noted, or PowerShell scripts. These can work to routinely check all of your file systems and detect any ADS that is present. Doing this on a regular basis (as frequently as needed based on the number of downloads you have) will help uncover any hidden streams you did not know were there. You can then determine if they are a risk.

Restrict non-essential use: Though ADS are very commonly present and not easy to detect, consider some guidelines that limit the use of ADS specifically. Even for legitimate reasons within your organization, they present a trust risk, and simply avoiding their use in any application minimizes the risk that these streams will be used to exploit activities, including harmful activities.

Talk about the risks with all levels of your company: This is not an IT situation, but something that should be taught to all levels of your company because of the risks it can present. Most people who do not work in the IT department may not know about this risk as it is not commonly discussed when it comes to malware and viruses.

Know the risks in all industries: Not just one industry can fall victim here, but numerous. ADS detection and management are critical in all applications. That includes the financial sector, the healthcare industry, and the corporate environment. There are numerous indications of such risks playing out across the industry, including malware authors using it to hide malicious payloads that add sensitive data or within the healthcare center to put patient information at risk.

Why Alternate Data Stream is Important

Alternate data streams are critical to understand and utilize as needed. They allow data to be associated with and hidden within a file, which means they can provide important and quite valuable information about the file. This includes comments or metadata that can be useful for a variety of situations.

While ADS has good uses, it is also important for organizations to recognize the risks it can present. They can allow attackers to store and hide information within files that can then present a risk to the entire system. For this reason, even if ADS is used within your organization for positive reasons, you also need to be sure there are security measures in place to minimize the risks of passing those ADS into high-risk areas.

Let Rayobyte Help You Protect Your Sensitive Data

Alternate data streaming is an important concept to understand. Whether you are using it for any reason, you need to take action to protect your business.

Proxies, like those provided by Rayobyte, are another layer of protection. When you utilize our residential or data center proxies, you can import layers of protection that help to minimize the risks of too sensitive information being passed on to other sites.

To learn more about what we can do to help you with our tools, contact our team now. Let us offer insight into how you can secure all of your sensitive information using proxies. We can also offer guidance with web scraping thanks to our partnership with Rayobyte’s Web Scraping API. Contact us to learn more.

The information contained within this article, including information posted by official staff, guest-submitted material, message board postings, or other third-party material is presented solely for the purposes of education and furtherance of the knowledge of the reader. All trademarks used in this publication are hereby acknowledged as the property of their respective owners.