![\"A](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/bb-bh-Your-Files-Hidden-DNA-and-You-2.jpg\")
<\/p>\n<\/p>\n
The files you use every day on your Mac or PC, whether at home or at work, carry around a slew of hidden data that can be incredibly useful to you… or problematically revealing to others. For example, the image in the header reveals latitude and longitude details in an iPhone photo that you could use to organize the photo and find others taken in the same place. But anyone else can access the same data and enter it directly into Google Maps to discover exactly where that picture was taken! Not quite as useful.<\/p>\n
But if you know what this hidden information is\u2014and how to use it\u2014it can be incredibly helpful in diagnosing problems with files, organizing or protecting data, and even removing information you don\u2019t want revealed! If you don\u2019t, it can be a huge annoyance, and potentially even dangerous.<\/p>\n
\u201cIt\u201d is \u201cmetadata\u201d and it\u2019s something everyone works with, even if they don\u2019t know it. Whenever you move a file\u2014through email, into or out of a sync or cloud storage service, or to another device\u2014you\u2019re likely altering its metadata. It\u2019s something we work with at Backblaze every day. And because moving files into and out of computer backup<\/a> and cloud storage services can affect metadata, we thought we’d take a high-level look at how this information works in common file types to help you understand how to optimize its use in your own file management.<\/p>\n You can follow along as we walk through several examples, then tackle some real world file mysteries with the power of metadata. At the end of the post, you will find a list of several tools for Macs, PC\u2019s, and command line to test out and add to your own \u2018metadata toolbox.\u2019<\/p>\n A great way to think of file metadata is as extra information about<\/em> a file, carried along with<\/em> that file, that makes it easier to use and find. So it\u2019s not the actual document or photo itself, it\u2019s information about it<\/em>\u2014like the file\u2019s name, thumbnail image, or creation date. This information is embedded in or associated with the file, and helps make it easier for you, your applications, and your computer to actually use<\/em> those files.<\/p>\n The most obvious kind of metadata is a file\u2019s name, extension, icon, and the timestamp of the its creation date. This simple metadata alone makes searching across an entire hard drive<\/a> of files and folders as easy as typing a part of the name into the finder or search bar, sorting the results by date, then singling out the file you want by the proper thumbnail or filename.<\/p>\n A less well-known example of file metadata is meant to make working with files easier or safer for your operating system. Your files might carry notes for the operating system that they should be opened with a specific application. Or a flag might be set on a file you’ve downloaded from the internet or mail attachment warning your OS that it may not be safe to use.<\/p>\n Other critical information about a file is the permissions<\/em>, or privilege levels, extended to users on that computer:<\/p>\n For example, files on UNIX-like systems, like Linux and macOS X, are marked with the name of the user account that created them (the \u2018owner\u2019), the computer account group they belong to, and the permissions for the owner and other users to open and view that file, or make changes to it.<\/p>\n When permissions on files are set correctly, you rarely need to think about them as a user. But if this permissions information changes, users could lose access to files, or files could be opened by users that shouldn\u2019t have access.<\/p>\n Another category of information is human-readable, but really intended for your applications to use. Some of this information can be incredibly detailed. The best-known example of ‘application metadata’ is camera and location data embedded in images by the cameras when you take pictures, such as the camera information and the camera\u2019s lens and shutter setting when the particular picture was taken.<\/p>\n All this information is read by your image editing software to enable new features. For example, in iPhoto you can search for all images taken in the same location, or find all images shot with the same camera. That means that these files are a trove of interesting information such as the camera type, shutter speed, and even GPS coordinates where the picture was taken.<\/p>\n You may already know that you do not want to broadcast the location of photos you share, but even plain old documents can have information embedded in them that you’d rather keep to yourself.<\/p>\n In the image above, you’ll see the file metadata of an old word processing document that happily includes names and email addresses for anyone to see! It\u2019s common for files to include information like usernames, email addresses, GPS coordinates, or server mount paths. This is the kind of information you might want to delete before making a file public.<\/p>\n As your files move around\u2014copied from user to user and system to system\u2014all of this useful metadata is vulnerable to being changed or lost. This has implications for your workflow, especially when you inevitably need to reconcile different versions and copies of files.<\/p>\n Unfortunately, the operating-system-specific tags or comments you place on files are the first to be lost when they move from location to location, and system to system.<\/p>\n For example, if I carefully color tag a folder of images on my Mac, then send them to be reviewed by a colleague who works on a PC, all those tags are gone when I get the files back. For this reason, true workflow-specific tags are usually applied in an external system that is dedicated to managing this kind of metadata for files\u2014like a photo manager or a digital asset manager.<\/p>\n It\u2019s also common for files received on one OS to come over with non-standard permissions set. For whatever reason, documents saved on a PC end up having the executable bit set when they are moved to a Mac. The files will still open, but there\u2019s no reason for them to be marked like an application.<\/p>\n When you create or change a file on your computer, the time is recorded as part of the file\u2019s metadata. But what happens when the time on one computer differs from another? Most modern OS’s do a good job of syncing to special time servers, and compensating for universal time based on location, but there are still changes introduced that make sorting files by time a challenge.<\/p>\n When files are copied to network servers, or the cloud, things can get completely changed. Depending on how the file is moved, and how the storage provider handles files, your modification dates could get completely blown away, and since the \u2018old\u2019 file you\u2019re uploading is new to the storage system, it becomes a new file with an entirely new creation date.<\/p>\n Individually, these changes are annoying, but collectively they threaten to kill with a thousand cuts. As time stamps, tags, and permissions are changed, your carefully organized file hierarchy or valuable archival information could be in tatters.<\/p>\n To see how metadata changes, let’s follow a single file downloaded to a Mac, then a PC, then upload and download them to different cloud storage options to see what changes get introduced.<\/p>\n In this test I downloaded a PDF from Backblaze’s website to a Mac. On the Mac, I added color tags, and even comments using the Finder\u2019s preview pane. Next, I downloaded that same file on a Windows system, then copied it over to the Mac.<\/p>\n Despite appearing to be the exact same PDF file, let\u2019s fire up a terminal window on the Mac to inspect them further and make sure.<\/p>\n To follow along, navigate to the folder of files you want to inspect so that it\u2019s handy. Then open another finder window and double click on the \u2018Terminal\u2019 application, which is found in the Utilities folder inside of your Applications folder. The terminal application will launch, and you\u2019re placed at the \u2018prompt\u2019 ready for your command.<\/p>\n To navigate to the folder you want to work with, type in \u2018cd\u2019 at the terminal prompt to change directory, enter a space, then drag the folder of files you want to work with into the terminal window and drop it. You\u2019ll see that the path to the folder is automatically resolved to that folder\u2019s location, saving you a lot of typing.<\/p>\n Now that I\u2019m in the proper folder, the tool I want to use is the humble \u2018ls\u2019 command to list a folder\u2019s files. To do so, type in \u201cls\u201d and then a space, then a dash, immediately followed by \u201cl@\u201d\u2014this will retrieve the long form of results, and the \u2018@\u2019 flag will explicitly show extended metadata on the Mac.<\/p>\n As you can already see, the following changes have been introduced:<\/p>\n Now, I\u2019ll move these files to and from three different types of cloud storage\u2014Backblaze B2 Cloud Storage<\/a>, Google Drive, and Dropbox\u2014and see how they change.<\/p>\n To move the files to Backblaze B2, I used rclone<\/a>, which is an extremely popular tool to copy and sync files from any mix of storage and cloud systems. For Google Drive, I used their web interface, and for Dropbox I uploaded via the web, then retrieved the files as a compressed file.<\/p>\n Now, when I compare all the files side by side I can see how different all of the file metadata is.<\/p>\n First, all of my user-entered metadata, like tags and comments, were not picked up by cloud storage, as expected. Secondly, the Mac\u2019s Gatekeeper security feature also promptly labeled every file downloaded with the \u2018Quarantine\u2019 flag. Backblaze B2 returned files with proper file permissions, (644 or read\/write for the user, read for the group, and read for all others) and preserves the creation date of the original file.<\/p>\n Both GDrive and Dropbox applied new file creation and file modification timestamps\u2014and bizarrely, the files returned by Dropbox have a “modified date” 8 hours in the future! Does Dropbox know something we don’t?<\/p>\n You can see how searching and sifting through all of these copies on my Mac has become tremendously complicated now.<\/p>\n Hopefully it’s clear that unless your files only live on your local system, as they move from system to system, the metadata they carry around will change.<\/p>\n Let\u2019s apply what we’ve learned in some common examples of how metadata is changed in files, how to inspect them, and some suggestions to correct them.<\/p>\n Inspecting a file\u2019s metadata information can be helpful in diagnosing misnamed files, or files that have lost their file extension. The operating system usually blindly trusts the file extension. For example any file named with a .pdf extension will try to open it as a PDF file even if it\u2019s really something else!<\/p>\n Above, I have a file from a very old backup that is missing an extension. The Mac is having trouble interpreting the way the original Windows OS file system encoded the date, so my Mac thinks the file was created December 31, 1969! (I\u2019m pretty sure I wasn\u2019t using MS Office in 1969.)<\/p>\n Without an extension, my Mac assumes this file must be a text file, and offers to open it in TextEdit, the default app for opening text files. When I double click on the file, the OS tries to open it but throws an error.<\/p>\n Reaching into the toolbox, I use a command-line program called exiftool, a powerful tool to reveal a file\u2019s embedded file metadata. (Navigate to the bottom of the post to read more about exiftool and where you can learn more about how to use it). By calling the exiftool from the terminal application, and passing in the name of the file I want to inspect, all is revealed! This is, in fact, a Microsoft Word file.<\/p>\n Looking closer, I can even see that this isn\u2019t the original file, it was autosaved from the original file, which has an entirely different name. Mystery solved! I can now safely add the ‘.doc’ extension to the file, and it will open properly with my word processor that can still import this version of Microsoft Word.<\/p>\n Next, let\u2019s take this entire folder of PDF copies that I used for upload tests. After all that uploading and downloading<\/a>, my single original file has 8 copies. I \u2018know\u2019 that I only need one of these, so let’s try de-duping them!<\/p>\n When I try to dedupe this folder using a tool like Gemini, a duplicate file finding tool, I\u2019m presented with several choices of duplicates for me to remove. In other words, Gemini 2 was able to determine that there are duplicates, but isn\u2019t sure which set of files it should keep.<\/p>\n If I select by \u2018oldest\u2019 duplicates, it leaves me with the Dropbox versions, by \u2018newest\u2019 it leaves me with the GDrive versions, etc. In this particular case, the \u2018automatic\u2019 selection tool lets me mark the GDrive and Dropbox versions as the duplicates I will delete. However, the differences in file permissions and extended attributes in Mac\u2019s Finder are preventing these files from being de-duped any further.<\/p>\n I still have two files\u2014the \u2018original\u2019 files downloaded to my Mac and PC. Gemini insists they are different files, but we know they are not, so let’s meet some new tools.<\/p>\n I could, of course, use Mac\u2019s Finder to reset the permissions of this single file downloaded from Windows. But what if I\u2019m faced with having to reset permissions on thousands of files at once?<\/p>\n To show how you can combine several tools at once, chain the \u2018find\u2019 and the \u2018chmod\u2019 commands together to first find all documents in my current folder, then change permissions on all of them at once.<\/p>\n Next, I\u2019ve decided that I want to clear all of the extended attributes that the Mac has set on these files. For this task, I\u2019ll use Apple\u2019s xattr tool.<\/p>\n Now, when I rerun Gemini 2 on this folder, I identify the last duplicate, delete it and I\u2019m back to one file<\/a> again.<\/p>\n As we\u2019ve seen, the metadata carried by the files you use every day changes over the life of the file as it moves from system to system, and server to server. And those changes can be problematic when it comes to the usefulness and security of your data.<\/p>\n You now have the power to see that information, inspect it, and\u2014with the tools listed below\u2014you can change it, solve the mysteries that crop up trying to mediate those changes, and clean up metadata you don\u2019t want made widely known when you share the files.<\/p>\n Do you have more questions about file metadata and how it affects how you use and save your files? Let us know! Meanwhile, the tools listed below are excellent starting points to aid in further exploration.<\/p>\n Here is a list of tools referenced in the article, and other interesting command-line and GUI tools to move, dedupe, and rename files:<\/p>\n exiftool<\/a><\/strong>\u2014Hands-down the most widely used metadata exploration tool, which lets you inspect and manipulate standard EXIF and other associated metadata. Latest Windows and macOS downloads are available on the exiftools.org website, via Linux package system, or on a mac with \u2018brew install exiftool.\u2019 There are many GUI ports available from the website as well.<\/p>\n rclone<\/a><\/strong>\u2014Uses rsync style syntax to copy and sync file locations to and from the widest variety of destinations including almost every known cloud storage choice.<\/p>\n xattr<\/strong>\u2014A macOS system tool to inspect, create, or remove file extended attributes.<\/p>\n ranger<\/a><\/strong>\u2014An old school \u2018file commander\u2019 that includes an embedded metadata pane. Binaries available, build from source, or on a Mac install with \u2018brew install ranger.\u2019<\/p>\n MacPaw Gemini2<\/a><\/strong>\u2014Still one of the most widely-used GUI de-dupe tools on the Mac.<\/p>\n fdupes<\/a><\/strong>\u2014One of several available command-line de-duping tools.<\/p>\n A Better Finder Rename<\/a><\/strong>\u2014A GUI tool to rename batches of files, and even rename according to parent folder structure and EXIF information.<\/p>\nWhat is file metadata?<\/strong><\/h2>\n
Information about a File for Humans<\/em><\/h3>\n
Information about a File for Computers<\/em><\/h3>\n
![\"Examples](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/os-preview-examples-878x1024.png\")
![\"An](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/macOS-user-permissions.png\")
Information about a File for Applications<\/em><\/h3>\n
![\"Application](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/app-specific-metadata-example.png\")
<\/a>Information You Won\u2019t<\/em> Want to Share<\/h3>\n
![\"A](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/ms-office-document-metadata-redacted-1024x656.png\")
<\/a>How Metadata Changes as You Move Files from Place to Place<\/strong><\/h2>\n
File Permissions Can Change from Macs, Windows, and Linux<\/h3>\n
File Creation and Modification Dates Can Change, Too<\/h3>\n
Permissions and Timestamps Can Change from Network and Cloud Storage File Metadata and Cloud Servers<\/h3>\n
A Real World Example of Changing File Metadata<\/strong><\/h2>\n
First: A Computer-to-Computer Test<\/h3>\n
![\"Comparing](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/mac-and-windows-dl-comparison-1024x469.png\")
<\/a>\n
Next Stop, the Cloud<\/h3>\n
![\"Comparing](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/finder-results-all-dl-files-1024x414.png\")
<\/a>![\"Command](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/ls-lafR-results-1024x615.png\")
<\/a>Solving Metadata Workflow Mysteries and Challenges<\/strong><\/h2>\n
Workflow Example 1: Using Metadata Tools to Learn About a \u2018Mystery\u2019 File<\/h3>\n
![\"MacOS](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/t4t-preview.png\")
<\/a>![\"Solving](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/exiftool-and-finder-compared-1024x631.png\")
<\/a>Workflow Example 2: Uncovering Duplicate Files<\/h3>\n
![\"De-Dupe](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/dedupe-confusion-1024x499.png\")
<\/a>Setting Proper Permissions<\/h4>\n
![\"Chaining](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/code-snippet1-1024x514.png\")
<\/a>Cleaning Mac Extended Attributes<\/h4>\n
![\"xattr](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/code-snippet2.png\")
<\/a>![\"The](\"https:\/\/www.backblaze.com\/blog\/wp-content\/uploads\/2020\/02\/gemini-final-results-1024x487.png\")
<\/a>File Metadata Takeaways<\/strong><\/h2>\n
Addendum: Tools Reference<\/strong><\/h2>\n