more idle thoughts - about file metadata

Posted by jimspoon
Nov 13, 2012 at 07:45 PM

Slartibartfarst wrote:
“We need to be able to attach user-defined metadata fields to any type
>of file.”
> >Is this a general requirement?
>To what purpose?
>Cold you taek advantage of the “pack to go” files/folders, where the
>NTFS streams are bundled into the package, before the file is sent out
>somewhere or moved to non-NTFS media?

Well - inevitably a large part of our information is stored in discrete files ... notes, documents, photos, sounds, etc., while other information is stored in structured databases - emails, notes, contacts, tasks, etc. Ideally we would be able to search both types of “information stores” at once, and filter and sort the retrieved information in useful ways. Traditionally we have stored files in the directory tree structure. But as files have many different properties/attributes etc, they could be organized in many different ways. Just for example, you could arrange software programs by the author/vendor (the default method in Windows in the Program Files folder), or by function (the method I try to use in structuring my program files). You could organize your photo files by camera, date taken, event, persons appearing, etc. etc. Now we could use “hard links” or “symlinks” to categorize files in different ways .. so they would appear in several different locations in the directory tree. But if that information is stored in the file or directory system ... rather than the files themselves, it can be easily lost when moved around.

With adequate file metadata capabiltiies, notes could be stored in individual files, and this metadata could be used to retrieve, filter, and sort both notes and other related files.

I think it’s been an ongoing goal of Microsoft to transition from a tree-based structure to a database-type or tag-based structure for a long time ... as far back as Vista. A number of third-party file-tagging programs have appeared - but these suffer from the same problem .. the tags are usually lost when the files are moved out of the particular file system where it resides. Microsoft has come out with a new file system for Windows 8 called ReFS or something like that .. Resilient File System, which allows for much more in the way of metadata capabilities. For now, though it appears to be only for servers.

That’s an interesting idea about bundling NTFS data streams together when the data is sent out of the file system. The receiving system would have to know how to handle the bundled streams, of course. I suppose it’s lack of a standard that has prevent this sort of thing from happening.

Just for example ... when EXIF/IPTC/XMP metadata started to be included in image files, you had the problem that many image viewers/editors did not know how to handle this extra data in the file. Now most of them do, I suppose. Similar standard could be useful for other types of files too.

MP3 files have a well-established metadata system - players/editors on different operating systems know how to handle it.

A related point ... if “hard links” and symlinks were used to implement categorization of files ... that categorization would be lost if files are sent out without the other hardlinks/symlinks. Somehow the metadata needs to be contained in the files themselves.