What's a good solution for file-tagging in linux?

Question

I've been looking for a way to tag my files and search/filter them based on those tags.

Here are my (updated) requirements :

• any file readable by the user can be tagged freely
• a user can search for files matching one or several tags
• files can be moved around without losing the previously associated tags
• the system could be backed up easily
• no dependencies on any desktop environment
• if any gui is involved, there must be a cli fallback

I've been hoping for some basic filesystem & coreutils hackery to handle this, but I haven' thought about this hard enough yet.
Meanwhile I'll review beagle and metatracker, which have been mentionned here, and see how they perform.

---

Ok so beagle has huge gnome dependencies, and tracker is okish, but still has some dependencies I don't like...

Been doing some more research, and the way to go could very well be extended file attributes.
That's a native solution for most recent filesystems, but they aren't very well supported yet (most coreutils destroys them by default, cp for example needs the -a flag to preserve them). Would like to hear some thoughts on using them while I try my hand at some hacks myself, eventhough this might warrant a new question.

Answer 1

It's not clear what kind of searching you want. If you want it to work anywhere in unix, rather that just your home directory, and you only want to do pathname-based searches, the following scheme is workable, with a little bit of shell hackery, and using the standard locatedb:

1. Each directory that contains at least one tagged file needs a standard subdirectory, say .path-tags;
2. Each file in the directory $FILE with link $TAG (which should not contain the char _) has a link $TAG_$FILE -> ../$FILE

I leave the details of the locate-tag script to you; it should be a two- or three-liner, using only the locate command and shell hackery. (If you're interested, I could write one).

Some of the KDE chaps talked about this sort of scheme for metadata, although I don't recall the details.

It should also be possible to do more sophisticated, content-examining tests based on this scheme with a similar script wrapped around find.

Thoughts on updated requirements

1. any file readable by the user can be tagged freely - Yes, should be no problem
2. a user can search for files matching one or several tags - Likewise
3. files can be moved around without losing the previously associated tags - The directories they inhabit can be freely moved about, but if the file is moved from the directory, we are in trouble. If the tags took the form $TAG_$INODE_$FILE and we have an efficient way to find which paths have a given inode, then we can do this, losing tags only if we move out of filesystems. Copying files might make some trouble, and this is clearly more complicated than my original suggestion.
4. the system could be backed up easily - not essentially difficult.
5. no dependencies on any desktop environment - none
6. if any gui is involved, there must be a cli fallback - that's where we live!

Postscript The "reverse-inode-lookup" file described by the link (2) you showed me in your answer to (1) can be used to give some additional infrastructure. We can run a service on the reverse lookup file, which checks that each inode given in the filename of a tag matches the inode of the file (if any) the tag points to. If there is no match, then the required surgery can be performed (does the inode still exists? where is it?), and the reverse lookup file being either mutated or regenerated, and the tag symlinks being updated.

I anticipate one tricky case: what if the tagged file is not where the tags say it should be, the reverse lookup file says it still exists, but the prodigal file is not where the lookup file says it is, the lookup file being out of date? There are a few ways to handle this case, none obviously ideal. Apart from this, this whole task seems to be the kind of thing Perl is well-suited to...

Answer 2

I've just released an alpha of my new program that attempts to provide this functionality. It currently meets some, but not all, of your requirements. It may be of interest to you anyway. It provides a command-line tool for tagging and a virtual file-system for browsing (where tags are represented by directories).

http://www.tmsu.org/

any file readable by the user can be tagged freely

Yes.

a user can search for files matching one or several tags

Yes. Either via the command-line tool or by browsing the tag directories in the virtual file-system.

files can be moved around without losing the previously associated tags

No. However the application stores fingerprints of the files tagged which are used to help identify moved files. A 'repair' command is provided that will update the paths of moved files. (Obviously this mechanism breaks down if a file is both moved and modified.)

the system could be backed up easily

Yes. It's a simple Sqlite 3 database file.

no dependencies on any desktop environment

Yes. No dependencies and as it can be run as a virtual file-system it is available to peruse as a file-system in any program that supports symbolic links.

if any gui is involved, there must be a cli fallback

No GUI at present.

Answer 3

Nobody mentioned, but you definitely should look at extended file system attributes. ext4 for example has them. there are tools getfattr and setfattr to deal with them. Of course you will have to write some shell scripts to search for files tagged with sometag. Regarding mentioned questions all the answers are "Yes". You should only take into account that it's depended on file system.

Answer 4

Suprised that nobody has mentioned TagSpaces. It meets all your requirements because tags are stored in the filename and TagSpaces is cross-platform.

Answer 5

I think this might meet all your requirements. In any case, it is a cool piece of code:

http://pages.stern.nyu.edu/~marriaga/software/oyepa

The GUI requires Qt, but there is a command-line application for searching and the fact that all tags are actually in the filename makes it trivial to manipulate tags|files from the cli.

Answer 6

You probably don't need to install entire KDE desktop for their tagging library, Nepomuk. You would still have to install KDE base libraries, though...

Answer 7

Some other alternatives might be tagasistant, tagfs or dantalian.

Answer 8

This recent article on Linux Desktop Search Tools mentions that Tracker supports tagging. Unfortunately it's supposed to be half-broken in the old version they tested. Maybe it's fixed now?

1. Not system wide.
2. You can back it up.
3. It's bundled with Gnome.

Answer 9

Try Beagle. I find it is pretty good.

It may not meet all the requirements, and I'm not sure what could. For example, do FIFO files support extended attributes? If they don't, Beagle has a fallback database.

Answer 10

So you won't find Nepomuk integration in gnome, at the command line, or elsewhere in Linux.

Conversely, with Tracker you won't find kde integration AFAIK. Not sure on CLI.

So unfortunately, the answer appears to be "no".

Even more unfortunately, this doesn't mean there's a good opportunity here for building one either. Linux commandline utilities dont have much in common with the GUI file manager, for example, so architecturally there's no common componentry which could be extended to support the concept.

Answer 11

TMSU

TMSU is a tool for tagging your files. It provides a simple command-line utility for applying tags and a virtual filesystem to give you a tag-based view of your files from any other program.

TMSU does not alter your files in any way: they remain unchanged on disk, or on the network, wherever your put them. TMSU maintains its own database and you simply gain an additional view, which you can mount where you like, based upon the tags you set up.

Surprised no one has mentioned it.

Answer 12

I made a little program that uses SQLite for this purpose. It solved my need, but maybe it helps you too:

https://github.com/alvatar/dfym

The only issue with this approach is that does not synchronize with moves and deletions, but it solves the problem for relatively static files.

Answer 13

I suggest taking a look at a version control system such as Subversion for these kinds of features above and beyond the file system. Some may be a better fit for you than others but generally:

• Many support tagging (certainly subversion).
• Many are cross platform; Windows, Mac, Linux, pretty much all Unixes.
• Many have both GUI front ends and command line clients.
• Many already have bindings for your favourite programming/scripting language.
• Many are easily backed up.
• Many are designed to be very easily shareable in one way or another.
• Many allow you to control access.
• You don't have to re-invent the wheel.
  • You learn and use standard commands/tools already used by millions.
• You can install it today for your favourite OS repo; apt-get install, yum install
• You also get version management "for free".

A cli example with Subversion: ~/svn/atestrepository: $ svn propset mytag "something" dir1 property 'mytag' set on 'dir1' $ svn propset myothertag "nothing" dir1/file1 property 'myothertag' set on 'dir1/file1' $ svn propset anemptytag "" dir1/file2 property 'anemptytag' set on 'dir1/file2'

$ svn propget -R mytag dir1 - something ~/svn/atestrepository: $ svn propget -R myothertag dir1/file1 - nothing $ svn propget -R anemptytag dir1/file2 - $ svn proplist dir1/file2 Properties on 'dir1/file2': anemptytag svn:keywords

I wouldn't recommend these tools is for large (gigabyte sized) regularly changing binary files but for everything else they are already well proven and scale to very large sizes.