I made heavy use of OpenMeta (that was a community effort on OSX, before Apple introduced something very similar), that placed tags in xattrs. It is great! Retrieving never caused any issues or slowdowns.