As I dig through the database structure, there seems to be a potential ambiguity in the wp_term_relationships intersection table. There is nothing that qualifies the object type referenced by object_id.
In the documentation an "object" in this context can be either a POST or a LINK. So it's possible to have a post with ID 5 and a link with ID 5.
Let's say we associate Link #5 with Category #1 and Post #5 with Category #2.
Now there will be 2 entries in wp_term_relationships -
object_id, term_taxonomy_id, term_order
5, 1, 0
5, 2, 0
If I am trying to find all categories for post ID #5, it will look like that post has been associated with Category #1 AND #2!
What am I missing?