In German, what number and type of objects a verb needs has to be learned. There are two relevant meanings of passen here, DWDS 1a and 1b.
The first states that a piece of clothing fits somebody with regard to size and cut. The piece of clothing is the subject, the person is a dative object (which is optional and can be left out).
Seit meiner Diät passen mir (dat.) meine Hosen (subj.) wieder.
The plural of the verb passen shows that meine Hosen and not mir is the subject.
The second meaning of passen states that two things accord well, go together. One thing is the subject, the other a prepositional object introduced by zu. When introducing prepositional objects, prepositions are meaningless. In particular, zu does not have a directional meaning in this case.
Der Hut (subj.) passt nicht zur Jacke (prep. obj.).
Note that this meaning is wider than the first and not limited to pieces of clothing and persons.
Die Verpflichtung des Schweden passt zur Strategie von YouTube.
I don't know whether this is part of the confusion, but note that German word order is free in so far as (almost) any constituent can appear in first position in declarative sentences. This includes prepositional objects.
Zu dieser Jacke passt der Hut nicht.
Zu dieser Strategie passt die Verpflichtung des Schweden.