Neither the word Passant nor the word Fußgänger have any intrinsic relation to a pavement. Both can be used for people on or off a pavement.
Fußgänger, as you correctly noted, is a person walking on foot, i.e. the nominalisation of zu Fuß gehen. It technically could apply to people walking on foot anywhere (in a building, on a train, in the street, in the forest …) but it is usually only used in contexts of traffic. It does not state that the person is walking on the pavement. It can be used and is used for people walking at the side of roads that don’t have pavements, walking on the middle of a road, etc.
Passant is a borrowing from the French verb passer meaning to pass. Thus, it does not refer to any specific mode of transport in itself, nor does it refer to a specific place of a road; it merely signifies that somebody is passing. In the overwhelming majority of cases, though, it will be somebody walking past you. However, it does not have to be in the context of traffic and most importantly there is no requirement for a pavement anywhere.