In a fantasy setting it is quite uncommon to use Sie. You would use du or the pluralis majestatispluralis majestatis form Ihr
So the question would rather be, why is Gandalf saying
Du kannst nicht vorbei!
Insteadinstead of
Ihr könnt nicht vorbei!
You are right that animals would be addressed as du in German and, furthermore, that the Balrog cannot be considered an animal. At least I would consider to saysaying Ihr to a powerful evil spirit, toso as not to anger it. However, diving into the depths of Tolkien's world, Gandalf and the Balrog go back to the same race (the primordial spirits, Maia). As such, it could be argued that Gandalf is somehow familiar with it (and he knows that the Balrog is to be expected in the mines) and as such chose the du form. In that way he also implies he's on the same level, if not above it, and emphasizes this with his words. Another argument would be that the Balrog was already angered and that it was not really time to use courtesies.
However, this could be deemed too much of an interpretation and the truth could simply be that the translator didn't care much onabout this particular translation.
To address the question about können:
The way it is emphasized in the movie, it does not mean
You are unable to pass.
But rather
You are not allowed to pass Du darfst nicht vorbei
Which agrees with You shall not pass.