In addition to RDBury's comprehensive answer, it's probably helpful to think of it as an "implied passive", so to speak. You could say, "who's doing the cold to whom" ;)
Das Glas ist kalt.
The glass is cold.
the glass "is doing the cold" and can, for example, chill the drink in it. On the other hand, in
Mir ist kalt.
I'm getting chilled by my cold environment. My environment is "doing the cold", I'm the recipient of the coldness, so to speak.
We're not talking about passive voice grammatically / technically, mind you, but more about an implication on the meaning level. And the idea of something "doing its cold to something else" will probably make everybody who remembers some middle school physics raise an eyebrow, but that's the way the phrasing works ;)
The basic idea is also used for other points on the temperature spectrum, like "Mir ist warm" or "Mir ist heiß", but also in phrase like "Mir ist langweilig" (something is boring me, something is doing the boredom to me). Occasionally, the "actor" is still present in the sentence, for example in "Das ist mir peinlich" (this is embarassing to me).