In this, "dir" is the dative object. And dative objects are the recipient of the action (to/for whom action is taken). Then, how "dir" become the one who is taking action instead of receiving action?
First off, "dir" is not the one taking any action.
But let us start at the beginning:
The sentence in question is of the form:
Ich war [attribute].
"egal" is an Adjektiv and hence an attribute. There is no other "action" involved than "war", which is Präteritum of "sein" (to be). The construction is the same as:
Ich war reich.
Ich war gesund.
Ich war jung.
and so on. Some attributes can be "isotropic" (e.g. if you are young, you are young in every direction), but some aren't: a house could be "open" in one direction but be "closed" in the other. There must be a way to express such things and exactly this is the case here:
Ich war egal. (in every regard, "omnidirectional", so to say)
Ich war dir egal. (only for you, others might value me).
The difference to "you didn't care about me" is - as I have explained at length here - that "to [not] care" indeed is an action, expressed by a verb which, in turn, has indeed an object. This is not the case with the German sentence, which describes the same fact by an attribute. German lacks a Verb meaning the same as "to not care about" in English, English, on the other hand, doesn't have an adjective/adverb with the meaning of "egal".