You're completely right: you could just as well say "ihm" here, of course. If you then also use the "standard" word order (with the subject in position 1), you end up with a perfectly neutral, matter-of-factly sounding sentence:
Nein, ich gebe ihm mein Auto nicht.
But the clue is already in the fact that in this sentence, they've put the dative object - "(to) him" - in the first position, which in itself serves to emphasise the dative object over the subject.
This effect is strengthened by using a demonstrative pronoun. "Demonstrate" comes from the Latin for "to show" or "to point", and demonstrative pronouns are about doing exactly that -- "pointing" at an object in either the direct (choosing stuff to buy at a store) and the more abstract (designating one of various options) sense. And in the way it is used here, this emphatic pointing at "him" has exactly the same meaning as saying "that guy" in English - you're singling him out for being particularly horrible/unreliable etc.
To recap:
- Ich gebe ihm mein Auto nicht. - totally neutral: I don't give him my car.
- Ihm gebe ich mein Auto nicht. - emphatic: I'm not going to give my car to him.
- Dem gebe ich mein Auto nicht. - very emphatic: I'm not going to give my car to that guy [out of all people].