Jeden Morgen tritt Jack Nicholson meinen kleinen Hund, der mich immer wütend macht.
A relative pronoun must match the gender of its antecedent. In the given sentence, der is masculine and there are therefore two possible antecedents in the matrix clause: the subject Jack Nicholson and the object meinen kleinen Hund. However, the relative clause can only be interpreted as referring to the latter.
Note that this is the case despite the fact that the relative clause has intentionally been phrased in such a way that it is semantically more plausible for Jack Nicholson to be the antecedent (dog-kicking tends to make people angry at the person doing the kicking). This makes the fact that the relative clause cannot have Jack Nicholson as its antecedent all the more salient.
Also note that if we replace masculine Hund by feminine Katze, the only possible antecedent of the relative clause is Jack Nicholson, and yet the relative clause still cannot be interpreted as referring to him. The following sentence is ungrammatical:
*Jeden Morgen tritt Jack Nicholson meine kleine Katze, der mich immer wütend macht.
We can conclude that it is not enough for a relative pronoun to match the gender of its antecedent. The relative clause it introduces must also be positioned correctly. To quote a rule from the same page:
The relative clause always comes right after the noun it is describing.
(But note the exception for "dangling verbs".) Since the relative clause in the original example follows the object, it can only modify the object, i.e. meinen kleinen Hund.
Finally, as far as case is concerned, note that the case of the relative pronoun is completely independent of the case of its antecedent. They occur in different sentences, after all! Let's look at some examples from the section How to choose the correct relative pronoun on this page with supplementary information.
Das ist der Laden [Nom.], den (Acc.) ich liebe.
Wir gehen in den Laden [Acc.], den (Acc.) ich liebe.
Wir sind in dem Laden [Dat.], den (Acc.) ich liebe.
Wer ist der Besitzer des Ladens [Gen.], den (Acc.) ich liebe?
Das ist der Laden [Nom.], dem (Dat.) ich €20.000 schulde.
Wir gehen in den Laden [Acc.], dem (Dat.) ich €20.000 schulde.
Wir sind in dem Laden [Dat.], dem (Dat.) ich €20.000 schulde.
Wer ist der Besitzer des Ladens [Gen.], dem (Dat.) ich €20.000 schulde?
Observe how the case of the relative pronoun is determined by the verb in the subordinate clause (lieben + accusative in the first four examples, schulden + dative in the others), whereas the case of the antecedent is determined within the matrix clause.