Tjyylli

The University of Michigan gives this sentence on their page about relative pronouns:

Jeden Morgen tritt Jack Nicholson meinen kleinen Hund, der mich immer wütend macht."

The page said the der referred back to meinem kleinen Hund but since Jack Nicholson is in the nominative position of this sentence (meinen kleinen Hund is in the accusative), why wouldn’t der refer back to Jack Nicholson?

Jeden Morgen tritt Jack Nicholson meinen kleinen Hund, der mich immer wütend macht.

A relative pronoun must match the gender of its antecedent. In the given sentence, der is masculine and there are therefore two possible antecedents in the matrix clause: the subject Jack Nicholson and the object meinen kleinen Hund. However, the relative clause can only be interpreted as referring to the latter.

Note that this is the case despite the fact that the relative clause has intentionally been phrased in such a way that it is semantically more plausible for Jack Nicholson to be the antecedent (dog-kicking tends to make people angry at the person doing the kicking). This makes the fact that the relative clause cannot have Jack Nicholson as its antecedent all the more salient.

Also note that if we replace masculine Hund by feminine Katze, the only possible antecedent of the relative clause is Jack Nicholson, and yet the relative clause still cannot be interpreted as referring to him. The following sentence is ungrammatical:

*Jeden Morgen tritt Jack Nicholson meine kleine Katze, der mich immer wütend macht.

We can conclude that it is not enough for a relative pronoun to match the gender of its antecedent. The relative clause it introduces must also be positioned correctly. To quote a rule from the same page:

The relative clause always comes right after the noun it is describing.

(But note the exception for "dangling verbs".) Since the relative clause in the original example follows the object, it can only modify the object, i.e. meinen kleinen Hund.

Finally, as far as case is concerned, note that the case of the relative pronoun is completely independent of the case of its antecedent. They occur in different sentences, after all! Let's look at some examples from the section How to choose the correct relative pronoun on this page with supplementary information.

Das ist der Laden [Nom.], den (Acc.) ich liebe.

Wir gehen in den Laden [Acc.], den (Acc.) ich liebe.

Wir sind in dem Laden [Dat.], den (Acc.) ich liebe.

Wer ist der Besitzer des Ladens [Gen.], den (Acc.) ich liebe?

Das ist der Laden [Nom.], dem (Dat.) ich €20.000 schulde.

Wir gehen in den Laden [Acc.], dem (Dat.) ich €20.000 schulde.

Wir sind in dem Laden [Dat.], dem (Dat.) ich €20.000 schulde.

Wer ist der Besitzer des Ladens [Gen.], dem (Dat.) ich €20.000 schulde?

Observe how the case of the relative pronoun is determined by the verb in the subordinate clause (lieben + accusative in the first four examples, schulden + dative in the others), whereas the case of the antecedent is determined within the matrix clause.

It's very much the same in English, "John loves Isi, who I just barely like". The grammatical case does not prohibit the construction, as @David said.

However, it might be more pleasing to continue in the same case, "... meinen Hund, den ...", leaving it in an object position. Although there's no overt preference, it stands to reason. Unless the relative clause restricts or explains the main clause (I mean what the hell, Jack, what was that for, and why would anyone permit that to happen repeatedly), it would be more natural to start a new sentence. Whereas, if restricting the main clause, the dog may well remain in object position, as the dog is not a subject that could be actively responsible for those actions. Phrasing the given relative clause with the dog as an accusative object would be rather difficult though (the passive transposition would use dativ "von dem"). Therefore it's not done and the resulting argument is subjective, not objective, and thus not conclusive. We would naturally have to ask, "what, wait a second, you don't even like the dog and let it be kicked?". Better would be thus, "... meinen Hund, den ich sowieso nicht mag, denn der ärgert mich immer". That's a matter of style more than grammar, and in many cases it is fine either way. But it's a good question.

BEcause the dog is the subject in the relative sentence.