Anhören is not the same as zuhören:
- Listen very carefully to what a person is saying, for example when a witness makes a statement in court. The word, you often use for a setting in which this verb is appropriate is called »hearing« (in German: Anhörung).
- Listen to something that has been recorded before, for example listen to a song.
- Listen to someone without particular attention, i.e. not so focused and carefully like a judge who is listening to a witness in court.
Ich habe dich nicht angehört.
This means, that there was no hearing where I listened to you. I guess, the setting, where your dialogue is happening, is not what you would call a hearing, so this sentence is not a good choice.
Ich habe dich nicht zugehört.
This is grammatically wrong. The pronoun for 2nd person must not be used in accusative case, but dative case. This would be correct:
Ich habe dir nicht zugehört.
This also is the sentence you are looking for. It exactly means »I didn't listen to you.«
Ich hörte nicht zu.
Although syntactically correct, this is the wrong tense. It's past tense (Präteritum) but present perfect tense (Perfekt) (like in all the examples above) should be used because what happened is closely related to the present. In fact, in many regions the past tense is not used in oral communication at all.