The original and your translation don't match here, starting with my impression that the English sentence When I had a lot of homework to do, I was irritated is not clear. That is because it is in the past and lacking a context. So, two kinds of translations seem possible:
Es störte mich (immer), wenn ich viele Hausaufgaben zu erledigen
hatte. (conditional clause)
als auch:
Als ich (einmal) viele Hausaufgaben zu erledigen hatte, ärgerte mich
das. (temporal clause)
However, the first version would rather assume a phrasing like "I used to be irritated when I had a lot of homework to do".
As soon as you put the English original into present tense, it becomes clear what should be the proper translation:
Wenn ich viele Hausaufgaben habe, stört mich das.
Thus, keep in mind as a rule of thumb:
When (conditional) = Wenn
When (temporal) = Als