The original and your translation don't match here, starting with my impression that the English sentence When I had a lot of homework to do, I was irritated is not clear. That is because it is in the past and lacking a context. So, two kinds of translations seem possible:
Es störte mich (immer), wenn ich viele Hausaufgaben zu erledigen
hatte. (conditional clause)
Als ich (einmal) viele Hausaufgaben zu erledigen hatte, ärgerte mich
das. (temporal clause)
However, the first version would rather assume a phrasing like "I used to be irritated when I had a lot of homework to do".
As soon as you put the English original into present tense, it becomes clear what should be the proper translation:
Wenn ich viele Hausaufgaben habe, stört mich das.
Thus, keep in mind as a rule of thumb:
When (conditional) = Wenn
When (temporal) = Als