It was already answered in the comments, but I'll give a detailed answer in case more people stumble across this.
This construction, as I assume you guessed from the English sentence, is a special form of the Perfekt. Many modal verbs, especially if they are accompagnied by an infinitive, use this alternative form of the past participle which resembles the infinitive. I disagree with calling it infinitive though, because it's grammatical purpose is that of a past participle, not of an infinitive.
(Minor side note: It also changes the word order, which in a standard perfect tense sentence would have hat last.)
The second verb (arbeiten) has to be an infinitive (because it's an infinitive construction), so it has to end in n. Thus, there is a typing error in your book.
Just for completeness, the correct sentences:
»Ich musste an dem Buch drei Jahre arbeiten.«
Er behauptet, dass er an diesem Buch drei Jahre arbeiten musste.
»Ich habe an diesem Buch drei Jahre arbeiten müssen.«
Er behauptet, dass er an diesem Buch drei Jahre hat arbeiten müssen.