Using the infinitive 'zu' construction can be a little difficult to grasp. The best way to think of it is substituting an object of the sentence for a verb which has no subject - but it also has some special grammatical rules associated with it too.
Du hättest früher anfangen sollen zu packen.
This is completely correct, because it's fine to move an infinitive to the VERY end of a sentence, almost as if it is its own clause.
Du hättest früher zu packen anfangen sollen. / Du hättest früher zum Packen anfangen sollen.
Also completely correct.
It's important to think of the infinitive as its own unique and independently standing object/subject, not really as a verb anymore.
Du hättest früher anfangen (zu packen) sollen.
This doesn't quite work because you can't treat infinitive 'zu packen' as a verb which goes in the place of a verb, but instead treat it as an object with the special property that it may go to the very end in this sentence.