Actually, über demands an accusative here, and that's why "alle" is correct here. ("über alle Maßen" means "über alle Maßen hinaus", so you have to ask "über wen?" and not "über wem?", über indicates a direction in this case).
But of course this does not solve your problem, since the mismatch seems to persist. The solution is that "Maßen" is actually not dative case here, but accusative as well.
But how can this be? The standard plural declension of "das Maß" would be
Nominative plural: die Maße
Genitive plural: der Maße
Dative plural: den Maßen
Akkusative plural: die Maße
But in the idiomatic expression über alle Maßen, "Maßen" is not the plural of the present-day German word das Maß, but of an elder variant die Maß, which has nearly completely vanished in Modern German from the 18th century on. Etymology of das Maß shows that it is a blending of Middle High German māʒe (feminine) and Middle High German meʒ (neuter). Die Maß has survived in Upper German die Maß and in present-day German dermaßen, gewissermaßen and einigermaßen (see Pfeifer for more detailed information) and - as I would conclude - in the idiomatic expression "über alle Maßen".
I have no proof for my hypothesis, that "über alle Maßen" is indeed a relict of the feminine variant die Maß, but it would explain the grammar of the idiomatic expression.