Blame the Swiss.
The ⟨ß⟩ is the result of two spelling conventions converging. I’ll describe the more prominent one first:
- At least in blackletter type, word-final s always took the round form ⟨s⟩ whereas pretty much everywhere else the lowercase letter retained the default long form ⟨ſ⟩.
German orthography has quite systematic rules for doubling of consonant letters, so double s should work by the same principles as ⟨ff⟩, ⟨ll⟩, ⟨mm⟩, ⟨nn⟩, ⟨pp⟩, ⟨rr⟩ and ⟨tt⟩. All of these can only appear at the end of a word or as the only letters between vowels, whereas (voiced/soft) ⟨bb⟩, ⟨dd⟩ and ⟨gg⟩ require the vowel following them.
Together with typesetter’s rules, we got ⟨ſſ⟩ in word-medial and ⟨ſs⟩ in final position. For the latter, it was common in calligraphy and typography to form a ligature. This tradition survived when Fraktur etc. were abandoned, because it had come to be treated as an orthographic rule, although it really was a typographic rule.
This convention was finally abolished in the 1996 orthographic reform. It was the one that originally informed the substitution of ⟨ß⟩ by ⟨ss⟩.
- Special treatment is not unique to s, by the way: ⟨ck⟩ is a special way of writing *⟨kk⟩ and ⟨tz⟩ is used instead of *⟨zz⟩. Some may go further and consider ⟨ch⟩ a replacement of *⟨hh⟩ (e.g. in sehen > Sicht), but unlike all other double consonants it can be separated from the preceding vowel by an ⟨r⟩, ⟨l⟩ or ⟨n⟩ (because it replaced a former single ⟨k⟩).
- A solution was needed to represent /s/, because single ⟨ſ⟩ (or ⟨s⟩) was already used for /z/ in initial and medial position and a double consonant like ⟨ſſ⟩ (or ⟨ss⟩) would have regular, but unwanted implications.
Single ⟨z⟩ (usually looking more like ⟨ʒ⟩, especially in final position) could have been employed for this purpose, although it represents /ts/ in initial (onset) position and after other consonants in codas, because ⟨tz⟩ (or ⟨tʒ⟩) in native and ⟨c⟩ in loan words was employed for that sound in medial and final position – but it wasn’t and when ⟨z⟩ replaced Romance ⟨c⟩ that alternative was gone.
Another possibility might have been to indicate altered vowel quality in a different way, e.g. by adding an /h/, but firstly that would have to be dropped sometimes (e.g. essen, aßen *⟨ahsen⟩, *⟨aassen⟩; äsen; aasen) and secondly double consonants generally follow a simple vowel without interruption.
The digraph that evolved instead was “Eszett” ⟨ſʒ⟩ (in classic shapes).
Grimm’s Deutsches Wörterbuch and Bundeswehr telex cables always used ⟨sz⟩ in place of ⟨ß⟩, for instance.
It survived in rare cases of adopted initial /s/ where it replaced former ⟨ſc/sc⟩. The digraph had long been replaced by ⟨ß⟩ when this phonographic rule was kept in the recent reforms. However, not least due to multilingual mechanical typewriters, Switzerland and Liechtenstein had abolished ⟨ß⟩ decades earlier – in favor of ⟨ss⟩ in all cases.
- In simple onsets, German natively only supports /ʃ/ and /z/, not /s/. In codas, it has only /ʃ/ and /s/ (due to “Auslautverhärtung”). Complex onsets require /ʃ/ except for /ts/ which is the only native one that has another consonant preceding an S sound.
In conclusion, where the ligature has lost orthographic status almost 20 years ago, it was rightfully replaced by ⟨ss⟩, because the long form isn’t in use any more. That part was compatible with what the Swiss were already doing anyway.
The remaining uses of ⟨ß⟩ really should be substituted by ⟨sz⟩ where necessary!
Alas, that was not often required to be done in the 20th century by enough writers, and many people already got the main ss/ß rules wrong too often (because the typographic rule wasn’t well understood once its use wasn’t obvious any more with the demise of ⟨ſ⟩), so it’s not a big surprise the non-mandatory, vague recommendations for replacement were simplified in practice.
The Swiss probably ditched the ⟨sz⟩ replacement because having a single variant was perceived as simpler and a double letter was the regular choice for the more frequently occurring case while creating hardly any (semantic) ambiguities for the less important case.
The reformers chose not to abolish ⟨ß⟩ completely (as had been proposed by some), but to promote it to a proper (lowercase-only) letter. It’s a pragmatic choice, even though a bad call by linguistic purity, to keep the fallback compatible with widely established practice within the largest affected user base, because most writers would have the ⟨ß⟩ available anyway.
I’m pretty sure that the reason was not an unfamiliar appearance of the digraph ⟨sz⟩, because it already appears (alongside stranger things) in loan words and ⟨tz⟩ works in much the same way. One could argue that ⟨tz⟩ should be hyphenated as ⟨z-z⟩, like regular double consonant letters and ⟨ck⟩ previously, which became ⟨k-k⟩, but ⟨sz⟩ would still have to be broken into ⟨s-s⟩, which would better match a *⟨cs⟩ digraph.
Corollary: Although the shape of lowercase ⟨ß⟩ builds upon the ⟨ſs⟩ ligature in most typefaces, a newly designed uppercase variant ⟨ẞ⟩ – often an ugly ⟨B⟩ lookalike – may well be based loosely upon an ⟨SZ⟩, ⟨Sz⟩, ⟨Sʒ⟩, ⟨ſz⟩, ⟨ſʒ⟩ or even ⟨SƷ⟩, ⟨ſƷ⟩ or ⟨ſZ⟩ ligature.
