4

I am making a German-language version of my iOS app, and I'm trying to make sure I have a great search experience for German-speakers, but I don't speak any German, so I'm having a hard time figuring out exactly how to handle some of the unique aspects of the language. In particular, I'm trying to figure out how German-speaking people would expect umlaut and ß to be handled in search.

  1. Should I expect that users will always use ß and not "ss", ö and not "oe", etc., or will there be users who would prefer to enter the latter? For example, if a user searches for "muesstet", should it match content containing "müßtet"?
  2. Should searches match when diacritics are simply removed? For example, if someone searches for "mussen" should it match "müssen"?
  3. Is ß always equivalent to "ss"? Looking at this conjugation chart, it looks like some conjugations use "ss" and others "ß". Should I simply not be messing with converting between these two?

Technical Context

There was some interest in the comments and answers for more technical details, so I'll provide them here, although you don't need to understand this to be able to answer the question.

My app is a Japanese-English dictionary, which I am expanding to also be a Japanese-German dictionary. When users search in German, it will be matching German definitions of Japanese words. I want to make sure I match whatever form German users would naturally type queries, including matching conjugated forms of words to matches that are unconjugated, or vice versa.

Internally, I’m using SQLite’s Full Text Search module. In English, this provides a tokenizer (“porter”) that handles all of this for you, including lemmatization of the words. But it doesn’t exist in other languages. So I’m using the built-in iOS lemmatization (NSLinguisticTagger) for German, and built-in String Transforms to convert diacritics.

However, what doesn’t exist in the system is the logic of when to use the string transform, so that I match as broad as possible, without matching things that would be obviously wrong to match to a German-speaking person. That’s what I’m looking for advice on here.

  • 5
    Usually, there is system or system-related software (e.g. POSIX) that will properly give a sort order for words, depending on a given locale (look for collation). I bet this exists on iOS too. Don't do this manually, use the existing libraries. Use them for conversion between upper and lower case too. – Rudy Velthuis Jun 18 '18 at 12:52
  • In other words, rather ask in a programming community how you can sort properly (according to locale) using iOS built-in tools, and not here. – Rudy Velthuis Jun 18 '18 at 12:53
  • 3
    Just a comment because I don't really know how to do that. For the ß it is a bit more complicated than at first sight: 1) your example müßtet is an old spelling which was reormed in 1996 to müsstet but your app may want to find the old spellings too 2) in Switzerland the 'ß' does not exist and is always spelt 'ss'. – Takkat Jun 18 '18 at 13:25
  • As you don't speak any German, I doubt your app is about the German language. What terms are users typically entering into your search mask? – Janka Jun 18 '18 at 13:31
  • 3
    @tofro: I am not telling him how to implement the comparison. Actually, the answers do that more than I do. I am merely saying he should not implement it himself at all. Not interesting from a German Language POV, but a solution to his real problem, which is doing a proper locale dependent search. As I said: this is an XY problem. – Rudy Velthuis Jun 18 '18 at 15:23
3

Make it as broadly as possible, but be reasonable.

The best user experience would be to type in "Gefaskrankheit" and your app knows its "Gefäßkrankheit" despite the input typo. But you are not Google and you only have a limited amount of time and priorities have to be made.

You have to ask yourself how many users will write in these non standard ways. Will there be a significant number, so that its worth your time or will it be 1 in 1000? Also we live in a world of "word completion" and smartphones give suggestions while writing.

But lets say you still want to do it

a) ß: as @Takkat says in his comment you will always have to assume that ß is written as ss, because there is no ß in swiss german. Every "ß-word" converts into ss in Switzerland.

b) a, o, u: Maybe. It's uncommon and lazy to write like this, but you already have to assume every é, â and č (etc.) as e, a and c, so its no big deal to add this to your logic.

c) ae, oe, ue: Probably yes. Lets say someone uses your app with a non-german keyboard layout without any umlaut. He won't have any chance to find what he is looking for when your logic doesn't consider it.

In the end it is also a question of valence and user expectation. Is you app cheap? Well, don't expect a state of art search engine. Is it expensive/fancy? I hope everything works as expected even in the eyes of the dumbest person ever.

  • 4
    That is really nice, but very hard to implement. That is why I say don't attempt this yourself, especially since he has no knowledge of the language. These problems are solved already, in the OS. – Rudy Velthuis Jun 18 '18 at 15:40
  • It's an iOS app and you maybe right it is already solved by the OS. But lets say it will be ported to android/windows/xyz phones next month. Or in the future someone looks this question up with an non-iOS app in mind. Are there similar APIs? Anyway this is rather a question for stackoverflow. – mtwde Jun 18 '18 at 16:19
  • 4
    Yes there are similar (often even the same) APIs in all popular OSes, including Android, macOS, Windows and Linux. That is why it is simply foolish to try to do this on one's own. These solutions are much better than one could do without any knowledge of the language and even if one knows the language, it would still be a lot of work to get it right. And the APIs would work for other (human) languages too, like, say, Spanish or Swedish, or even Swiss German, etc. – Rudy Velthuis Jun 18 '18 at 16:41
  • @RudyVelthuis I think you are overemphasizing the support from OSs a bit - For IOS, for example, the support is relatively limited: You can handle "A" + <compose diaretic> like "Ä" and like "A", but, for example not as "AE", similarily, an "ss" and "ß" will only mach for case-insensitive compares. That doesn't for example handle triple consonants properly. – tofro Jun 19 '18 at 11:47
  • 1
    @RudyVelthuis I share your sentiment on re-inventing the wheel, but here I think a link to prove your points about how this is implemented in iOS would ease the tension (btw: the unix part of ApfelProdukte is severely overrated, so POSIX seems strange in this context). The degree of control over the fuzziness for the parameters are unclear to me on the question level, the OS level and the comment level, if that still makes any sense. The best experience starts with sensible defaults and gives the user fine grained controls over/degrees of freedom to refine the parameters. – LаngLаngС Jun 19 '18 at 18:26
2

Obviously, what everyone would expect is a pure literal search - Return exactly what was searched for. In case you target for something more, you can, however, open up a can of worms:

Simple thing is German Umlauts - I would expect you can cover these with a literal search. Transcriptions like "ue", "oe", "ae" are rarely used these days. In case you try to handle them, be aware that not every "ue" really is an "ü" transcription: "Quelle" does not translate into "Qülle"

There have, however been some relatively recent changes to the spelling rules, especially around "ss", "ß", and even "sss" that you might want to cover because these changes are not entirely mandatory and only slowly make their way into real written text. Even if you are asking only for Umlauts and ß, there are other pitfalls, like double and even triple consonants in composite words.

Flussschiffer (riverboat captain), for example, was until 1996 written as Flußschiffer, then changed to Flusschiffer for a short period, today proper spelling is Flussschiffer. If you want one search term come back with hits on all three possible spellings, you might get yourself into a bit of work.

Note that Swiss German doesn't know the "ß", so they are somewhat easier to serve, there.

The same problem with double and triple consonants as above does not only apply to the "s/ss/sss/ßs" combinations, but rather with many more consonants in composites ("Schifffahrt/Schiffahrt", "Schrittempo/Schritttempo",...)

Note people are somewhat spoiled by Google, which seems to first do a literal search and then come back with results based on possible variations.

Whatever you do, make sure you are presenting your search results in a match order (i.e. present exact matches first, then in some appropriate order of match distance) - Nothing is more annoying than searching for "rasen" and be presented with a list of results that has "Rasen", "Räsen", "Rassen", "Rässen" ... before the literal match.

  • 3
    Warum sollte ein Suchprogramm bei der Suche nach "Quelle" nicht das Wort "Qülle" finden, wenn dieses Wort (es könnte ein Firmenname sein) in einem Text vorkommt? – Martin Rosenau Jun 18 '18 at 18:44
  • 1
    @MartinRosenau Das hängt ein bisschen davon ab, was man sucht - In einem Firmenverzeichnis mag das anders sein als in einem Wörterbuch. Es steht auch nirgends, dass man nicht "Qülle" finden soll, sondern nur, dass die Umlauttranskription eben nicht symmetrisch ist. Ich denke, der Hinweis ist wertvoll für einen Fremdsprachler. – tofro Jun 19 '18 at 11:17
1

I'll be answering your questions from 3 to 1, because I have the feeling that this way the answers can build upon each other.

  • The "ß" is not equivalent to "ss". These are two different letters. However it is allowed to replace "ß" with "ss" if the typeset doesn't contain an appropriate symbol. The same applies technically for the other "Umlaute" ä, ö, ü, which can only be replaced if the typeset doesn't contain a representation for those letters.

  • As a native speaker I would expect the "Umlaute" to be matched,

    • when correctly used,
    • when replaced by ae, oe, ue,
    • and when left out (as a,o,u).

    While I personally prefer the first or the second option (when my keyboard doesn't contain the letter), I know a number of people who actually use the simple letter even when the keyboard provides the "Umlaut" version.

  • As I pointed out before, there are three cases (correct, replaced, omitted) that I expect to work, when using a search functionality. You should at least be able to handle these three cases correctly (as they are pretty simple indeed). However you might run into problems where the "corrected" spelling might result in a (correct but nevertheless) wrong word. E.g. "musste" (past tense) could be corrected to "müsste" (subjunctive mood). This however is not very probable if you are just searching for keywords or settings and in either case you should probably list all results for both cases.

1
  1. Both should be possible. Although 99% of all German users have German keyboards you must enable the search for users not having a German keyboard.
  2. The simple search shall not find "mussen" if you search for "müssen". The advanced search could for instance say "0 matches for müssen, but we have 30 matches with mussen"
  3. Same situation as we have with 2., actually. For instance, you know Switzerland does not have the ß and writes "Strasse" although they might mean a German street "Max-Mustermann-Straße".
  • 1
    I would actually never try entering "mussen" when I'd be searching "müssen" - "muessen" comes to mind much more easily. – tofro Jun 19 '18 at 13:28
  • @tofro: it's more a use case for when a non-German tries to use that software with a non-German keyboard – äüö Jun 20 '18 at 6:54
0

Aside from already mentioned points I would like to hint to the lexicographical order of umlauts.
In earlier days the german Umlauts were simlified (ä->a, ö->o, ü->u, ß->ss !, ...) for seeking in printed lexicons or name lists.
Then it split up as the umlauts should be treated as individual characters. The umlauts were treated as written with their simple representation (ä->ae,...)
Also possible: an ä was inserted in the alphabet after the a: a,ä, b, c, ... m, o, ö, p, ... , s, ß, t, u, ü ... z

Today with the usage of utf8 we don't need any replacements and characters can be entered and used as intended. There are sorting orders for each language (maybe with local differentiation), in the way that variants of the base character (like äáàâ) are sorted in line with the base character.

One problem occurs if you want to enter an umlaut and your keyboards does not has a key for it. Some of the character variants can be composed with dead keys like ´ or ´^´ if your keyboard driver supports it.
On virtual keyboards from smartphones you find the popup for alternative keys on longer touch. For the SMS like input with numeric keypad you often have enhanced lists of characters mapped to the single key (not only abc but abcäáàâ).

If you do not want to be dependend on the OS you can enhance your input with a popup for lists of special characters. here: lists of all available chracter variants. (Have you noticed the little keyboard in the input field of google translate if you try to enter a sentence in a language with special characters?)

Here we come back to the simplified input:
if you enter a word with (simple) umlaut replacements (just aou) in google it will find the word with umlauts and might ask if you meant the word with corrected writing. The results still would prefer the literal writing if any exist.
And don't forget: today the search engines can analyze the words and can identify the word stems so any word flexion would identify the word stem, which finds all possible flexions of the word stem.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.