The probable reason behind all this is that the German language does not have many primary adjectives, i.e., adjectives that are neither loaned nor derived from some other word (like hässlich, einig, porös, chaotisch, machbar, golden, verwandt, kafkaesk, Berliner, …).
Wikipedia states their number to be ca. 250 and gives a list that looks quite complete to me.
Of those, only 18 are ending on a pure vowel letter (e.g., müde, neu, schlau, frei) and all of those either end on a voiceless e (schwa) or a diphthong.
There are also some adjectives who end on a vowel, but not on a pure vowel letter like froh, zäh or klar, whose ending turns into a consonant upon inflection (froher, zäher, klarer).
When a language loans a word, it has to assimilate it into its own grammar, which means especially that inflected forms have to be provided, which can be instantly recognised as what they are supposed to be. For nouns, this is usually not a problem, since we have a plethora of nouns already and thus many different inflection schemes, one of which can be applied to the newly loaned word. For verbs, we can usually just append an -en to form the infinitive and continue from there, e.g. downloaden. For adjectives, however, we have only one inflection scheme available, which mandates forms ending on -e, -er, -es, -en, -em and so on.
And this cannot be simply applied to everything, e.g., if we tried to inflect rosa like any other adjective, we would get:
ein rosa’er Kasten, eine rosa’e Blüte, ein rosa’es Kleid, auf rosa’em Untergrund, … [the apostrophe indicates that no diphthongisation or something similar happens]
Though these are pronounceable, the a-e collision does not appear in any other German word and therefore sounds weird to German ears and is avoided.
The most common fix to this (and what most native speakers will actually do), is to insert an n for every inflected form, i.e.:
ein rosaner Kasten, eine rosane Blüte, ein rosanes Kleid, auf rosanem Untergrund, … – But: Das Haus ist rosa.
However with most adjectives ending on consonants (e.g. türkis or pink), there is no such or a similar problem.
Now, for reasons that are beyond me (see this question, some language authorities¹ in the past decided, that some adjectives (especially colour adjectives) should not be inflected at all.
While some native speakers will actually do so for lila, rosa, extra, prima, super and klasse, I have never seen anybody do so for any other adjectives (e.g., orange, pink or türkis) and most people will consider it a mistake, if you use them attributively but not inflected. Also I would not know what problem should arise from inflecting these adjectives. There are also some rarer adjectives, which are not used attributively at all in my experience, but are suffixed with -farben or replaced by an entirely different adjective:
Das Kleid ist khaki. Ein khakifarbenes Kleid. (Also: Das Kleid ist khakifarben.)
However, for most such rare colour adjectives, I would stumble over them if somebody used them without -farben, since it would take me some time to identify them as what they are.
Therefore I would only use them with -farben (if at all) and would also recommend this – except for audiences that deal with these colours on a daily basis.
Finally, going back to the actual question, I would recommend the following (if you want to be understood and need not be correct for correctness’s sake according to some weird authority):
- Inflect normally:
- Any non-loaned adjective.
- Any relatively common loaned adjective, if it ends on schwa or does not end on another vowel in both, pronunciation and spelling. For example: orange, beige, pink, violett, purpurn, türkis, prüde. Note, that for orange appending an n before inflecting is not uncommon: »Ein orange(ne)s Haus.«
- Do not inflect:
- Non-loaned adjectives ending on -lei or -tel (like achtel) and adjectives ending on -er derived from geographical names.
- The following six words: rosa, lila, prima, extra, super, klasse. For rosa and lila, adding an n for inflection is also acceptable in my opinion. (I went through Canoo’s list of non-inflected adjectives multiple times and did not see any other word that falls into this category for me.)
- Do not use attributively (or do not use at all):
- All other loaned adjectives ending on a vowel – use alternatives instead, such as: khakifarben, heterosexuell
- Some non-loaned adjectives like allein.
¹ That is non-official language authorities. There has never been an official authority on German grammar.