For plural nouns, like das Gemüse (the vegetable (sg) and the vegetables (pl) are both das Gemüse), why isn't it die Gemüse? And when an adjective is placed before it in accusative, neuter nouns take an -es ending, so frisches gemüse. If it's plural accusative, shouldn't it be frische gemüse?
I propose the following impromptu explanation:
Compare das Gemüse to das Gekröse (tripe, mesentery). These are words that do not that much refer to single, individual objects; rather they seem to refer to a plurality of things as one. Or the other way round: they are used for things that consist of a mess of other things (or are perceived as such). Similar also das Getöse (many noises form one Getöse), or many others such as das Gestöhne, das Gemurmel, das Gemeckere, das Gesindel, das Gesudel, das Gedöns and so on.
Gemüse is initially a word for a mess of various edibles; only in a second step it is used for the individual edibles.
For your case endings:
Das Gemüse is still singular, so it is
Ich kaufe frisches Gemüse.
You could theoretically also say
Ich kaufe frische Gemüse.
but then you are treating the word indeed as a plural, which is simply pretty unusual (although technically possible without leaving the area of well-formed sentences).
Note that this is possible with Gemüse but not with das Gestöhne, das Gesindel, das Gesudel, etc. You cannot have "viele Gestöhne" (plural), only viel Gestöhne (singular). I suppose this is because with das Gemüse you have relatively often to deal with the single items (a tomato, a cucumber, potatoes, leeks, etc.), whereas the other Ge-things occur in your life almost exclusively as bulk.