Both are fine
Ich ging wählen.
as well as
Ich war wählen.
With both constructions, the location were the action takes place comes into play - in this example, the polling station. With "gehen", the movement to the location is emphasized a little more, with "sein", the stay at the location is emphasized a little more. But that doesn't give a big difference in meaning.
Especially the construction of "sein" plus infinitive verb is typically used for actions that are bound to a specific location, or typically take place at a specific location. You might consider
Ich war wählen.
as a bit of a shortened
Ich war im Wahllokal und habe dort gewählt.
or
Ich war an dem Ort, wo man wählt (und habe dort das getan, was man dort eben tut).
The same principle can be used with a lot of verbs, for example
Ich war einkaufen. (typically done at a grocery store or the like)
Ich war reiten. (typically done at a riding hall or similar)
Ich war schwimmen. (not typically done in your living room ;) )
When used in past tense, this construction has the connotation of a finished action: "Ich war einkaufen" implies that I left home, went to the store, did my shopping and went back again.
It can also be used in present tense. Then, it has a subtext of business or occupation. "Ich bin (gerade) einkaufen" often implies something like "... deshalb kann ich leider gerade nicht um Dein Anliegen kümmern."