The letter combination "oe" is pronounced like "ö" in the name of the city Oelde, but like "o" in the name of the city Coesfeld. Is there an explanation of this difference?
The explanation is that Coesfeld is situated in a region influenced by Middle Dutch. The e is a lengthening vocal in this case ("Dehnungs-e").
This is similar to Dutch. In German, this lengthening e is now only present in names. It occurs at the lower Rhine river due to the influence of Middle Dutch. Other names I know are Soest and Fuest, but the linked Wikipedia article has more examples.
The oe in Oelde is just indicating an Umlaut, as it is common in German. It seems that Oelde has not been under this influence of Middle Dutch. I don't know where the exact border of the phenomenon of the Dehnungs-e is. Probably, the town of Oelde is across that border.
Names of places are often centuries old and have not been adapted to modern spellings. For a bit of fun, consider the place names Voerde, Itzehoe and Buchloe where the oe is pronounced as /øː/, /oː/ and /oə/, respectively.
Buchloe is explained simply by there being multiple syllables colliding with the penultimate one ending in o and the final one being just a shwa. Some other places write this with an h to separate the o and e (e.g. Eschenlohe; the lohe being pronounced exactly like Buchloe’s loe), but the h is silent so it makes phonetic sense to leave it out.
Voerde and the likes are due to the entirely valid replacement of umlaut letters by letter + e. While this normally does not happen in German words (as umlauts exist), exceptions can be found and are more common in names whose spelling has been fixed for over a century. Arguably, having this letter + e combination at the beginning of the name (Oelde, Uelzen, Aerzen) makes even more sense as capital umlaut letters were uncommon when German was still primarily written in blackletter typefaces; This also applied to common words, so one would have written Oel and Aepfel.
Finally, in some cases an e can simply be a length marker. Most commonly, e as a length marker is found after i (or when e is doubled for e), but it can occur after all other vowels too. This usage strikes me as more typical in the North and West but that is no guarantee that you won’t run into them elsewhere.
There is no way to guess which variant you have a priori, so you will have to rely on a written pronunciation guide or listening to people who know how to pronounce it.