According to the wiki page:
[x] occurs after /uː oː/ (for instance in Buch [buːx] 'book') and [χ] after /a aː/ (for instance in Bach [baχ] 'brook'), while either [x] or [χ] may occur after /ʊ ɔ aʊ̯/, with [χ] predominating.
When pronouncing the [χ] sound I end up trilling, otherwise no sound really comes out. So, it ends up sounding like a voiceless 'r'. Is this a mistake and would native speakers notice, or does it really not matter?