how wrong? Would a native understand it? Is it irritating the ears of a native speaker? To what degree?
It would sound totally wrong, but we would understand it just fine. It would clearly show that you're a non-native speaker.
Whether it is irritating depends on whether the listener is generally annoyed with non-natives making errors. I have contact to many non-natives, and there are much worse problems (with respect to irritation/trouble understanding) than your example.
An equivalently broken english sentence would be "He has upbrought something" instead of "He brought something up".