You're right. In the first place, it is comparable to "I like green olives better than black ones".
However, it would be possible to emphasise "ich mag dich" in such a way that it means "I have feelings for you", "I love you", etc.
When speaking directly to a person, it would probably be better to stress why you like the person, or what exactly you like:
Ich mag dich, weil du ein guter Zuhörer bist.
Just telling someone "ich mag dich" would perhaps imply that more romantic feelings are meant. However, this depends very much on the person themselves and the extent to which they are able to talk about their own (romantic) feelings in a clear-cut way.
You may find this Q&A helpful as well: What are the differences in meaning among "mögen", "stehen auf", "liebhaben", and "lieben"?