"Im Hinterkopf" would be the most appropriate translation of "in the back of my mind".
The word "Hintergedanken" may refer to a concealed intention you have while stating a different. Also it can be related to indirect consequences caused by your actions which you are aware of. Example:
Mit dem Hintergedanken, dass jemand ausrutschen könnte, legte ich
Banenenschalen auf den Gehweg.
However, "[...] jedoch liebte ich sie so oder so." would be more appropriate because "sowieso" is mainly used for confirming statements which are already acknowledged as legit or are legit regardless if the discussed subject is.