First of all, as it shown in the link given by rogermue or at this page by canoo.net, possible candidates for such verbs can be made out by their prefix: Only durch, über, um, unter, wider and wieder can lead to verbs that are both separable and inseparable. Note that both sources list wieder as an always-separable prefix, which is wrong by counter-example: wiederholen (separable, to bring back) and wiederholen (inseparable, to repeat).
Second, as they also point out there, separable verbs have the stress on the prefix, while inseparable verbs have an unstressed prefix.
So (first rule, no need for a thumb): Memorize how to pronounce these verbs in their different meaning, and you get the sep./insep. distinction for granted.
Something more closer to a rule of thumb, less need for memorizing: Estimate which part of the verb is more important for the composed meaning. That’s the part that is stressed, and then again you also know whether it’s a separable verb or not.
Some examples (don’t take my attempts of explanation too literally, please)
durchlaufen (separable) – It runs, yes; but it runs through.
durchlaufen (inseparable) – They run and run and run, and eventually their are through.
umschreiben (separable) – It’s writing with the purpose of changing something (um signifies some sort of change here)
umschreiben (inseparable) – Very similar to the always inseparable beschreiben: The action is in the focus, and um qualifies it as going around (compare with to circumscribe) looking from different angles to get the picture.
umfahren (separable) – Something went down, and it is this result that matters. That this was reached by some driving activity is secondary.
umfahren (inseparable) – Again, the action of driving is in the focus, and um is used to qualify it as going around.
As for transitive vs. intransitive - I don't think that this is of great help here. E.g. both versions of umfahren are transitive, and you will likely find examples for all four possible combinations of separable/inseparable with transitive/intransitive.