In Harry Potter, the sentence
Mr. Dursley hummed as he picked out his most boring tie for work, ...
is translated as
Mr. Dursley summte vor sich hin und suchte sich für die Arbeit seine langweiligste Krawatte aus, ...
My question is: What does "vor sich hin" mean here?
vor sich hin can be used as an adverb to a series of verbs, with various possible different meanings, some of which carry a lot of connotation. It therefore impossible to give one translation that fits all cases.
The literal translation (or at least how I understand it, being a native speaker) is in front of oneself, which refers to the product of the verb. So literally translating your example, Mr. Dursley is creating humming in front of himself. Being less literal this usually signifies that nobody is interested in the product and not much effort is being put into it.
Depending on the verb and context, translations may be: without giving a thought; to oneself; along; away; without effort; without passion; lazily; slowly; without anybody caring.
Some examples other than your one:
Er lächelt vor sich hin.
He smiles just to himself, not caring if anybody notices or might be offended, maybe not even noticing himself. This is stereotypical of somebody who just experienced a major success and is not outright celebrating it at the moment.
Er lebte nur noch vor sich hin.
He is only doing the minimum requirements for sustaining the status quo, i.e., going to work, eating, sleeping, essential chores, but does not partake in any social or hobby activities, enjoy life or work towards some goal. This is stereotypical of somebody who just suffered a big loss (e.g., death of spouse).
Das Auto rostet vor sich hin.
The car is slowly rusting, but nobody cares, most likely, because it is out of use.
Er starrt vor sich hin.
He is staring ahead, but he is not looking at anything in particular and may not even notice what’s happening right in his view. He also does not react very well to anything else.
Das Projekt dümpelt nur noch vor sich hin.
The project is not cancelled or abandoned, but nobody is putting any more effort into it than necessary, so the progress is very slow.
Er hatte sich eine Staublunge geholt und starb nun zu Hause vor sich hin.
He caught himself a black lung and was now slowly dying away at home. Nobody cares very much for him and he lacks the motivation or energy to improve his situation. (Example thanks to Alexander.)
Er murmelte etwas vor sich hin.
He mumbled something under his breath. Nobody could possibly understand it and it most probably wasn’t intended to be understood.
Der Bach plätscherte vor sich hin.
The brook purled. The flow rate is at most medium for this place. The sound is clearly audible but not disturbing in any way. This is usally used to describe an idyllic setting.
"vor sich hin" is meant here as humming to himself in a way you can describe as a sort of mental absence, focussing on the selection of the tie not knowing about his humming
The vor sich hin means to oneself. For instance
Ein Liedchen vor sich hin summen
or
Sie lächelte vor sich hin.
The verb summen has usually an accusative object: einen Ton summen, eine Melodie summen. So, in your concrete example, in vor sich hin summen you supress this accusative object with the vor sich hin anymore and it means to hum to oneself:
Mr. Dursley hummed (to himself) as he picked out his most boring tie for work, ...
