I saw this sentence and don't understand why Schülerin is dative but not accusative. helfen is transitive verb so the object must be direct object so why it is dative?
Willst du der Schülerin helfen?
There are a few definitions and assumptions in your question that are going haywire and maybe causing the confusion you have.
While it is technically not wrong to speak of direct and indirect objects in German, it is not the usual practice. Neither dative nor accusative objects require any indirect attachment to the verb — much unlike in English or French, and therefore they could technically both be called direct objects. However, due to similarities with other European languages, direct object can be understood to mean accusative object while indirect object can mean dative object.
I would advise you to use the case names for German.
There are two definitions of transitive verbs out there. School grammar usually assumes a transitive verb immediately if and only if the verb has an accusative object which, if the sentence is transformed into the passive voice, becomes a subject.
Linguistically, a verb is called transitive if it has two required arguments, the first being the subject and the second being an object. So in the example sentence of I help him we have the verb to help and its two arguments I and him. If we leave out either argument, we get an invalid sentence proving both arguments are obligatory. The same is true for the sentence Ich helfe dir. This means that helfen also has two required arguments and can therefore be called transitive.
Transitive verbs must have a direct object
Using the school definition of transitivity and the assumption direct object = accusative this might be correct (a.k.a. I’m too lazy to go looking for a counterexample). Using the linguistic definition it is wrong for German if the same object assumption is made. It might work for English and French but different languages have different rules. (And if you want to remember just one thing from this post, remember the second half of that sentence.)
Concerning your example:
Jemandem helfen is a verb that requires a dative object and only a dative object. Using an accusative object for the verb is wrong, and in the passive voice the object remains dative.
Da werden Sie geholfen.(quote from a TV advertising campaign)
Da wird Ihnen geholfen.
Helfen can now be classified:
by school grammar as an intransitive verb with an obligatory dative argument
linguistically as a transitive verb that requires a nominative and a dative argument.
In neither case would helfen ever take accusative.
Helfen is always dative. It's just a rule. It's always 'Hilf mir!' never 'Hilf mich.' Just one of the things you have to learn with the language. Similar to the way it's always 'Sag mir' but 'Frag mich.' Doesn't make any sense, but that's the way it is.
Here's a good list of dative, German verbs. http://german.about.com/library/verbs/blverb_dativ.htm