Orthogonality of invariant vectors

Let G be a finite group with given subgroups H and K. Let π be an irreducible complex representation of G such that its space of H-invariant vectors as well as the space of K-invariant vectors are both one dimensional. Let vH (resp. vK) denote an H-invariant (resp. K-invariant) vector of unit norm in a given G-invariant inner product 〈,〉π on π. Our interest is in computing the square of the absolute value of 〈vH,vK〉π. This is the correlation constant c(π;H,K) defined by Gross [3]. In this paper, we study this question for G=GL2(Fq), where Fq is the finite field of q=pm elements of odd characteristic p, H is its split torus and K is a non-split torus. The first main theorem of this paper gives an explicit formula for |〈vH,vK〉π|2 modulo p. The key idea here is to analyse the mod p reduction of π. The second main theorem relates the behaviour of 〈vH,vK〉π under Shintani base change and gives a sufficient condition for 〈vH,vK〉π to vanish for an irreducible representation π=BC(τ) of PGL2(E), in terms of the epsilon factor of the base changing representation τ of PGL2(F), where E/F is a finite extension of finite fields.