We consider the action of the mapping class group of a compact  surface S of genus g>1 on the character variety of the fundamental group of S  in a connected semisimple real Lie group G. We identify the fixed points of the action of a finite subgroup  Γ of the modular group on the character variety, in terms of G-Higgs bundles equipped with a Γ-equivariant structure on a Riemann surface X=(S,J), where J is an element in the Teichm\"uller space of S for which Γ is included in the group of automorphisms of X, whose existence is guaranteed by Kerckhoff's solution of the Nielsen realization problem. The Γ-equivariant G-Higgs bundles are in turn in correspondence with parabolic Higgs bundles on Y=X/Γ, where the weights on the parabolic points are determined by the Γ-equivariant structure. This generalizes work of Nasatyr & Steer for G=SL(2,R), and Andersen & Grove and Furuta & Steer for G=SU(n).