On the rational subset problem for groups