Spectral refinement on quasi-diagonal matrices