“Don’t say it again. We don’t have to say it. I get it,” the House speaker said testily when asked about the comments