“I mean, we used to talk a lot more,” the actress told Andy Cohen on his SiriusXM show