“It’s because when you do get involved in it, no matter how you feel, someone passionately disagrees with you,” the rapper says