I can only drink clear rum and tequila.

I can drink clear rum and tequila without a problem, but colored , darker rum and tequila make me feel unwell. What’s the difference? Is it just food coloring or dye?

Also, I took a break from drinking (over a year) for health reasons, and I recently started drinking a little again, if that makes a difference.