Charts Changing Color When Using Slicers

Hello all,

I am in a bit of a pickle. I have created some charts for a dashboard that I am developing with the usage of slicers - so it's dynamic when looking at specific values. What I have noticed is that the colors within the chart change to different categories when one of them are not available. I will have pictures below to show what I am talking about. Would there be anyway to override this? I would like to set one category a specific color and not have it to change to a different color when one of the race variables are missing from using the slicers to filter. Would I need to do something related to VBA?

unfiltered:
Diversity First.png


When filtered:
Diversity Filtered.png