Tying Chart Bars to specific cells in source data

Creating an excel chart template with a bar chart.

Changed the colors of several bars and wish to keep the color tied to the rows in the source.

However when I sort or hide some of the rows, the colored bars move to other data points.

The data points on the graph seem to be tied to the order in the source data rather than the specific cells.

If anyone knows a way around this, would appreciate it. Having to make the source changes and then changing the color would be a great deal of manual work.