Custom formats in charts

I'm creating a chart that needs to change the X axis based on the type of data that is pulled - Currency or Number or Percentage. I'm using conditional formatting to determine the cell format and that is working fine, but the chart is staying on currency regardless of which option is selected.

In the chart options, the "Linked to Source" box is checked and a custom format has been applied by Excel: _($* #,##0_);_($* (#,##0);_($* "-"??_);_(@_)