Optional 5th x,y value messes up the graph's RSQ and Linear Formula

I have a graph based on 4 x,y values, with an optional 5th value
The 5th x,y cells have a function in each cell that is basically

=IFERROR(formula I'm using,"")

In other words, if the formula errors, "" is returned.

I made cells underneath which calculate the correct slope, RSQ, and Intercept values, but the chart will only calculate correctly if it is given cells for x,y 1-4. If the 5th option is included, the graph only works if there is actual numbers in the 5th positions. Is there a way to fix this?

The chart on the left is set up with 5 points, the chart on the right is set up with only the 4 active points.

Capture.PNG