Chart by non-XY-related data

I have a chart that I need to make based on this data:

Size, X-coor, Y-coor

This is all in one sheet and needs to stay that way. "Size" can be any number 1-10. I need to make a scatter chart of all the X,Y data where each Size is a different color. Within the Edit Series window, my X and Y values must be the data in my X and Y columns, so how do I go about accessing the Size field to set the color for each series? There will be an unknown number of (Size, X-coor, Y-coor) entries in the sheet, and the data CANNOT be sorted by Size (or other calculations will fail).

This is going to be pretty hard unless you can stick the y-coords in different series by "Size".

Like:

Not going to work. I am going to try a different approach with moving all the "size" data to individual sheets and then adding to this sheet a button to update from all the other tabs. That should produce the same calculations.