I don't think you can accomplish exactly what you describe. There might be "pay" upgrades to Solver that can do this, but I am not certain.

You don't give any details of what you algorithm is doing or how it is coded. My first thought is to rewrite your VBA procedures so that they are "User-Defined Functions" (function procedures that can be called from a spreadsheet cell like a regular Excel function). These can work very well as part of Solver models.

Of course, if you understand the functions well enough, you can program your own optimization algorithms (Newton Raphson or bisection or whatever) into VBA.