Button changing size and font size

Hi!

I have made a toggle button to change language between Swedish and English. Although, every time I click it, the font becomes smaller and smaller. I have tried formatting it by changing to any of the alternatives "Move and size with cells/Move but don't size with cells/Don't move or size with cells" but it makes no difference whatsoever.

Any suggestions is appreciated!