We are given positive x and y. Let's go in the first loop.
Let's do some method refactoring for better understanding:
y0 = x*x+y;
x0 = x*x+y0;
y1 = sqrt(x0+(y0/labs(y0))*(-labs(y0)));
for (j = 1; j <= 2*y1; j++)
    x0 = x0-y1;
x = x0;
y = y1;
Let's go through the lines:
y0 = x*x + y
Next:
x0 = x*x + y0 = 2*x*x + y
As y0 > 0 (x, y are positive) => y0/labs(y0) = 1
So y1 = sqrt(x0+(y0/labs(y0))*(-labs(y0))) = sqrt(x0-labs(y0)) = sqrt(2*x*x + y - (x*x + y)) = sqrt(x*x) = x
Next 2 lines equals to this:
x0 = x0 - 2*y1*y1 = 2*x*x + y - 2*x*x = y
So, x=y and y=x. x and y are swapped. After that you need to count amount of swaps and print appropriate answer

Thank you very much.