Hi Balu, the mistake you might have done is after taking the side as √2x you are finding the value of x and checking for it in the options. But the question asked is the side of the octagon. For that you have to multiply your answer by √2 and your answer will be Option B. So please check your answer again.

Hi Akshaya
The octagon formed by cutting the sides is a regular octagon, so all its sides must be equal. Let the length of each side be x.
Cconsider the leftmost top corner triangle (Cutaway portion) The two sides of this triangle are equal. Moreover its a right angled triangle. So we can say that it is an isosceles right triangle. The hypotenuse of this triangle is equal to the side of the octagon, so we have
y^2 + y^2 = x^2
y^2 = x^2/2
=> y = x/rt2
Now 2y + x = 2 (Given that side of the square is 2)
So x + 2x/rt2 = 2
x(1+rt2) = 2
x = 2/(1+rt 2)
Hope this makes it clear to you.

If length of corner cut is x, Length of the side of the octagon is x*sqrt(2). So, x will be option A and x*sqrt(2) will be option B which is the correct answer.

The small triangles are isosceles right angled triangles with hypotenuse = x. Hence the length of the side will be x/sqroot(2)