A box open from top is made from a rectangular sheet of dimension $$a \times b$$ by cutting squares each of side $$x$$ from each of the four corners and folding up the flaps. If the volume of the box is maximum, then $$x$$ is equal to:

Let us begin by describing the geometry of the situation. From a rectangle of length $$a$$ and breadth $$b$$, we remove four identical squares, each of side $$x$$, one from every corner. After removing these squares we fold the resulting flaps upward to obtain an open-top box. In this box

length $$=a-2x$$,   breadth $$=b-2x$$,   height $$=x$$.

Therefore the volume $$V$$ of the box, expressed as a function of $$x$$, is

$$V(x)=x\,(a-2x)\,(b-2x).$$

We expand the product so that differentiation becomes simple:

$$\begin{aligned} V(x)&=x\bigl[(a-2x)(b-2x)\bigr] \\ &=x\bigl[ab-2ax-2bx+4x^2\bigr] \\ &=abx-2a x^{2}-2b x^{2}+4x^{3} \\ &=abx-2(a+b)x^{2}+4x^{3}. \end{aligned}$$

To find the value of $$x$$ that gives the maximum volume, we differentiate $$V$$ with respect to $$x$$ and set the derivative to zero. The formula we use is: “The maximum (or minimum) of a differentiable function occurs where its derivative is zero.”

First derivative:

$$\begin{aligned} \frac{dV}{dx}&=\frac{d}{dx}\bigl[abx-2(a+b)x^{2}+4x^{3}\bigr] \\ &=ab-4(a+b)x+12x^{2}. \end{aligned}$$

Setting $$\dfrac{dV}{dx}=0$$ for extreme values gives

$$ab-4(a+b)x+12x^{2}=0.$$

We rewrite this quadratic in the standard form $$Ax^{2}+Bx+C=0$$:

$$12x^{2}-4(a+b)x+ab=0.$$

Dividing every term by $$4$$ to simplify, we obtain

$$3x^{2}-(a+b)x+\frac{ab}{4}=0.$$

Now we apply the quadratic‐formula statement: “For $$\alpha x^{2}+\beta x+\gamma=0$$, the roots are $$x=\dfrac{-\beta\pm\sqrt{\beta^{2}-4\alpha\gamma}}{2\alpha}.$$” Here $$\alpha=3$$, $$\beta=-(a+b)$$ and $$\gamma=\dfrac{ab}{4}$$. Substituting, we get

$$\begin{aligned} x&=\frac{(a+b)\pm\sqrt{(a+b)^{2}-3ab}}{6}. \end{aligned}$$

We simplify the discriminant inside the square root:

$$\begin{aligned} (a+b)^{2}-3ab&=a^{2}+2ab+b^{2}-3ab\\ &=a^{2}+b^{2}-ab. \end{aligned}$$

Hence the critical values of $$x$$ are

$$x=\frac{\,a+b\pm\sqrt{a^{2}+b^{2}-ab}\,}{6}.$$

Next, we decide which of the two roots is admissible. Because the squares we cut out must actually fit inside the original rectangle, we need $$0<x<\dfrac{a}{2}$$ and $$0<x<\dfrac{b}{2}$$. The expression with the plus sign, $$\dfrac{a+b+\sqrt{a^{2}+b^{2}-ab}}{6}$$, is evidently larger than $$\dfrac{a+b}{6}$$ and in many practical configurations exceeds either $$\dfrac{a}{2}$$ or $$\dfrac{b}{2}$$, violating the geometric constraint. The expression with the minus sign, however, is always the smaller positive root and therefore satisfies the condition of fitting inside both $$\dfrac{a}{2}$$ and $$\dfrac{b}{2}.$$

Thus the only feasible choice is

$$x=\frac{a+b-\sqrt{a^{2}+b^{2}-ab}}{6}.$$

We still have to confirm that this root corresponds to a maximum volume. For that we examine the second derivative:

$$\frac{d^{2}V}{dx^{2}}=\frac{d}{dx}\bigl[ab-4(a+b)x+12x^{2}\bigr]=-4(a+b)+24x.$$

Substituting our admissible value of $$x$$, which is smaller than $$\dfrac{a+b}{6}$$, makes $$24x<4(a+b)$$, so

$$\frac{d^{2}V}{dx^{2}}=-4(a+b)+24x<0.$$

A negative second derivative at the critical point tells us that the volume indeed attains a maximum there.

Therefore, the value of $$x$$ that maximizes the volume of the open-top box is

$$x=\frac{a+b-\sqrt{a^{2}+b^{2}-ab}}{6}.$$

Hence, the correct answer is Option C.