As shown in figure. When a spherical cavity (centred at O) of radius 1 is cut out of a uniform sphere of radius R (centred at C), the centre of mass of remaining (shaded part of sphere is at G, i.e., on the surface of the cavity. R can be determined by the equation:

Let us denote the uniform mass-density of the material by $$\rho$$.

We first write the masses of the two spheres explicitly. For the big sphere (centre at $$C$$, radius $$R$$)

$$M_{\text{big}}=\dfrac{4\pi}{3}\rho R^{3}$$

and for the cavity (centre at $$O$$, radius $$1$$)

$$M_{\text{cav}}=\dfrac{4\pi}{3}\rho \;1^{3}= \dfrac{4\pi}{3}\rho.$$

The mass of the remaining (shaded) solid is therefore

$$M_{\text{rem}}=M_{\text{big}}-M_{\text{cav}}=\dfrac{4\pi}{3}\rho\left(R^{3}-1\right).$$

Because the situation is completely symmetrical about the straight line $$CO$$, the centre of mass of the remainder must lie somewhere on this very line. We choose the origin of the one-dimensional coordinate axis at $$O$$ and measure every distance along $$OC$$ as positive. With this choice

$$x_{O}=0,\qquad x_{C}=d \;(=\;OC).$$

The point $$G$$, by statement of the problem, is situated on the surface of the cavity, that is exactly one unit away from $$O$$ along the same line, so

$$x_{G}=1.$$

Now we invoke the centre-of-mass formula for a composite body:

$$M_{\text{rem}}\;x_{G}=M_{\text{big}}\;x_{C}-M_{\text{cav}}\;x_{O}.$$

Substituting the known quantities, we have

$$\left[\dfrac{4\pi}{3}\rho\left(R^{3}-1\right)\right](1)=\left[\dfrac{4\pi}{3}\rho R^{3}\right]\,d-\left[\dfrac{4\pi}{3}\rho\right](0).$$

All common factors $$\dfrac{4\pi}{3}\rho$$ cancel immediately, leaving the simple relation

$$R^{3}-1=R^{3}\,d\qquad\Longrightarrow\qquad d=\dfrac{R^{3}-1}{R^{3}}=1-\dfrac{1}{R^{3}}.$$

Up to this point, every value of $$R>1$$ would merely give a different permissible separation $$d$$ of the two centres. However, the drawing that accompanies the question shows the cavity touching the outer surface of the large sphere at a single point. For two spheres of radii $$R$$ (big) and $$1$$ (cavity) this “tangential” condition means quite simply

$$OC+1=R\qquad\Longrightarrow\qquad d=R-1.$$

Putting this geometric condition together with the mass-balance condition just obtained we write

$$1-\dfrac{1}{R^{3}}=R-1.$$

Rearranging,

$$2-R-\dfrac{1}{R^{3}}=0\qquad\Longrightarrow\qquad 1=R^{3}(2-R).$$

We still have the factor $$R^{3}$$ on the right. It is convenient to eliminate it by first observing from an earlier step that

$$R^{3}-1=R^{3}(R-1).$$

Dividing both members of this identity by $$R-1$$ (with $$R\neq1$$) we get the well-known algebraic result

$$R^{3}=R^{2}+R+1.$$

Substituting this expression for $$R^{3}$$ in the equation $$1=R^{3}(2-R)$$ leads directly to

$$1=(R^{2}+R+1)(2-R).$$

Exactly the same algebra appears, term for term, in option A, viz.

$$(R^{2}+R+1)(2-R)=1.$$

Hence, the correct answer is Option A.