From a sphere of mass M and radius R, a smaller sphere of radius $$\frac{R}{2}$$ is carved out such that the cavity made in the original sphere is between its centre and the periphery (see the figure). For the configuration in the figure where the distance between the centre of the original sphere and the removed sphere is 3R, the gravitational force between the two spheres is:

The mass of the removed sphere ($$m$$) is $$m = M \times \left( \frac{R/2}{R} \right)^3 = \frac{M}{8}$$

Center of the Original Sphere ($$O$$): $$(0,0)$$

Center of the Cavity ($$C$$): Since the cavity is carved between the center and periphery, its center is at $$(R/2, 0)$$.

Center of the External removed sphere ($$S$$): The distance between the center of the original sphere and the removed sphere is $$3R$$. Its position is $$(3R, 0)$$.

Distance from $$O$$ to $$S$$ ($$r_1$$) = $$3R$$. Distance from $$C$$ to $$S$$ ($$r_2$$) = $$3R - R/2 = 2.5R = \frac{5R}{2}$$

$$F_{net} = F_{original} - F_{cavity}$$

$$F_{net} = G \frac{M \cdot m}{r_1^2} - G \frac{m \cdot m}{r_2^2}$$

$$F_{net} = G \frac{M(M/8)}{(3R)^2} - G \frac{(M/8)(M/8)}{(5R/2)^2}$$

$$F_{net} = \frac{GM^2}{8 \cdot 9R^2} - \frac{GM^2}{64 \cdot \frac{25R^2}{4}}$$

$$F_{net} = \frac{GM^2}{72R^2} - \frac{GM^2}{16 \cdot 25R^2} = \frac{GM^2}{72R^2} - \frac{GM^2}{400R^2}$$

$$F_{net} = \frac{50GM^2}{3600R^2} - \frac{9GM^2}{3600R^2} = \frac{41GM^2}{3600R^2}$$