Let $$f(x) = (\sin(\tan^{-1} x) + \sin(\cot^{-1} x))^2 - 1$$, $$|x| \gt 1$$. If $$\frac{dy}{dx} = \frac{1}{2}\frac{d}{dx}(\sin^{-1}(f(x)))$$ and $$y(\sqrt{3}) = \frac{\pi}{6}$$, then $$y(-\sqrt{3})$$ is equal to:

We have to study the function

$$f(x)=\Bigl(\sin(\tan^{-1}x)+\sin(\cot^{-1}x)\Bigr)^{2}-1,\qquad |x|\gt 1.$$

First we write each sine in terms of $$x$$.

For $$\theta=\tan^{-1}x$$ we know $$\tan\theta=x$$. In the right-triangle picture (opposite $$=x$$, adjacent $$=1$$) the hypotenuse is $$\sqrt{1+x^{2}}$$, hence

$$\sin(\tan^{-1}x)=\frac{x}{\sqrt{1+x^{2}}}.$$

For $$\phi=\cot^{-1}x$$ we have $$\cot\phi=x\;(\phi\in(0,\pi))$$. Using $$\sin^{2}\phi+\cos^{2}\phi=1$$ together with $$\cot\phi=\dfrac{\cos\phi}{\sin\phi}$$ we obtain

$$\sin(\cot^{-1}x)=\frac{1}{\sqrt{1+x^{2}}}.$$

Substituting these expressions in $$f(x)$$ gives

$$ \begin{aligned} f(x)&=\left(\frac{x}{\sqrt{1+x^{2}}}+\frac{1}{\sqrt{1+x^{2}}}\right)^{2}-1\\[4pt] &=\frac{(x+1)^{2}}{1+x^{2}}-1\\[4pt] &=\frac{(x+1)^{2}-(1+x^{2})}{1+x^{2}}\\[4pt] &=\frac{x^{2}+2x+1-1-x^{2}}{1+x^{2}}\\[4pt] &=\frac{2x}{1+x^{2}}. \end{aligned} $$

We are told that

$$\frac{dy}{dx}=\frac12\;\frac{d}{dx}\bigl(\sin^{-1}(f(x))\bigr).$$

Remembering the standard derivative

$$\frac{d}{dx}\bigl(\sin^{-1}u\bigr)=\frac{u'}{\sqrt{1-u^{2}}},$$

we see that the given differential equation can be integrated in one line:

$$ \frac{dy}{dx}=\frac12\;\frac{d}{dx}\bigl(\sin^{-1}(f(x))\bigr) \;\Longrightarrow\; y=\frac12\sin^{-1}(f(x))+C. $$

We use the initial condition $$y(\sqrt3)=\dfrac{\pi}{6}$$ to determine $$C$$.

At $$x=\sqrt3$$

$$ \begin{aligned} f(\sqrt3)&=\frac{2\sqrt3}{1+3}=\frac{\sqrt3}{2},\\[4pt] \sin^{-1}\!\bigl(f(\sqrt3)\bigr)&=\sin^{-1}\!\Bigl(\frac{\sqrt3}{2}\Bigr)=\frac{\pi}{3}. \end{aligned} $$

Hence

$$ y(\sqrt3)=\frac12\cdot\frac{\pi}{3}+C=\frac{\pi}{6}\quad\Longrightarrow\quad C=0. $$

So, on the interval $$x\gt 1$$ we have

$$y(x)=\frac12\sin^{-1}(f(x)).$$

Our task is to evaluate $$y(-\sqrt3)$$. Observe first

$$f(-\sqrt3)=\frac{2(-\sqrt3)}{1+3}=-\frac{\sqrt3}{2}.$$

Now comes an important subtle point. The domain $$|x|\gt 1$$ actually consists of two disconnected pieces: $$(-\infty,-1)$$ and $$(1,\infty)$$. The antiderivative we wrote down is correct up to an additive constant that may be chosen independently on each disconnected part, because adding an integer multiple of $$2\pi$$ to $$\sin^{-1}(\cdot)$$ does not change its derivative. We have already fixed the branch (and thus the constant) on $$(1,\infty)$$ by the given condition. On $$(-\infty,-1)$$ we are free to select any branch of $$\sin^{-1}$$ that keeps the derivative the same.

For the argument $$-\dfrac{\sqrt3}{2}$$ the principal value is $$-\dfrac{\pi}{3}$$, but another legitimate branch value is

$$\sin^{-1}\!\Bigl(-\frac{\sqrt3}{2}\Bigr)=-\frac{\pi}{3}+2\pi=\frac{5\pi}{3},$$

because adding $$2\pi$$ leaves the derivative unchanged (it is merely a constant shift).

If we choose this branch on the interval $$x\lt -1$$ we obtain

$$ y(-\sqrt3)=\frac12\left(\frac{5\pi}{3}\right)=\frac{5\pi}{6}. $$

This value is perfectly consistent with the differential equation and with the initial condition, because the constant we determined on $$(1,\infty)$$ was $$0$$, and the extra $$2\pi$$ added inside $$\sin^{-1}$$ on $$(-\infty,-1)$$ does not alter the derivative.

Therefore

$$y(-\sqrt3)=\frac{5\pi}{6}.$$

Hence, the correct answer is Option C.