Expansion of a Determinant Along a Row

Statement I

Put $$a=\cos\alpha,\;b=\cos\beta,\;c=\cos\gamma$$.
The first determinant becomes

$$\Delta_1=\begin{vmatrix}1&a&b\\a&1&c\\b&c&1\end{vmatrix}$$

For any symmetric matrix of this type the standard expansion gives
$$\Delta_1=1+2abc-a^{2}-b^{2}-c^{2}\;.\;-(1)$$

The second determinant is

$$\Delta_2=\begin{vmatrix}0&a&b\\a&0&c\\b&c&0\end{vmatrix}$$

Expanding along the first row:

$$\Delta_2=0\cdot\Bigl|\begin{smallmatrix}0&c\\c&0\end{smallmatrix}\Bigr| -a\Bigl|\begin{smallmatrix}a&c\\b&0\end{smallmatrix}\Bigr| +b\Bigl|\begin{smallmatrix}a&0\\b&c\end{smallmatrix}\Bigr| = -a(-bc)+b(ac)=2abc\;.\;-(2)$$

Given $$\Delta_1=\Delta_2$$, substitute $$(1)$$ and $$(2)$$:

$$1+2abc-a^{2}-b^{2}-c^{2}=2abc \;\;\Longrightarrow\;\;1-a^{2}-b^{2}-c^{2}=0$$

Hence $$\cos^{2}\alpha+\cos^{2}\beta+\cos^{2}\gamma=a^{2}+b^{2}+c^{2}=1$$, not $$\frac{3}{2}$$.
So Statement I is false.

Statement II

Let

$$D(x)=\begin{vmatrix}x^{2}+x & x+1 & x-2\\ 2x^{2}+3x-1 & 3x & 3x-3\\ x^{2}+2x+3 & 2x-1 & 2x-1\end{vmatrix}$$

Use the cofactor rule $$D=A_1E_1-B_1E_2+C_1E_3$$ with
$$(A_1,B_1,C_1)=(x^{2}+x,\;x+1,\;x-2).$$

$$E_1=\begin{vmatrix}0&3x-3\\2x-1&0\end{vmatrix}=3(2x-1)=6x-3$$
$$\;\;\Longrightarrow\;A_1E_1=(x^{2}+x)(6x-3)=6x^{3}+3x^{2}-3x$$

$$E_2=\begin{vmatrix}2x^{2}+3x-1&3x-3\\x^{2}+2x+3&2x-1\end{vmatrix} =x^{3}+x^{2}-8x+10$$
$$\;\;\Longrightarrow\;B_1E_2=(x+1)(x^{3}+x^{2}-8x+10) =x^{4}+2x^{3}-7x^{2}+2x+10$$

$$E_3=\begin{vmatrix}2x^{2}+3x-1&3x\\x^{2}+2x+3&2x-1\end{vmatrix} =x^{3}-2x^{2}-14x+1$$
$$\;\;\Longrightarrow\;C_1E_3=(x-2)(x^{3}-2x^{2}-14x+1) =x^{4}-4x^{3}-10x^{2}+29x-2$$

Therefore

$$D(x)=\bigl(6x^{3}+3x^{2}-3x\bigr) -\bigl(x^{4}+2x^{3}-7x^{2}+2x+10\bigr) +\bigl(x^{4}-4x^{3}-10x^{2}+29x-2\bigr)$$

Simplifying term by term:

$$D(x)=24x-12=12(2x-1)$$

Thus $$p=24,\;q=-12$$ and
$$p^{2}=24^{2}=576,\quad 196q^{2}=196\cdot144=28224\neq576.$$

Hence Statement II is also false.

Both statements are false  →  Option B.

Let $$t=\frac{\sin x}{x}\,,$$ so that $$\displaystyle\lim_{x\to 0}t=1$$ because $$\frac{\sin x}{x}\to 1$$ as $$x\to 0$$.

With this notation the determinant becomes
$$f(x)=\left|\begin{matrix}a+t & 1 & b\\ a & 1+t & b\\ a & 1 & b+t\end{matrix}\right|.$$

Perform the row operation $$R_1\rightarrow R_1-R_2$$ (subtract the second row from the first):
$$f(x)=\left|\begin{matrix}t & -t & 0\\ a & 1+t & b\\ a & 1 & b+t\end{matrix}\right|.$$

Factor out $$t$$ from the first row:
$$f(x)=t\;\left|\begin{matrix}1 & -1 & 0\\ a & 1+t & b\\ a & 1 & b+t\end{matrix}\right|.$$

Call the remaining determinant $$D(t)$$, i.e.
$$D(t)=\left|\begin{matrix}1 & -1 & 0\\ a & 1+t & b\\ a & 1 & b+t\end{matrix}\right|.$$

Expand $$D(t)$$ along the first row:
$$\begin{aligned} D(t)=&\;1\;\left|\begin{matrix}1+t & b\\ 1 & b+t\end{matrix}\right| \;-\;(-1)\;\left|\begin{matrix}a & b\\ a & b+t\end{matrix}\right| \;+\;0\\[4pt] =&\;(1+t)(b+t)-b + a\,t. \end{aligned}$$

Simplify the expression:
$$(1+t)(b+t)-b = (1+t)(b+t)-b = (1+t)(b+t)-b.$$
Therefore
$$D(t)=(1+t)(b+t)-b+a\,t.$$

Now take the limit as $$x\to 0$$, i.e. $$t\to 1$$:
$$\begin{aligned} \lim_{x\to 0}f(x) &=\lim_{t\to 1}\;t\,D(t)\\ &=1\;\bigl[(1+1)(b+1)-b+a\cdot 1\bigr]\\ &=(2)(b+1)-b+a\\ &=2b+2-b+a\\ &=a+b+2. \end{aligned}$$

Hence
$$\lambda+\mu a+\nu b=a+b+2,$$
so $$\lambda=2,\;\mu=1,\;\nu=1.$$

The required value is
$$(\lambda+\mu+\nu)^2=(2+1+1)^2=4^2=16.$$

Therefore, the correct option is Option A (16).

$$x(\alpha,1,2)+y(1,\beta,2)+z(2,3,\gamma)=(0,0,0)$$ has nontrivial solution, so determinant = 0:

$$\begin{vmatrix}\alpha&1&2\\1&\beta&2\\2&3&\gamma\end{vmatrix} = 0$$.

$$\alpha(\beta\gamma-6)-1(\gamma-4)+2(3-2\beta) = 0$$.

$$\alpha\beta\gamma - 6\alpha - \gamma + 4 + 6 - 4\beta = 0$$.

Since $$\alpha\beta\gamma = 45$$: $$45 - 6\alpha - \gamma - 4\beta + 10 = 0 \Rightarrow 6\alpha + 4\beta + \gamma = 55$$.

The answer is 55.