Let the vectors $$\vec{a} = -\hat{i} + \hat{j} + 3\hat{k}$$ and $$\vec{b} = \hat{i} + 3\hat{j} + \hat{k}$$. For some $$\lambda, \mu \in \mathbb{R}$$, let $$\vec{c} = \lambda\vec{a} + \mu\vec{b}$$. If $$\vec{c} \cdot (3\hat{i} - 6\hat{j} + 2\hat{k}) = 10$$ and $$\vec{c} \cdot (\hat{i} + \hat{j} + \hat{k}) = -2$$, then $$|\vec{c}|^2$$ is equal to :

The given vectors are
$$\vec{a}= -\hat{i}+ \hat{j}+ 3\hat{k},\qquad \vec{b}= \hat{i}+ 3\hat{j}+ \hat{k}.$$

Let $$\vec{c}= \lambda\vec{a}+ \mu\vec{b}$$ where $$\lambda,\mu\in\mathbb{R}.$$
First write $$\vec{c}$$ in component form:

$$\vec{c}= \lambda(-1,1,3)+ \mu(1,3,1)=(-\lambda+\mu,\; \lambda+3\mu,\; 3\lambda+\mu).$$

The two scalar-product conditions are

1. $$\vec{c}\cdot(3\hat{i}-6\hat{j}+2\hat{k})=10,$$
2. $$\vec{c}\cdot(\hat{i}+ \hat{j}+ \hat{k})=-2.$$

Condition 1:
$$(-\lambda+\mu,\; \lambda+3\mu,\; 3\lambda+\mu)\cdot(3,-6,2)=10.$$

Compute the dot product:
$$3(-\lambda+\mu)+(-6)(\lambda+3\mu)+2(3\lambda+\mu)=10.$$

$$(-3\lambda+3\mu)\;+\;(-6\lambda-18\mu)\;+\;(6\lambda+2\mu)=10.$$

Combine like terms:
$$(-3\lambda-6\lambda+6\lambda)+ (3\mu-18\mu+2\mu)=10 \implies -3\lambda-13\mu=10.$$ $$\Rightarrow\quad -(3\lambda+13\mu)=10 \quad -(1)$$

Condition 2:
$$(-\lambda+\mu,\; \lambda+3\mu,\; 3\lambda+\mu)\cdot(1,1,1)=-2.$$

Compute the dot product:
$$(-\lambda+\mu)+(\lambda+3\mu)+(3\lambda+\mu)=-2.$$

Simplify:
$$(-\lambda+\lambda+3\lambda)+( \mu+3\mu+\mu)=-2 \implies 3\lambda+5\mu=-2 \quad -(2)$$

We now have the simultaneous linear equations

$$-3\lambda-13\mu=10 \quad -(1)$$
$$\;\;3\lambda+5\mu=-2 \quad\;\, -(2)$$

Add equations $$(1)+(2):$$
$$(-3\lambda+3\lambda)+(-13\mu+5\mu)=10-2 \implies -8\mu=8 \implies \mu=-1.$$

Substitute $$\mu=-1$$ in $$(2):$$
$$3\lambda+5(-1)=-2 \implies 3\lambda-5=-2 \implies 3\lambda=3 \implies \lambda=1.$$

Hence $$\lambda=1,\quad \mu=-1.$$

Therefore
$$\vec{c}=1\cdot\vec{a}+(-1)\cdot\vec{b}= \vec{a}-\vec{b}.$$

Compute $$\vec{c}$$ explicitly:
$$\vec{c}=(-1,1,3)-(1,3,1)=(-2,-2,2).$$

The square of its magnitude is
$$|\vec{c}|^{2}=(-2)^{2}+(-2)^{2}+2^{2}=4+4+4=12.$$

Hence, $$|\vec{c}|^{2}=12.$$

Option B which is: $$12$$