Let $$\alpha \in R$$ and the three vectors $$\vec{a} = \alpha\hat{i} + \hat{j} + 3\hat{k}$$, $$\vec{b} = 2\hat{i} + \hat{j} - \alpha\hat{k}$$ and $$\vec{c} = \alpha\hat{i} - 2\hat{j} + 3\hat{k}$$. Then the set S = {$$\alpha$$: $$\vec{a}$$, $$\vec{b}$$ and $$\vec{c}$$ are coplanar}

We are asked to find all real numbers $$\alpha$$ for which the three vectors $$\vec{a}= \alpha\hat{i}+\hat{j}+3\hat{k}$$, $$\vec{b}=2\hat{i}+\hat{j}-\alpha\hat{k}$$ and $$\vec{c}= \alpha\hat{i}-2\hat{j}+3\hat{k}$$ are coplanar.

Coplanarity of three vectors is tested by the scalar triple product. The condition is: a set of three vectors is coplanar if and only if the scalar triple product vanishes, that is, we must have

$$\vec{a}\cdot\left( \vec{b}\times\vec{c}\right)=0.$$

Now we compute the cross product $$\vec{b}\times\vec{c}$$ first. Writing the two vectors in component form

$$\vec{b}=(2,\;1,\;-\alpha),\qquad \vec{c}=(\alpha,\;-2,\;3),$$

we use the determinant formula for the cross product:

$$\vec{b}\times\vec{c}= \begin{vmatrix} \hat{i} & \hat{j} & \hat{k}\\[2pt] 2 & 1 & -\alpha\\[2pt] \alpha & -2 & 3 \end{vmatrix}.$$

Expanding this determinant along the first row we obtain

$$ \vec{b}\times\vec{c}= \hat{i}\bigl(1\cdot3-(-\alpha)(-2)\bigr) -\hat{j}\bigl(2\cdot3-(-\alpha)(\alpha)\bigr) +\hat{k}\bigl(2\cdot(-2)-1\cdot\alpha\bigr). $$

Simplifying every bracket step by step:

For the $$\hat{i}$$ component: $$1\cdot3 = 3,\qquad (-\alpha)(-2)=2\alpha,\qquad 3-2\alpha=3-2\alpha.$$

For the $$\hat{j}$$ component: $$2\cdot3 = 6,\qquad (-\alpha)(\alpha)=-\alpha^{2},\qquad 6-(-\alpha^{2})=6+\alpha^{2}.$$

For the $$\hat{k}$$ component: $$2\cdot(-2)=-4,\qquad 1\cdot\alpha=\alpha,\qquad -4-\alpha=-4-\alpha.$$

Putting these results back, we have

$$\vec{b}\times\vec{c}= \bigl(3-2\alpha\bigr)\hat{i}-\bigl(6+\alpha^{2}\bigr)\hat{j}+\bigl(-4-\alpha\bigr)\hat{k}.$$

Thus in component form $$\vec{b}\times\vec{c}=(\,3-2\alpha,\;-\bigl(6+\alpha^{2}\bigr),\;-4-\alpha\,).$$

Next we take the dot product of $$\vec{a}=(\alpha,\,1,\,3)$$ with this result. The dot-product formula is

$$\vec{a}\cdot(\vec{b}\times\vec{c}) = \alpha\,(3-2\alpha)+1\cdot\!\Bigl(-\bigl(6+\alpha^{2}\bigr)\Bigr)+3\cdot(-4-\alpha).$$

We now expand each term carefully:

First term: $$\alpha(3-2\alpha)=3\alpha-2\alpha^{2}.$$

Second term: $$1\cdot\!\Bigl(-\bigl(6+\alpha^{2}\bigr)\Bigr)=-6-\alpha^{2}.$$

Third term: $$3\cdot(-4-\alpha)=-12-3\alpha.$$

Adding all three expressions gives

$$ (3\alpha-2\alpha^{2})+(-6-\alpha^{2})+(-12-3\alpha). $$

Collecting like terms for clarity:

• $$\alpha$$ terms: $$3\alpha-3\alpha=0.$$

• $$\alpha^{2}$$ terms: $$-2\alpha^{2}-\alpha^{2}=-3\alpha^{2}.$$

• Constant terms: $$-6-12=-18.$$

Thus the scalar triple product simplifies to

$$-3\alpha^{2}-18.$$

For coplanarity we set this equal to zero:

$$-3\alpha^{2}-18 = 0.$$

Dividing both sides by $$-3$$ (so that the sign and coefficient become simpler), we get

$$\alpha^{2}+6 = 0.$$

We now solve this quadratic equation. Rearranged, it is

$$\alpha^{2} = -6.$$

But $$\alpha^{2}$$, being a square of a real number, is always non-negative, while the right-hand side $$-6$$ is strictly negative. Hence there is no real value of $$\alpha$$ satisfying this equation.

Therefore the set $$S=\{\alpha\in\mathbb{R}:\vec{a},\vec{b},\vec{c}\text{ are coplanar}\}$$ contains no elements; in other words, $$S$$ is empty.

Hence, the correct answer is Option C.