Let $$\vec{a} = \hat{i} + 2\hat{j} + \hat{k}$$ and $$\vec{b} = 2\hat{i} + \hat{j} - \hat{k}$$. Let $$\hat{c}$$ be a unit vector in the plane of the vectors $$\vec{a}$$ and $$\vec{b}$$ and be perpendicular to $$\vec{a}$$. Then such a vector $$\hat{c}$$ is :

We want a unit vector $$\hat{c}$$ which

• lies in the plane of $$\vec{a}$$ and $$\vec{b}$$ (so it is a linear combination of them), and
• is perpendicular to $$\vec{a}$$.

Write $$\vec{a} = \hat{i} + 2\hat{j} + \hat{k}$$ and $$\vec{b} = 2\hat{i} + \hat{j} - \hat{k}$$.

A convenient way to obtain a vector lying in the required plane and orthogonal to $$\vec{a}$$ is to remove from $$\vec{b}$$ its component along $$\vec{a}$$. The result is automatically in the span of $$\vec{a},\vec{b}$$ and perpendicular to $$\vec{a}$$.

First compute the projection of $$\vec{b}$$ on $$\vec{a}$$.
Formula: projection of $$\vec{b}$$ on $$\vec{a}$$ is $$\displaystyle\frac{\vec{a}\cdot\vec{b}}{\vec{a}\cdot\vec{a}}\;\vec{a}$$.

Dot products:
$$\vec{a}\cdot\vec{b} = 1\cdot2 + 2\cdot1 + 1\cdot(-1) = 2 + 2 - 1 = 3$$
$$\vec{a}\cdot\vec{a} = 1^2 + 2^2 + 1^2 = 1 + 4 + 1 = 6$$

Hence the projection is
$$\frac{3}{6}\,\vec{a} = \frac{1}{2}\,( \hat{i} + 2\hat{j} + \hat{k}) = 0.5\,\hat{i} + 1\,\hat{j} + 0.5\,\hat{k}$$.

Subtract this from $$\vec{b}$$ to get the component of $$\vec{b}$$ perpendicular to $$\vec{a}$$:

$$\vec{d} = \vec{b} - \text{proj}_{\vec{a}}\vec{b}$$

$$= (2,\,1,\,-1) - (0.5,\,1,\,0.5)$$

$$= (1.5,\,0,\,-1.5)$$.

Multiply by $$2$$ to clear the decimal:

$$\vec{d} = (3,\,0,\,-3) = 3(1,\,0,\,-1).$$

Any non-zero scalar multiple of $$\vec{d}$$ is still in the same direction, so we choose the simple vector
$$\vec{m} = (1,\,0,\,-1) = \hat{i} - \hat{k}$$.

Check orthogonality with $$\vec{a}$$:
$$(\hat{i} - \hat{k})\cdot(\hat{i} + 2\hat{j} + \hat{k}) = 1 - 1 = 0,$$
so it is indeed perpendicular to $$\vec{a}$$.

Now normalise to get a unit vector:

Magnitude of $$\vec{m}$$:
$$|\vec{m}| = \sqrt{1^2 + 0^2 + (-1)^2} = \sqrt{2}$$.

Hence

$$\hat{c} = \frac{\vec{m}}{|\vec{m}|} = \frac{1}{\sqrt{2}}(\hat{i} - \hat{k}).$$

The negative of a unit vector is also a valid unit vector, so $$-\hat{c} = \dfrac{1}{\sqrt{2}}(-\hat{i} + \hat{k})$$ is equally acceptable.

Comparing with the options, Option D gives $$\dfrac{1}{\sqrt{2}}(-\hat{i} + \hat{k})$$, which matches (up to sign).

Therefore the required vector is given by Option D.