Let $$\lambda \in \mathbb{Z}$$, $$\vec{a} = \lambda \hat{i} + \hat{j} - \hat{k}$$ and $$\vec{b} = 3\hat{i} - \hat{j} + 2\hat{k}$$. Let $$\vec{c}$$ be a vector such that $$(\vec{a} + \vec{b}) \times \vec{c} = 0$$, $$\vec{a} \cdot \vec{c} = -17$$ and $$\vec{b} \cdot \vec{c} = -20$$. Then $$|\vec{c} \times (\lambda\hat{i} + \hat{j} + \hat{k})|^2$$ is equal to

Given: $$\vec{a} = \lambda\hat{i} + \hat{j} - \hat{k}$$, $$\vec{b} = 3\hat{i} - \hat{j} + 2\hat{k}$$, $$\lambda \in \mathbb{Z}$$.

$$(\vec{a} + \vec{b}) \times \vec{c} = \vec{0}$$, $$\vec{a} \cdot \vec{c} = -17$$, $$\vec{b} \cdot \vec{c} = -20$$.

$$\vec{a} + \vec{b} = (\lambda+3)\hat{i} + 0\hat{j} + \hat{k}$$

So $$\vec{c} = t[(\lambda+3)\hat{i} + 0\hat{j} + \hat{k}]$$ for some scalar $$t$$.

$$\vec{a} \cdot \vec{c} = t[\lambda(\lambda+3) + 0 - 1] = t[\lambda^2 + 3\lambda - 1] = -17$$ ... (i)

$$\vec{b} \cdot \vec{c} = t[3(\lambda+3) + 0 + 2] = t[3\lambda + 11] = -20$$ ... (ii)

From (i)/(ii): $$\frac{\lambda^2 + 3\lambda - 1}{3\lambda + 11} = \frac{17}{20}$$

$$20(\lambda^2 + 3\lambda - 1) = 17(3\lambda + 11)$$

$$20\lambda^2 + 60\lambda - 20 = 51\lambda + 187$$

$$20\lambda^2 + 9\lambda - 207 = 0$$

Using the quadratic formula: $$\lambda = \frac{-9 \pm \sqrt{81 + 4 \cdot 20 \cdot 207}}{40} = \frac{-9 \pm \sqrt{81 + 16560}}{40} = \frac{-9 \pm \sqrt{16641}}{40} = \frac{-9 \pm 129}{40}$$

$$\lambda = \frac{120}{40} = 3$$ or $$\lambda = \frac{-138}{40}$$ (not an integer).

So $$\lambda = 3$$.

From (ii): $$t(9 + 11) = -20$$, so $$t = -1$$.

$$\vec{c} = -1 \cdot (6\hat{i} + 0\hat{j} + \hat{k}) = -6\hat{i} - \hat{k}$$

Verify: $$\vec{a} \cdot \vec{c} = 3(-6) + 1(0) + (-1)(-1) = -18 + 1 = -17$$ ✓

$$\lambda\hat{i} + \hat{j} + \hat{k} = 3\hat{i} + \hat{j} + \hat{k}$$

$$\vec{c} \times (3\hat{i} + \hat{j} + \hat{k}) = \begin{vmatrix} \hat{i} & \hat{j} & \hat{k} \\ -6 & 0 & -1 \\ 3 & 1 & 1 \end{vmatrix}$$

$$= \hat{i}(0+1) - \hat{j}(-6+3) + \hat{k}(-6-0) = \hat{i}(1) - \hat{j}(-3) + \hat{k}(-6) = \hat{i} + 3\hat{j} - 6\hat{k}$$

$$|\vec{c} \times (3\hat{i} + \hat{j} + \hat{k})|^2 = 1 + 9 + 36 = 46$$

The correct answer is Option A: $$46$$.