Let $$S$$ be the set of all $$(\lambda, \mu)$$ for which the vectors $$\lambda\hat{i} - \hat{j} + \hat{k}$$, $$\hat{j} + 2\hat{j} + \mu\hat{k}$$ and $$3\hat{i} - 4\hat{j} + 5\hat{k}$$, where $$\lambda - \mu = 5$$, are coplanar, then $$\sum_{(\lambda,\mu) \in S} 80(\lambda^2 + \mu^2)$$ is equal to

$$\begin{vmatrix} \lambda & -1 & 1 \\ 1 & 2 & \mu \\ 3 & -4 & 5 \end{vmatrix} = 0$$

$$\lambda(10 + 4\mu) + 1(5 - 3\mu) + 1(-10) = 0$$

$$10\lambda + 4\lambda\mu - 3\mu - 5 = 0$$

Substitute $$\mu = \lambda - 5$$:

$$10\lambda - 3(\lambda - 5) + 4\lambda(\lambda - 5) - 5 = 0$$

$$4\lambda^2 - 13\lambda + 10 = 0$$

$$\lambda_1 = 2 \implies \mu_1 = -3$$

$$\lambda_2 = \frac{5}{4} \implies \mu_2 = -\frac{15}{4}$$

$$S = \left\{ (2, -3), \left(\frac{5}{4}, -\frac{15}{4}\right) \right\}$$

$$\sum 80(\lambda^2 + \mu^2) = 80(2^2 + (-3)^2) + 80\left(\left(\frac{5}{4}\right)^2 + \left(-\frac{15}{4}\right)^2\right)$$

$$\sum = 80(13) + 80\left(\frac{25 + 225}{16}\right)$$

$$\sum = 1040 + 1250 = 2290$$