If the system of equations $$x + (\sqrt{2}\sin\alpha)y + (\sqrt{2}\cos\alpha)z = 0$$, $$x + (\cos\alpha)y + (\sin\alpha)z = 0$$, $$x + (\sin\alpha)y - (\cos\alpha)z = 0$$ has a non-trivial solution, then $$\alpha \in \left(0,\frac{\pi}{2}\right)$$ is equal to:

A homogeneous system $$AX = 0$$ has a non-trivial solution if and only if the determinant of the coefficient matrix is zero ($$|A| = 0$$).

Set up the determinant:

$$\begin{vmatrix} 1 & \sqrt{2}\sin\alpha & \sqrt{2}\cos\alpha \\ 1 & \cos\alpha & \sin\alpha \\ 1 & \sin\alpha & -\cos\alpha \end{vmatrix} = 0$$

Perform row operations ($$R_2 \to R_2 - R_1$$ and $$R_3 \to R_3 - R_1$$) or expand directly. Expanding along the first column:

$$1(-\cos^2\alpha - \sin^2\alpha) - 1(-\sqrt{2}\sin\alpha\cos\alpha - \sqrt{2}\sin\alpha\cos\alpha) + 1(\sqrt{2}\sin^2\alpha - \sqrt{2}\cos^2\alpha) = 0$$

Simplify:

$$-1 + 2\sqrt{2}\sin\alpha\cos\alpha - \sqrt{2}(\cos^2\alpha - \sin^2\alpha) = 0$$

$$-1 + \sqrt{2}\sin 2\alpha - \sqrt{2}\cos 2\alpha = 0 \implies \sin 2\alpha - \cos 2\alpha = \frac{1}{\sqrt{2}}$$

Divide by $$\sqrt{2}$$: $$\frac{1}{\sqrt{2}}\sin 2\alpha - \frac{1}{\sqrt{2}}\cos 2\alpha = \frac{1}{2} \implies \sin(2\alpha - \frac{\pi}{4}) = \frac{1}{2}$$

$$2\alpha - \frac{\pi}{4} = \frac{\pi}{6} \implies 2\alpha = \frac{\pi}{6} + \frac{\pi}{4} = \frac{5\pi}{12} \implies \alpha = \frac{5\pi}{24}$$