The system of equations $$x+y+z=6\\x+2y+5z=9,\\x+5y+\lambda z=\mu,$$ has no solution if

The system: $$x+y+z=6$$, $$x+2y+5z=9$$, $$x+5y+\lambda z=\mu$$ has no solution when the determinant of coefficients is zero but the system is inconsistent.

Compute the determinant of the coefficient matrix: $$D = \begin{vmatrix}1&1&1\\1&2&5\\1&5&\lambda\end{vmatrix} = 1(2\lambda-25)-1(\lambda-5)+1(5-2) = 2\lambda-25-\lambda+5+3 = \lambda-17$$.

For no solution one requires $$D = 0 \implies \lambda = 17$$.

With $$\lambda = 17$$, the determinant of the augmented matrix replacing the first column by the constants becomes $$D_x = \begin{vmatrix}6&1&1\\9&2&5\\\mu&5&17\end{vmatrix} = 6(34-25)-1(153-5\mu)+1(45-2\mu) = 54-153+5\mu+45-2\mu = 3\mu-54$$.

Inconsistency requires $$D_x \neq 0 \implies 3\mu \neq 54 \implies \mu \neq 18$$.

Thus the system has no solution when $$\lambda = 17$$ and $$\mu \neq 18$$.

The correct answer is Option 3: $$\lambda = 17, \mu \neq 18$$.