The values of $$a$$ and $$b$$, for which the system of equations
$$2x + 3y + 6z = 8$$
$$x + 2y + az = 5$$
$$3x + 5y + 9z = b$$
has no solution, are:

We have the linear system

$$$\begin{aligned} 2x + 3y + 6z &= 8, \\ x + 2y + az &= 5, \\ 3x + 5y + 9z &= b. \end{aligned}$$$

To decide when it has no solution, we compare the rank of the coefficient matrix with the rank of the augmented matrix. A system has no solution when these two ranks are different.

First, write the coefficient matrix $$A$$ and the augmented matrix $$[A\,|\,\mathbf{c}]$$:

$$$A=\begin{bmatrix}2&3&6\\[2pt]1&2&a\\[2pt]3&5&9\end{bmatrix},\qquad [A\,|\,\mathbf{c}]=\begin{bmatrix}2&3&6&8\\[2pt]1&2&a&5\\[2pt]3&5&9&b\end{bmatrix}.$$$

We first find the determinant of $$A$$. If this determinant is non-zero, the rank of $$A$$ is 3, the same as the rank of $$[A\,|\,\mathbf{c}]$$, and the system has a unique solution. So a necessary condition for no solution is

$$\det(A)=0.$$

Compute the determinant using cofactor expansion along the first row:

$$$ \begin{aligned} \det(A) &= 2\begin{vmatrix}2&a\\5&9\end{vmatrix} -3\begin{vmatrix}1&a\\3&9\end{vmatrix} +6\begin{vmatrix}1&2\\3&5\end{vmatrix} \\[6pt] &= 2\bigl(2\cdot9 - a\cdot5\bigr) -3\bigl(1\cdot9 - a\cdot3\bigr) +6\bigl(1\cdot5 - 2\cdot3\bigr) \\[6pt] &= 2(18 - 5a) - 3(9 - 3a) + 6(5 - 6) \\[6pt] &= (36 - 10a) - (27 - 9a) - 6 \\[6pt] &= 36 - 10a - 27 + 9a - 6 \\[6pt] &= 3 - a. \end{aligned} $$$

Thus $$\det(A)=0$$ exactly when $$a = 3$$. So for inconsistency we must have $$a=3$$.

Now substitute $$a=3$$ back into the equations:

$$$\begin{aligned} 2x + 3y + 6z &= 8, \quad -(1)\\ x + 2y + 3z &= 5, \quad -(2)\\ 3x + 5y + 9z &= b. \quad -(3) \end{aligned}$$$

Because the determinant vanished, the rank of the coefficient matrix can be at most 2. We check whether the third row is a linear combination of the first two.

Add equations (1) and (2):

$$$ (2x+3y+6z) + (x+2y+3z) = 8 + 5 \;\;\Longrightarrow\;\; 3x + 5y + 9z = 13. \quad -(4) $$$

The left side of (4) matches exactly the left side of equation (3). Hence, in the coefficient matrix we have

$$\text{Row}_3 = \text{Row}_1 + \text{Row}_2.$$

Because of this relation, the rank of the coefficient matrix is 2. For the augmented matrix to have the same rank (and thus give infinitely many solutions), its third right-hand side must satisfy the same combination; that is, we must also have

$$b = 8 + 5 = 13.$$

If instead $$b \neq 13,$$ the third augmented entry fails to satisfy the combination, so the rank of the augmented matrix becomes 3 while the rank of the coefficient matrix remains 2.

Whenever the two ranks differ, the system is inconsistent and has no solution. Therefore,

$$a = 3 \quad\text{and}\quad b \neq 13$$

are precisely the required conditions.

Hence, the correct answer is Option A.