If the system of linear equations :
$$x_1 + 2x_2 + 3x_3 = 6$$
$$x_1 + 3x_2 + 5x_3 = 9$$
$$2x_1 + 5x_2 + ax_3 = b$$
is consistent and has infinite number of solutions, then :

First we rewrite the three linear equations in matrix form. We have the coefficient matrix $$A$$ and the augmented matrix $$[A\;|\;B]$$ as

$$ A=\begin{bmatrix} 1 & 2 & 3\\ 1 & 3 & 5\\ 2 & 5 & a \end{bmatrix}, \qquad [A\;|\;B]=\begin{bmatrix} 1 & 2 & 3 & \big| & 6\\ 1 & 3 & 5 & \big| & 9\\ 2 & 5 & a & \big| & b \end{bmatrix}. $$

For a system to possess infinitely many solutions two conditions must hold together: (1) The system is consistent, that is $$\operatorname{rank}(A)=\operatorname{rank}([A\;|\;B]).$$ (2) This common rank is strictly less than the number of unknowns, here $$3.$$ Thus we need the rank to come out $$2.$$

To find the rank we perform elementary row operations on the augmented matrix. We start with

$$ \begin{bmatrix} 1 & 2 & 3 & \big| & 6\\ 1 & 3 & 5 & \big| & 9\\ 2 & 5 & a & \big| & b \end{bmatrix}. $$

First we create zeros below the leading 1 in the first column:

Row-operation: $$R_2 \rightarrow R_2 - R_1.$$ Using it,

$$ R_2 = (1,\,3,\,5\,|\,9)\;-\;(1,\,2,\,3\,|\,6) =(0,\,1,\,2\,|\,3). $$

Row-operation: $$R_3 \rightarrow R_3 - 2R_1.$$ So

$$ R_3=(2,\,5,\,a\,|\,b)\;-\;2(1,\,2,\,3\,|\,6) =(0,\,1,\,a-6\,|\,b-12). $$

After these two operations the matrix becomes

$$ \begin{bmatrix} 1 & 2 & 3 & \big| & 6\\ 0 & 1 & 2 & \big| & 3\\ 0 & 1 & a-6 & \big| & b-12 \end{bmatrix}. $$

Next we eliminate the entry below the leading 1 of the second row:

Row-operation: $$R_3 \rightarrow R_3 - R_2.$$ Explicitly,

$$ R_3=(0,\,1,\,a-6\,|\,b-12)\;-\;(0,\,1,\,2\,|\,3) =(0,\,0,\,a-8\,|\,b-15). $$

The matrix is now in (upper) triangular form:

$$ \begin{bmatrix} 1 & 2 & 3 & \big| & 6\\ 0 & 1 & 2 & \big| & 3\\ 0 & 0 & a-8 & \big| & b-15 \end{bmatrix}. $$

The first two rows are clearly non-zero, so the rank is at least $$2.$$ The third row will

• contribute nothing new if and only if every entry in it is zero, i.e. $$a-8=0$$ and $$b-15=0.$$ • otherwise it will raise the rank to $$3.$$

For the rank to settle at $$2,$$ we must have

$$a-8=0 \;\Longrightarrow\; a=8,$$ $$b-15=0 \;\Longrightarrow\; b=15.$$

With these values the third row indeed becomes $$\bigl(0,\,0,\,0\,\big|\,0\bigr),$$ so

$$\operatorname{rank}(A)=\operatorname{rank}([A\;|\;B])=2<3,$$

and the system is consistent with infinitely many solutions. No other choice of $$a$$ and $$b$$ satisfies both requirements simultaneously.

Hence, the correct answer is Option D.