Let $$R_1$$ and $$R_2$$ be two relations defined as follows:
$$R_1 = \{(a, b) \in R^2 : a^2 + b^2 \in Q\}$$ and $$R_2 = \{(a, b) \in R^2 : a^2 + b^2 \notin Q\}$$, where Q is the set of all rational numbers, then

We have two relations on the set of real numbers $$\mathbb R$$.

$$R_1=\{(a,b)\in\mathbb R^2 : a^2+b^2\in\mathbb Q\}$$

$$R_2=\{(a,b)\in\mathbb R^2 : a^2+b^2\notin\mathbb Q\}$$

To decide which of the two relations is transitive, recall the definition of transitivity:

A relation $$R$$ on a set is transitive if and only if

$$ (a,b)\in R \;\text{and}\; (b,c)\in R \;\Longrightarrow\; (a,c)\in R \quad\text{for every}\; a,b,c. $$

First we check $$R_1$$. Assume $$(a,b)\in R_1$$ and $$(b,c)\in R_1.$$ So

$$a^2+b^2\in\mathbb Q\quad\text{and}\quad b^2+c^2\in\mathbb Q.$$

It does not automatically follow that $$a^2+c^2$$ is rational. Indeed we exhibit three concrete numbers that break the implication.

Select

$$b^2=\sqrt2,\qquad a^2=5-\sqrt2,\qquad c^2=5-\sqrt2.$$

Because $$5-\sqrt2>0,$$ the square-roots exist in $$\mathbb R$$, so put

$$a=\sqrt{\,5-\sqrt2\,},\quad b=\sqrt[\,4]2,\quad c=\sqrt{\,5-\sqrt2\,}.$$

Now compute the three sums of squares:

$$a^2+b^2=(5-\sqrt2)+\sqrt2=5\in\mathbb Q,$$

$$b^2+c^2=\sqrt2+(5-\sqrt2)=5\in\mathbb Q,$$

$$a^2+c^2=(5-\sqrt2)+(5-\sqrt2)=10-2\sqrt2\notin\mathbb Q.$$

Thus $$(a,b)\in R_1,\;(b,c)\in R_1$$ but $$(a,c)\notin R_1.$$ Therefore $$R_1$$ is not transitive.

Next we examine $$R_2$$. Again start with two pairs in the relation and see whether the third one must follow. Choose

$$a=1,\qquad b=\sqrt\pi,\qquad c=2.$$

We have

$$a^2=1\in\mathbb Q,\qquad b^2=\pi\notin\mathbb Q,\qquad c^2=4\in\mathbb Q.$$

Calculate the relevant sums:

$$a^2+b^2=1+\pi\notin\mathbb Q\;\Longrightarrow\;(a,b)\in R_2,$$

$$b^2+c^2=\pi+4\notin\mathbb Q\;\Longrightarrow\;(b,c)\in R_2,$$

$$a^2+c^2=1+4=5\in\mathbb Q\;\Longrightarrow\;(a,c)\notin R_2.$$

Hence $$(a,b)\in R_2$$ and $$(b,c)\in R_2$$ do not force $$(a,c)\in R_2$$. So $$R_2$$ also fails to be transitive.

We have shown that neither of the two relations satisfies the transitivity condition. Hence, the correct answer is Option C.