Discriminant

We are given the quadratic equation $$x^{2}-(m+1)x+m+4=0$$ and we must find all real numbers $$m$$ for which both its roots are real.

For any quadratic equation $$ax^{2}+bx+c=0,$$ the condition for real roots is that its discriminant is non-negative.

So we state the formula: $$\text{If } ax^{2}+bx+c=0,\; \text{then real roots} \Longleftrightarrow b^{2}-4ac\ge 0.$$

Here, by comparison, we have $$a=1,\; b=-(m+1),\; c=m+4.$$

Now we compute the discriminant:

$$\begin{aligned} D &= b^{2}-4ac \\ &= (-(m+1))^{2}-4(1)(m+4) \\ &= (m+1)^{2}-4m-16. \end{aligned}$$

Next, we expand $$(m+1)^{2}:$$

$$\begin{aligned} (m+1)^{2} &= m^{2}+2m+1, \\ \therefore D &= (m^{2}+2m+1)-4m-16 \\ &= m^{2}+2m+1-4m-16 \\ &= m^{2}-2m-15. \end{aligned}$$

For real roots we require $$D\ge 0,$$ so

$$m^{2}-2m-15\ge 0.$$

We factor the left side completely:

$$m^{2}-2m-15=(m-5)(m+3).$$

Thus,

$$ (m-5)(m+3)\ge 0. $$

For a product of two linear factors, the inequality $$PQ\ge 0$$ holds when both factors are non-negative or both are non-positive. Analysis of the critical points $$m=-3$$ and $$m=5$$ gives:

$$m\le -3 \quad \text{or} \quad m\ge 5.$$

Therefore

$$A = (-\infty,\,-3]\;\cup\;[5,\infty).$$

We are also given

$$B=[-3,\,5).$$

Now we examine each option one by one.

Option A: $$A-B = (-\infty,-3)\cup(5,\infty)$$

We calculate $$A-B$$ carefully.

• On the left of $$-3,$$ the interval $$(-\infty,-3]$$ is in $$A$$; removing the point $$-3$$ (because $$-3\in B$$) leaves $$(-\infty,-3).$$
• On the right, $$[5,\infty)$$ lies totally outside $$B$$ because $$B$$ stops just before $$5.$$ Hence the entire closed interval $$[5,\infty)$$ stays in the difference.

Thus

$$A-B = (-\infty,-3)\;\cup\;[5,\infty).$$

But Option A omits the point $$5,$$ writing $$(5,\infty)$$ instead of $$[5,\infty).$$ Hence Option A is not true.

Option B: $$A\cap B=\{-3\}$$

The intersection of $$(-\infty,-3]$$ with $$[-3,5)$$ gives just the single point $$-3,$$ and $$[5,\infty)$$ has empty overlap with $$[-3,5).$$ Hence Option B is true.

Option C: $$B-A=(-3,5)$$

Inside $$B$$ the point $$-3$$ is removed because $$-3\in A,$$ and $$5$$ is not in $$B$$ at all, so what remains is the open interval $$(-3,5).$$ Thus Option C is true.

Option D: $$A\cup B=\mathbb{R}$$

The union $$(-\infty,-3]\cup[-3,5)\cup[5,\infty)$$ clearly covers every real number, so Option D is also true.

We see that the only statement that fails is Option A.

Hence, the correct answer is Option A.

Let us denote the given quadratic by

$$f(x)=(\lambda^{2}+1)\,x^{2}-4\lambda\,x+2.$$

Since $$\lambda^{2}+1\gt 0$$ for every real $$\lambda$$, the parabola $$y=f(x)$$ always opens upward.

To be able to talk about the location of its roots we first ensure that real roots exist. For a quadratic $$ax^{2}+bx+c=0$$ the condition for real roots is given by the discriminant formula

$$\Delta=b^{2}-4ac\ge 0.$$

Here $$a=\lambda^{2}+1,\;b=-4\lambda,\;c=2,$$ so

$$\Delta=(-4\lambda)^{2}-4(\lambda^{2}+1)(2)=16\lambda^{2}-8(\lambda^{2}+1)=8\lambda^{2}-8=8(\lambda^{2}-1).$$

Thus

$$\Delta\ge 0\iff 8(\lambda^{2}-1)\ge 0\iff \lambda^{2}-1\ge 0\iff |\lambda|\ge 1.$$

Hence, if $$|\lambda|\lt 1$$ the equation has no real root at all, so no value of $$\lambda$$ from that part can be accepted. From now on we restrict ourselves to

$$|\lambda|\ge 1.$$

We want the quadratic to have exactly one root in the open interval $$(0,1).$$ Because the parabola opens upward, a convenient way to count the crossings of the $$x$$-axis inside $$(0,1)$$ is to look at the signs of $$f(x)$$ at the end-points of the interval and make use of the Intermediate Value Theorem.

At $$x=0$$ we have

$$f(0)=2,$$

which is always positive.

At $$x=1$$ we have

$$f(1)=(\lambda^{2}+1)\cdot 1^{2}-4\lambda\cdot 1+2=\lambda^{2}-4\lambda+3.$$

Factoring the quadratic in $$\lambda$$ gives

$$f(1)=\lambda^{2}-4\lambda+3=(\lambda-1)(\lambda-3).$$

Now we examine the sign of $$f(1).$$

• If $$1\lt \lambda\lt 3,$$ then $$\lambda-1\gt 0$$ and $$\lambda-3\lt 0,$$ so

$$f(1)\lt 0.$$

• If $$\lambda=1$$ or $$\lambda=3,$$ then $$f(1)=0.$$

• If $$\lambda\lt 1$$ or $$\lambda\gt 3,$$ then both factors have the same sign, hence

$$f(1)\gt 0.$$

Because $$f(0)\gt 0$$ always, the only way to obtain exactly one sign change (and therefore exactly one root) between $$x=0$$ and $$x=1$$ is to have

$$f(1)\lt 0.$$

Indeed, when $$f(1)\lt 0$$ the function is positive at $$x=0$$ and negative at $$x=1$$, so it must cross the $$x$$-axis precisely once within the open interval $$(0,1)$$. After that single crossing the graph eventually rises again (because the parabola opens upward) and meets the $$x$$-axis a second time at some $$x\gt 1$$, giving the total of two real roots with exactly one of them lying inside $$(0,1).$$

Therefore the required condition is

$$f(1)\lt 0\quad\Longleftrightarrow\quad (\lambda-1)(\lambda-3)\lt 0\quad\Longleftrightarrow\quad 1\lt \lambda\lt 3.$$

We still need to check the boundary points:

• For $$\lambda=1$$ we get $$f(1)=0$$ and the quadratic becomes $$2(x-1)^{2}=0,$$ whose double root is $$x=1,$$ lying outside the open interval $$(0,1)$$. Hence no root lies in $$(0,1)$$ — this value is excluded.

• For $$\lambda=3$$ we get one root at $$x=1$$ and the other at $$x=\dfrac15,$$ so exactly one root is in $$(0,1).$$ Thus $$\lambda=3$$ is admissible.

Combining everything, the set of all real $$\lambda$$ for which the quadratic has exactly one root in the interval $$(0,1)$$ is

$$(1,3].$$

Among the given options this matches Option C.

Hence, the correct answer is Option C.