Two co-axial conducting cylinders of same length $$\ell$$ with radii $$\sqrt{2}R$$ and $$2R$$ are kept, as shown in Fig. 1. The charge on the inner cylinder is $$Q$$ and the outer cylinder is grounded. The annular region between the cylinders is filled with a material of dielectric constant $$\kappa = 5$$. Consider an imaginary plane of the same length $$\ell$$ at a distance $$R$$ from the common axis of the cylinders. This plane is parallel to the axis of the cylinders. The cross-sectional view of this arrangement is shown in Fig. 2. Ignoring edge effects, the flux of the electric field through the plane is ($$\epsilon_0$$ is the permittivity of free space):

Let the inner and outer conducting cylinders be coaxial with the common $$z$$-axis.
Radius of inner cylinder  : $$\sqrt{2}\,R$$
Radius of outer cylinder  : $$2\,R$$
Free charge on inner cylinder  : $$Q$$ (uniformly distributed along its length $$\ell$$)
Surface charge density per unit length (line charge)  : $$\lambda=\dfrac{Q}{\ell}$$.

The space between the two cylinders is filled with a dielectric of relative permittivity $$\kappa=5$$.
The outer cylinder is earthed, hence the electric field exists only in the dielectric region $$\sqrt{2}\,R \le r \le 2R$$ and is purely radial.

Electric field inside the dielectric
Take a cylindrical Gaussian surface of radius $$r$$ (with $$\sqrt{2}\,R \le r \le 2R$$) and length $$\ell$$. Using Gauss’s law in dielectrics,

$$\oint \mathbf{D}\cdot d\mathbf{A}=Q_{\text{free enclosed}}\; \Longrightarrow \; D(2\pi r\ell)=\lambda\ell \Rightarrow D=\dfrac{\lambda}{2\pi r}.$$ Since $$\mathbf{D}=\kappa\varepsilon_0\mathbf{E}$$, we get

$$E(r)=\dfrac{\lambda}{2\pi\kappa\varepsilon_0\,r}, \qquad \mathbf{E}=E(r)\,\hat{\mathbf{r}}.$$ (Inside $$r\lt \sqrt{2}\,R$$ the electric field is zero.)

The required open surface
We need the electric flux through a plane that

• is parallel to the axis, of length $$\ell$$,
• is at a perpendicular distance $$R$$ from the axis (take it as the plane $$x=R$$).

In the cross-section (the $$xy$$-plane) this surface appears as the straight line $$x=R$$. Only the part of this line lying in the dielectric region ($$r\ge\sqrt{2}\,R$$) contributes to flux.

Limits of the contributing strip on the plane
Put $$x=R$$ and write $$r=\sqrt{x^{2}+y^{2}}=\sqrt{R^{2}+y^{2}}.$$

• Inner boundary: $$r=\sqrt{2}\,R \;\Rightarrow\; y=\pm R$$
• Outer boundary: $$r=2\,R \;\Rightarrow\; y=\pm\sqrt{3}\,R$$

Element of area and flux through it
For a strip of width $$dy$$ on this plane, area element $$dA=\ell\,dy.$$ The unit normal to the plane is $$\hat{\mathbf{n}}=\hat{\mathbf{x}}$$ (along $$+x$$), while the electric field is radial, so the cosine of the angle between $$\mathbf{E}$$ and $$\hat{\mathbf{n}}$$ equals $$\dfrac{x}{r}=\dfrac{R}{r}.$$ Therefore, the differential flux is

$$d\Phi=E(r)\left(\dfrac{R}{r}\right)dA =\dfrac{\lambda}{2\pi\kappa\varepsilon_0\,r}\left(\dfrac{R}{r}\right)\ell\,dy =\dfrac{\lambda R\ell}{2\pi\kappa\varepsilon_0}\dfrac{dy}{(R^{2}+y^{2})}.$$

Total flux through the plane

$$\Phi=\int_{-\sqrt{3}R}^{-R}+\int_{R}^{\sqrt{3}R} \dfrac{\lambda R\ell}{2\pi\kappa\varepsilon_0}\; \dfrac{dy}{R^{2}+y^{2}} =\dfrac{\lambda R\ell}{\pi\kappa\varepsilon_0} \int_{R}^{\sqrt{3}R}\dfrac{dy}{R^{2}+y^{2}} \quad (\text{using symmetry}).$$

Evaluate the integral:

$$\int_{R}^{\sqrt{3}R}\dfrac{dy}{R^{2}+y^{2}} =\left[\dfrac{1}{R}\arctan\left(\dfrac{y}{R}\right)\right]_{R}^{\sqrt{3}R} =\dfrac{1}{R}\left[\arctan(\sqrt{3})-\arctan(1)\right] =\dfrac{1}{R}\left(\dfrac{\pi}{3}-\dfrac{\pi}{4}\right) =\dfrac{\pi}{12R}.$$

Substituting,

$$\Phi=\dfrac{\lambda R\ell}{\pi\kappa\varepsilon_0}\cdot\dfrac{\pi}{12R} =\dfrac{\lambda\ell}{12\kappa\varepsilon_0} =\dfrac{Q}{12\kappa\varepsilon_0}.$$

With $$\kappa=5$$,

$$\Phi=\dfrac{Q}{12\times5\;\varepsilon_0} =\dfrac{Q}{60\,\varepsilon_0}.$$

Therefore, the electric flux through the given plane is $$\dfrac{Q}{60\varepsilon_0}$$.

Option C which is: $$\displaystyle\frac{Q}{60\varepsilon_0}$$