If $$\vec{a} = \dfrac{1}{\sqrt{10}}(3\hat{i} + \hat{k})$$ and $$\vec{b} = \dfrac{1}{7}(2\hat{i} + 3\hat{j} - 6\hat{k})$$, then the value of $$(2\vec{a} - \vec{b}) \cdot [(\vec{a} \times \vec{b}) \times (\vec{a} + 2\vec{b})]$$ is:

Given
$$\vec{a}= \dfrac{1}{\sqrt{10}}\,(3\hat i+0\hat j+1\hat k), \qquad \vec{b}= \dfrac{1}{7}\,(2\hat i+3\hat j-6\hat k)$$

Compute the basic scalars first.

Magnitude of $$\vec{a}: \; a^{2}= \vec{a}\cdot\vec{a}= \dfrac{9+0+1}{10}=1$$
Magnitude of $$\vec{b}: \; b^{2}= \vec{b}\cdot\vec{b}= \dfrac{4+9+36}{49}=1$$
Dot product $$\vec{a}\cdot\vec{b}= \dfrac{1}{7\sqrt{10}}\,[3(2)+0(3)+1(-6)] =\dfrac{6-6}{7\sqrt{10}}=0$$

Hence $$a^{2}=1,\; b^{2}=1,\; \vec{a}\cdot\vec{b}=0$$. The two given vectors are mutually perpendicular and both are unit vectors.

The required expression is
$$E=(2\vec{a}-\vec{b})\cdot\big[(\vec{a}\times\vec{b})\times(\vec{a}+2\vec{b})\big]$$

First expand the double cross product using the identity
$$(\vec{p}\times\vec{q})\times\vec{r}= \vec{q}(\vec{p}\cdot\vec{r})-\vec{p}(\vec{q}\cdot\vec{r})$$

Write
$$(\vec{a}\times\vec{b})\times(\vec{a}+2\vec{b}) =(\vec{a}\times\vec{b})\times\vec{a}+2(\vec{a}\times\vec{b})\times\vec{b}$$

Apply the identity to each term.

1. $$(\vec{a}\times\vec{b})\times\vec{a}= \vec{b}(\vec{a}\cdot\vec{a})-\vec{a}(\vec{b}\cdot\vec{a})$$
2. $$(\vec{a}\times\vec{b})\times\vec{b}= \vec{b}(\vec{a}\cdot\vec{b})-\vec{a}(\vec{b}\cdot\vec{b})$$

Add them with the proper coefficients:

$$\big[(\vec{a}\times\vec{b})\times(\vec{a}+2\vec{b})\big] =\vec{b}\!\big[a^{2}+2(\vec{a}\cdot\vec{b})\big] -\vec{a}\!\big[(\vec{a}\cdot\vec{b})+2b^{2}\big]$$

Now take the dot product with $$2\vec{a}-\vec{b}:$$

$$E=(2\vec{a}-\vec{b})\cdot\vec{b}\,[a^{2}+2(\vec{a}\cdot\vec{b})] -(2\vec{a}-\vec{b})\cdot\vec{a}\,[(\vec{a}\cdot\vec{b})+2b^{2}]$$

Calculate the two simple dot products:
$$(2\vec{a}-\vec{b})\cdot\vec{b}=2(\vec{a}\cdot\vec{b})-b^{2} =2(0)-1=-1$$
$$(2\vec{a}-\vec{b})\cdot\vec{a}=2a^{2}-(\vec{a}\cdot\vec{b}) =2(1)-0=2$$

Substitute all scalar values $$a^{2}=1,\; b^{2}=1,\; \vec{a}\cdot\vec{b}=0$$:

$$E=(-1)[\,1+2(0)\,]- (2)[\,0+2(1)\,] =(-1)(1)-2(2) =-1-4 =-5$$

Therefore the value of the given expression is $$-5$$.

Option D which is: $$-5$$