If $$tan\alpha = ntan\beta$$ and $$sin\alpha = msin\beta$$, then $$cos^2\alpha$$ is

$$sin\alpha = m sin\beta$$
squaring on both sides 
$$1-cos^2\alpha = m^2 sin^2\beta$$
As it is given $$tan\alpha = n tan\beta$$ or $$sin\beta = \frac{tan\alpha \times cos\beta}{n}$$ now squaring on both sides and putting value above
it will get reduce to $$n^2cos^2\alpha = m^2cos^2\beta$$
$$cos^2\alpha = \frac{m^2}{n^2} (1-\frac{sin^2\alpha}{m^2})$$
Now solving above equation we will get value of $$cos^2\alpha$$ as $$\frac{m^2 -1}{n^2 - 1}$$