Area of Triangle

We are given $$AB = 5$$, $$\angle B = \cos^{-1}\!\left(\frac{3}{5}\right)$$ (so $$\cos B = \frac{3}{5}$$ and $$\sin B = \frac{4}{5}$$), and circumradius $$R = 5$$.

By the law of sines, the side opposite to $$B$$ (i.e., $$AC$$) satisfies: $$\frac{AC}{\sin B} = 2R \implies AC = 2 \times 5 \times \frac{4}{5} = 8.$$

Now applying the law of cosines with $$BC = a$$: $$AC^2 = AB^2 + BC^2 - 2 \cdot AB \cdot BC \cdot \cos B$$ $$64 = 25 + a^2 - 2(5)(a)\!\left(\frac{3}{5}\right) = 25 + a^2 - 6a.$$

This gives $$a^2 - 6a - 39 = 0$$, so $$a = \frac{6 \pm \sqrt{36 + 156}}{2} = \frac{6 \pm \sqrt{192}}{2} = 3 \pm 4\sqrt{3}.$$

Since $$a = BC > 0$$, we have $$BC = 3 + 4\sqrt{3}$$.

The area of triangle $$ABC$$ is: $$\text{Area} = \frac{1}{2} \cdot AB \cdot BC \cdot \sin B = \frac{1}{2} \cdot 5 \cdot (3 + 4\sqrt{3}) \cdot \frac{4}{5} = 2(3 + 4\sqrt{3}) = 6 + 8\sqrt{3}.$$