Let $$ABC$$ be a triangle. Consider four points $$p_{1},p_{2},p_{3},p_{4}$$ on the side AB, five points $$p_{5},p_{6},p_{7},p_{8},p_{9}$$ on the side $$BC$$, and four points $$p_{10},p_{11},p_{12},p_{13}$$ on the side $$AC$$. None of these points is a vertex of the trinagle $$ABC$$. Then the total number of pentagons, that can be formed by taking all the vertices from the points $$p_{1},p_{2},... ,p_{13}$$, is_______

Total number of ways to choose any five points from the thirteen given points is $$\binom{13}{5}=1287\quad-(1)$$

These points lie on the three sides of triangle $$ABC$$: - Side $$AB$$ has 4 points, - Side $$BC$$ has 5 points, - Side $$AC$$ has 4 points. A pentagon cannot have three collinear vertices, so we must exclude all choices that pick three or more points from any one side.

Let $$A$$ = selections with at least 3 points on $$AB$$, $$B$$ = selections with at least 3 points on $$BC$$, $$C$$ = selections with at least 3 points on $$AC$$. We will use inclusion-exclusion to count the invalid selections.

Compute $$|A|$$: choose $$k$$ points from the 4 on $$AB$$ and the remaining $$5-k$$ from the other 9 points. $$|A|=\sum_{k=3}^{4}\binom{4}{k}\binom{9}{5-k} =\binom{4}{3}\binom{9}{2}+\binom{4}{4}\binom{9}{1} =4\times36+1\times9 =144+9=153\quad-(2)$$

Compute $$|B|$$ similarly, choosing $$\ell$$ points from the 5 on $$BC$$ and $$5-\ell$$ from the other 8 points: $$|B|=\sum_{\ell=3}^{5}\binom{5}{\ell}\binom{8}{5-\ell} =\binom{5}{3}\binom{8}{2}+\binom{5}{4}\binom{8}{1}+\binom{5}{5}\binom{8}{0} =10\times28+5\times8+1\times1 =280+40+1=321\quad-(3)$$

Compute $$|C|$$ the same way as $$|A|$$ (since $$AC$$ also has 4 points): $$|C|=\sum_{m=3}^{4}\binom{4}{m}\binom{9}{5-m} =\binom{4}{3}\binom{9}{2}+\binom{4}{4}\binom{9}{1} =144+9=153\quad-(4)$$

For any two of these events (say $$A$$ and $$B$$), requiring at least 3 points from $$AB$$ and at least 3 from $$BC$$ would select at least 6 points, which exceeds 5. Hence all intersections $$A\cap B$$, $$B\cap C$$, $$C\cap A$$ and the triple intersection are empty. By inclusion-exclusion, the total number of invalid selections is $$|A\cup B\cup C|=|A|+|B|+|C|=153+321+153=627\quad-(5)$$

Therefore, the number of valid pentagons is $$1287-627=660\quad-(6)$$

Answer: 660