\[ \int \frac {\cot ^3(x)}{\left (a+b \cot ^2(x)\right )^{5/2}} \, dx \]
Optimal antiderivative \[ -\frac {\arctanh \left (\frac {\sqrt {a +b \left (\cot ^{2}\left (x \right )\right )}}{\sqrt {a -b}}\right )}{\left (a -b \right )^{\frac {5}{2}}}+\frac {a}{3 \left (a -b \right ) b \left (a +b \left (\cot ^{2}\left (x \right )\right )\right )^{\frac {3}{2}}}+\frac {1}{\left (a -b \right )^{2} \sqrt {a +b \left (\cot ^{2}\left (x \right )\right )}} \]
command
integrate(cot(x)^3/(a+b*cot(x)^2)^(5/2),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ -\frac {\log \left ({\left | b \right |}\right ) \mathrm {sgn}\left (\sin \left (x\right )\right )}{2 \, {\left (\sqrt {a - b} a^{2} - 2 \, \sqrt {a - b} a b + \sqrt {a - b} b^{2}\right )}} + \frac {\frac {{\left (\frac {{\left (a^{3} + a^{2} b - 5 \, a b^{2} + 3 \, b^{3}\right )} \sin \left (x\right )^{2}}{a^{3} b - 3 \, a^{2} b^{2} + 3 \, a b^{3} - b^{4}} + \frac {3 \, {\left (a b^{2} - b^{3}\right )}}{a^{3} b - 3 \, a^{2} b^{2} + 3 \, a b^{3} - b^{4}}\right )} \sin \left (x\right )}{{\left (a \sin \left (x\right )^{2} - b \sin \left (x\right )^{2} + b\right )}^{\frac {3}{2}}} + \frac {3 \, \log \left ({\left | -\sqrt {a - b} \sin \left (x\right ) + \sqrt {a \sin \left (x\right )^{2} - b \sin \left (x\right )^{2} + b} \right |}\right )}{{\left (a^{2} - 2 \, a b + b^{2}\right )} \sqrt {a - b}}}{3 \, \mathrm {sgn}\left (\sin \left (x\right )\right )} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \text {Exception raised: TypeError} \]________________________________________________________________________________________