\[ \int \frac {\coth ^{-1}(x)}{\left (a-a x^2\right )^{5/2}} \, dx \]
Optimal antiderivative \[ -\frac {1}{9 a \left (-a \,x^{2}+a \right )^{\frac {3}{2}}}+\frac {x \,\mathrm {arccoth}\left (x \right )}{3 a \left (-a \,x^{2}+a \right )^{\frac {3}{2}}}-\frac {2}{3 a^{2} \sqrt {-a \,x^{2}+a}}+\frac {2 x \,\mathrm {arccoth}\left (x \right )}{3 a^{2} \sqrt {-a \,x^{2}+a}} \]
command
integrate(arccoth(x)/(-a*x^2+a)^(5/2),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ -\frac {\sqrt {-a x^{2} + a} x {\left (\frac {2 \, x^{2}}{a} - \frac {3}{a}\right )} \log \left (-\frac {\frac {1}{x} + 1}{\frac {1}{x} - 1}\right )}{6 \, {\left (a x^{2} - a\right )}^{2}} - \frac {6 \, a x^{2} - 7 \, a}{9 \, {\left (a x^{2} - a\right )} \sqrt {-a x^{2} + a} a^{2}} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int \frac {\operatorname {arcoth}\left (x\right )}{{\left (-a x^{2} + a\right )}^{\frac {5}{2}}}\,{d x} \]________________________________________________________________________________________