\[ \int x^5 \coth ^{-1}(a x)^2 \, dx \]
Optimal antiderivative \[ \frac {4 x^{2}}{45 a^{4}}+\frac {x^{4}}{60 a^{2}}+\frac {x \,\mathrm {arccoth}\left (a x \right )}{3 a^{5}}+\frac {x^{3} \mathrm {arccoth}\left (a x \right )}{9 a^{3}}+\frac {x^{5} \mathrm {arccoth}\left (a x \right )}{15 a}-\frac {\mathrm {arccoth}\left (a x \right )^{2}}{6 a^{6}}+\frac {x^{6} \mathrm {arccoth}\left (a x \right )^{2}}{6}+\frac {23 \ln \left (-a^{2} x^{2}+1\right )}{90 a^{6}} \]
command
integrate(x^5*arccoth(a*x)^2,x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \frac {1}{90} \, {\left (\frac {15 \, {\left (\frac {3 \, {\left (a x + 1\right )}^{5}}{{\left (a x - 1\right )}^{5}} + \frac {10 \, {\left (a x + 1\right )}^{3}}{{\left (a x - 1\right )}^{3}} + \frac {3 \, {\left (a x + 1\right )}}{a x - 1}\right )} \log \left (\frac {a x + 1}{a x - 1}\right )^{2}}{\frac {{\left (a x + 1\right )}^{6} a^{7}}{{\left (a x - 1\right )}^{6}} - \frac {6 \, {\left (a x + 1\right )}^{5} a^{7}}{{\left (a x - 1\right )}^{5}} + \frac {15 \, {\left (a x + 1\right )}^{4} a^{7}}{{\left (a x - 1\right )}^{4}} - \frac {20 \, {\left (a x + 1\right )}^{3} a^{7}}{{\left (a x - 1\right )}^{3}} + \frac {15 \, {\left (a x + 1\right )}^{2} a^{7}}{{\left (a x - 1\right )}^{2}} - \frac {6 \, {\left (a x + 1\right )} a^{7}}{a x - 1} + a^{7}} + \frac {2 \, {\left (\frac {45 \, {\left (a x + 1\right )}^{4}}{{\left (a x - 1\right )}^{4}} - \frac {90 \, {\left (a x + 1\right )}^{3}}{{\left (a x - 1\right )}^{3}} + \frac {140 \, {\left (a x + 1\right )}^{2}}{{\left (a x - 1\right )}^{2}} - \frac {70 \, {\left (a x + 1\right )}}{a x - 1} + 23\right )} \log \left (\frac {a x + 1}{a x - 1}\right )}{\frac {{\left (a x + 1\right )}^{5} a^{7}}{{\left (a x - 1\right )}^{5}} - \frac {5 \, {\left (a x + 1\right )}^{4} a^{7}}{{\left (a x - 1\right )}^{4}} + \frac {10 \, {\left (a x + 1\right )}^{3} a^{7}}{{\left (a x - 1\right )}^{3}} - \frac {10 \, {\left (a x + 1\right )}^{2} a^{7}}{{\left (a x - 1\right )}^{2}} + \frac {5 \, {\left (a x + 1\right )} a^{7}}{a x - 1} - a^{7}} + \frac {4 \, {\left (\frac {11 \, {\left (a x + 1\right )}^{3}}{{\left (a x - 1\right )}^{3}} - \frac {16 \, {\left (a x + 1\right )}^{2}}{{\left (a x - 1\right )}^{2}} + \frac {11 \, {\left (a x + 1\right )}}{a x - 1}\right )}}{\frac {{\left (a x + 1\right )}^{4} a^{7}}{{\left (a x - 1\right )}^{4}} - \frac {4 \, {\left (a x + 1\right )}^{3} a^{7}}{{\left (a x - 1\right )}^{3}} + \frac {6 \, {\left (a x + 1\right )}^{2} a^{7}}{{\left (a x - 1\right )}^{2}} - \frac {4 \, {\left (a x + 1\right )} a^{7}}{a x - 1} + a^{7}} - \frac {46 \, \log \left (\frac {a x + 1}{a x - 1} - 1\right )}{a^{7}} + \frac {46 \, \log \left (\frac {a x + 1}{a x - 1}\right )}{a^{7}}\right )} a \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int x^{5} \operatorname {arcoth}\left (a x\right )^{2}\,{d x} \]________________________________________________________________________________________