\[ \int x^5 \coth ^{-1}(a x) \, dx \]
Optimal antiderivative \[ \frac {x}{6 a^{5}}+\frac {x^{3}}{18 a^{3}}+\frac {x^{5}}{30 a}+\frac {x^{6} \mathrm {arccoth}\left (a x \right )}{6}-\frac {\arctanh \left (a x \right )}{6 a^{6}} \]
command
integrate(x^5*arccoth(a*x),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \frac {1}{45} \, a {\left (\frac {\frac {45 \, {\left (a x + 1\right )}^{4}}{{\left (a x - 1\right )}^{4}} - \frac {90 \, {\left (a x + 1\right )}^{3}}{{\left (a x - 1\right )}^{3}} + \frac {140 \, {\left (a x + 1\right )}^{2}}{{\left (a x - 1\right )}^{2}} - \frac {70 \, {\left (a x + 1\right )}}{a x - 1} + 23}{a^{7} {\left (\frac {a x + 1}{a x - 1} - 1\right )}^{5}} + \frac {15 \, {\left (\frac {3 \, {\left (a x + 1\right )}^{5}}{{\left (a x - 1\right )}^{5}} + \frac {10 \, {\left (a x + 1\right )}^{3}}{{\left (a x - 1\right )}^{3}} + \frac {3 \, {\left (a x + 1\right )}}{a x - 1}\right )} \log \left (-\frac {\frac {\frac {{\left (a x + 1\right )} a}{a x - 1} - a}{a {\left (\frac {a x + 1}{a x - 1} + 1\right )}} + 1}{\frac {\frac {{\left (a x + 1\right )} a}{a x - 1} - a}{a {\left (\frac {a x + 1}{a x - 1} + 1\right )}} - 1}\right )}{a^{7} {\left (\frac {a x + 1}{a x - 1} - 1\right )}^{6}}\right )} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int x^{5} \operatorname {arcoth}\left (a x\right )\,{d x} \]________________________________________________________________________________________