\[ \int x^2 \coth ^{-1}(a x) \, dx \]
Optimal antiderivative \[ \frac {x^{2}}{6 a}+\frac {x^{3} \mathrm {arccoth}\left (a x \right )}{3}+\frac {\ln \left (-a^{2} x^{2}+1\right )}{6 a^{3}} \]
command
integrate(x^2*arccoth(a*x),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \frac {1}{3} \, a {\left (\frac {\log \left (\frac {{\left | a x + 1 \right |}}{{\left | a x - 1 \right |}}\right )}{a^{4}} - \frac {\log \left ({\left | \frac {a x + 1}{a x - 1} - 1 \right |}\right )}{a^{4}} + \frac {{\left (\frac {3 \, {\left (a x + 1\right )}^{2}}{{\left (a x - 1\right )}^{2}} + 1\right )} \log \left (-\frac {\frac {\frac {{\left (a x + 1\right )} a}{a x - 1} - a}{a {\left (\frac {a x + 1}{a x - 1} + 1\right )}} + 1}{\frac {\frac {{\left (a x + 1\right )} a}{a x - 1} - a}{a {\left (\frac {a x + 1}{a x - 1} + 1\right )}} - 1}\right )}{a^{4} {\left (\frac {a x + 1}{a x - 1} - 1\right )}^{3}} + \frac {2 \, {\left (a x + 1\right )}}{{\left (a x - 1\right )} a^{4} {\left (\frac {a x + 1}{a x - 1} - 1\right )}^{2}}\right )} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int x^{2} \operatorname {arcoth}\left (a x\right )\,{d x} \]________________________________________________________________________________________