\[ \int x^2 \coth ^{-1}(\tanh (a+b x)) \, dx \]
Optimal antiderivative \[ -\frac {b \,x^{4}}{12}+\frac {x^{3} \mathrm {arccoth}\left (\tanh \left (b x +a \right )\right )}{3} \]
command
integrate(x^2*arccoth(tanh(b*x+a)),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ -\frac {1}{12} \, b x^{4} + \frac {1}{6} \, x^{3} \log \left (-\frac {\frac {e^{\left (2 \, b x + 2 \, a\right )} + 1}{e^{\left (2 \, b x + 2 \, a\right )} - 1} + 1}{\frac {e^{\left (2 \, b x + 2 \, a\right )} + 1}{e^{\left (2 \, b x + 2 \, a\right )} - 1} - 1}\right ) \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int x^{2} \operatorname {arcoth}\left (\tanh \left (b x + a\right )\right )\,{d x} \]________________________________________________________________________________________