\[ \int x^{5/2} \tanh ^{-1}(\tanh (a+b x))^3 \, dx \]
Optimal antiderivative \[ -\frac {32 b^{3} x^{\frac {13}{2}}}{3003}+\frac {16 b^{2} x^{\frac {11}{2}} \arctanh \left (\tanh \left (b x +a \right )\right )}{231}-\frac {4 b \,x^{\frac {9}{2}} \arctanh \left (\tanh \left (b x +a \right )\right )^{2}}{21}+\frac {2 x^{\frac {7}{2}} \arctanh \left (\tanh \left (b x +a \right )\right )^{3}}{7} \]
command
integrate(x**(5/2)*atanh(tanh(b*x+a))**3,x)
Sympy 1.10.1 under Python 3.10.4 output
\[ - \frac {32 b^{3} x^{\frac {13}{2}}}{3003} + \frac {16 b^{2} x^{\frac {11}{2}} \operatorname {atanh}{\left (\tanh {\left (a + b x \right )} \right )}}{231} - \frac {4 b x^{\frac {9}{2}} \operatorname {atanh}^{2}{\left (\tanh {\left (a + b x \right )} \right )}}{21} + \frac {2 x^{\frac {7}{2}} \operatorname {atanh}^{3}{\left (\tanh {\left (a + b x \right )} \right )}}{7} \]
Sympy 1.8 under Python 3.8.8 output
\[ \text {Timed out} \]________________________________________________________________________________________