\[ \int x^3 \tanh ^{-1}(\tanh (a+b x))^{5/2} \, dx \]
Optimal antiderivative \[ \frac {2 x^{3} \arctanh \left (\tanh \left (b x +a \right )\right )^{\frac {7}{2}}}{7 b}-\frac {4 x^{2} \arctanh \left (\tanh \left (b x +a \right )\right )^{\frac {9}{2}}}{21 b^{2}}+\frac {16 x \arctanh \left (\tanh \left (b x +a \right )\right )^{\frac {11}{2}}}{231 b^{3}}-\frac {32 \arctanh \left (\tanh \left (b x +a \right )\right )^{\frac {13}{2}}}{3003 b^{4}} \]
command
integrate(x**3*atanh(tanh(b*x+a))**(5/2),x)
Sympy 1.10.1 under Python 3.10.4 output
\[ \begin {cases} \frac {2 x^{3} \operatorname {atanh}^{\frac {7}{2}}{\left (\tanh {\left (a + b x \right )} \right )}}{7 b} - \frac {4 x^{2} \operatorname {atanh}^{\frac {9}{2}}{\left (\tanh {\left (a + b x \right )} \right )}}{21 b^{2}} + \frac {16 x \operatorname {atanh}^{\frac {11}{2}}{\left (\tanh {\left (a + b x \right )} \right )}}{231 b^{3}} - \frac {32 \operatorname {atanh}^{\frac {13}{2}}{\left (\tanh {\left (a + b x \right )} \right )}}{3003 b^{4}} & \text {for}\: b \neq 0 \\\frac {x^{4} \operatorname {atanh}^{\frac {5}{2}}{\left (\tanh {\left (a \right )} \right )}}{4} & \text {otherwise} \end {cases} \]
Sympy 1.8 under Python 3.8.8 output
\[ \text {Timed out} \]________________________________________________________________________________________