\[ \int x \tanh ^{-1}(\tanh (a+b x))^{5/2} \, dx \]
Optimal antiderivative \[ \frac {2 x \arctanh \left (\tanh \left (b x +a \right )\right )^{\frac {7}{2}}}{7 b}-\frac {4 \arctanh \left (\tanh \left (b x +a \right )\right )^{\frac {9}{2}}}{63 b^{2}} \]
command
integrate(x*atanh(tanh(b*x+a))**(5/2),x)
Sympy 1.10.1 under Python 3.10.4 output
\[ \begin {cases} \frac {2 x \operatorname {atanh}^{\frac {7}{2}}{\left (\tanh {\left (a + b x \right )} \right )}}{7 b} - \frac {4 \operatorname {atanh}^{\frac {9}{2}}{\left (\tanh {\left (a + b x \right )} \right )}}{63 b^{2}} & \text {for}\: b \neq 0 \\\frac {x^{2} \operatorname {atanh}^{\frac {5}{2}}{\left (\tanh {\left (a \right )} \right )}}{2} & \text {otherwise} \end {cases} \]
Sympy 1.8 under Python 3.8.8 output
\[ \text {Timed out} \]________________________________________________________________________________________