\[ \int x^{5/2} (a+b x)^{3/2} (A+B x) \, dx \]
Optimal antiderivative \[ \frac {\left (12 A b -7 a B \right ) x^{\frac {7}{2}} \left (b x +a \right )^{\frac {3}{2}}}{60 b}+\frac {B \,x^{\frac {7}{2}} \left (b x +a \right )^{\frac {5}{2}}}{6 b}-\frac {a^{5} \left (12 A b -7 a B \right ) \arctanh \! \left (\frac {\sqrt {b}\, \sqrt {x}}{\sqrt {b x +a}}\right )}{512 b^{\frac {9}{2}}}-\frac {a^{3} \left (12 A b -7 a B \right ) x^{\frac {3}{2}} \sqrt {b x +a}}{768 b^{3}}+\frac {a^{2} \left (12 A b -7 a B \right ) x^{\frac {5}{2}} \sqrt {b x +a}}{960 b^{2}}+\frac {a \left (12 A b -7 a B \right ) x^{\frac {7}{2}} \sqrt {b x +a}}{160 b}+\frac {a^{4} \left (12 A b -7 a B \right ) \sqrt {x}\, \sqrt {b x +a}}{512 b^{4}} \]
command
integrate(x**(5/2)*(b*x+a)**(3/2)*(B*x+A),x)
Sympy 1.10.1 under Python 3.10.4 output
\[ \text {Timed out} \]
Sympy 1.8 under Python 3.8.8 output
\[ \text {output too large to display} \]