\[ \int x^{3/2} (a+b x)^{5/2} (A+B x) \, dx \]
Optimal antiderivative \[ \frac {a \left (12 A b -5 a B \right ) x^{\frac {5}{2}} \left (b x +a \right )^{\frac {3}{2}}}{96 b}+\frac {\left (12 A b -5 a B \right ) x^{\frac {5}{2}} \left (b x +a \right )^{\frac {5}{2}}}{60 b}+\frac {B \,x^{\frac {5}{2}} \left (b x +a \right )^{\frac {7}{2}}}{6 b}+\frac {a^{5} \left (12 A b -5 a B \right ) \arctanh \! \left (\frac {\sqrt {b}\, \sqrt {x}}{\sqrt {b x +a}}\right )}{512 b^{\frac {7}{2}}}+\frac {a^{3} \left (12 A b -5 a B \right ) x^{\frac {3}{2}} \sqrt {b x +a}}{768 b^{2}}+\frac {a^{2} \left (12 A b -5 a B \right ) x^{\frac {5}{2}} \sqrt {b x +a}}{192 b}-\frac {a^{4} \left (12 A b -5 a B \right ) \sqrt {x}\, \sqrt {b x +a}}{512 b^{3}} \]
command
integrate(x**(3/2)*(b*x+a)**(5/2)*(B*x+A),x)
Sympy 1.10.1 under Python 3.10.4 output
\[ \text {Timed out} \]
Sympy 1.8 under Python 3.8.8 output
\[ \text {output too large to display} \]