\[ \int x^2 \left (a x+b x^3\right )^{3/2} \, dx \]
Optimal antiderivative \[ \frac {2 x^{3} \left (b \,x^{3}+a x \right )^{\frac {3}{2}}}{15}-\frac {8 a^{3} \sqrt {b \,x^{3}+a x}}{231 b^{2}}+\frac {8 a^{2} x^{2} \sqrt {b \,x^{3}+a x}}{385 b}+\frac {4 a \,x^{4} \sqrt {b \,x^{3}+a x}}{55}+\frac {4 a^{\frac {15}{4}} \sqrt {\frac {\cos \left (4 \arctan \left (\frac {b^{\frac {1}{4}} \sqrt {x}}{a^{\frac {1}{4}}}\right )\right )}{2}+\frac {1}{2}}\, \EllipticF \left (\sin \left (2 \arctan \left (\frac {b^{\frac {1}{4}} \sqrt {x}}{a^{\frac {1}{4}}}\right )\right ), \frac {\sqrt {2}}{2}\right ) \left (\sqrt {a}+x \sqrt {b}\right ) \sqrt {x}\, \sqrt {\frac {b \,x^{2}+a}{\left (\sqrt {a}+x \sqrt {b}\right )^{2}}}}{231 \cos \left (2 \arctan \left (\frac {b^{\frac {1}{4}} \sqrt {x}}{a^{\frac {1}{4}}}\right )\right ) b^{\frac {9}{4}} \sqrt {b \,x^{3}+a x}} \]
command
integrate(x^2*(b*x^3+a*x)^(3/2),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \frac {2 \, {\left (20 \, a^{4} \sqrt {b} {\rm weierstrassPInverse}\left (-\frac {4 \, a}{b}, 0, x\right ) + {\left (77 \, b^{4} x^{6} + 119 \, a b^{3} x^{4} + 12 \, a^{2} b^{2} x^{2} - 20 \, a^{3} b\right )} \sqrt {b x^{3} + a x}\right )}}{1155 \, b^{3}} \]
Fricas 1.3.7 via sagemath 9.3 output
\[ {\rm integral}\left ({\left (b x^{5} + a x^{3}\right )} \sqrt {b x^{3} + a x}, x\right ) \]