\[ \int x^2 \sinh ^{-1}\left (a x^2\right ) \, dx \]
Optimal antiderivative \[ \frac {x^{3} \arcsinh \left (a \,x^{2}\right )}{3}-\frac {2 x \sqrt {a^{2} x^{4}+1}}{9 a}+\frac {\left (a \,x^{2}+1\right ) \sqrt {\frac {\cos \left (4 \arctan \left (x \sqrt {a}\right )\right )}{2}+\frac {1}{2}}\, \EllipticF \left (\sin \left (2 \arctan \left (x \sqrt {a}\right )\right ), \frac {\sqrt {2}}{2}\right ) \sqrt {\frac {a^{2} x^{4}+1}{\left (a \,x^{2}+1\right )^{2}}}}{9 \cos \left (2 \arctan \left (x \sqrt {a}\right )\right ) a^{\frac {3}{2}} \sqrt {a^{2} x^{4}+1}} \]
command
integrate(x^2*arcsinh(a*x^2),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \frac {3 \, a x^{3} \log \left (a x^{2} + \sqrt {a^{2} x^{4} + 1}\right ) + 2 \, a \left (-\frac {1}{a^{2}}\right )^{\frac {3}{4}} {\rm ellipticF}\left (\frac {\left (-\frac {1}{a^{2}}\right )^{\frac {1}{4}}}{x}, -1\right ) - 2 \, \sqrt {a^{2} x^{4} + 1} x}{9 \, a} \]
Fricas 1.3.7 via sagemath 9.3 output
\[ {\rm integral}\left (x^{2} \operatorname {arsinh}\left (a x^{2}\right ), x\right ) \]