\[ \int e^{3 \coth ^{-1}(a x)} x^2 \, dx \]
Optimal antiderivative \[ \frac {11 \arctanh \! \left (\sqrt {1-\frac {1}{a^{2} x^{2}}}\right )}{2 a^{3}}-\frac {4 \sqrt {1-\frac {1}{a^{2} x^{2}}}}{a^{2} \left (a -\frac {1}{x}\right )}+\frac {14 x \sqrt {1-\frac {1}{a^{2} x^{2}}}}{3 a^{2}}+\frac {3 x^{2} \sqrt {1-\frac {1}{a^{2} x^{2}}}}{2 a}+\frac {x^{3} \sqrt {1-\frac {1}{a^{2} x^{2}}}}{3} \]
command
integrate(1/((a*x-1)/(a*x+1))^(3/2)*x^2,x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \text {could not integrate} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \frac {1}{6} \, a {\left (\frac {33 \, \log \left (\sqrt {\frac {a x - 1}{a x + 1}} + 1\right )}{a^{4}} - \frac {33 \, \log \left ({\left | \sqrt {\frac {a x - 1}{a x + 1}} - 1 \right |}\right )}{a^{4}} - \frac {24}{a^{4} \sqrt {\frac {a x - 1}{a x + 1}}} + \frac {2 \, {\left (\frac {52 \, {\left (a x - 1\right )} \sqrt {\frac {a x - 1}{a x + 1}}}{a x + 1} - \frac {21 \, {\left (a x - 1\right )}^{2} \sqrt {\frac {a x - 1}{a x + 1}}}{{\left (a x + 1\right )}^{2}} - 39 \, \sqrt {\frac {a x - 1}{a x + 1}}\right )}}{a^{4} {\left (\frac {a x - 1}{a x + 1} - 1\right )}^{3}}\right )} \]