\[ \int \left (c+d x^2\right )^4 \coth ^{-1}(a x) \, dx \]
Optimal antiderivative \[ \frac {d \left (420 a^{6} c^{3}+378 a^{4} c^{2} d +180 a^{2} c \,d^{2}+35 d^{3}\right ) x^{2}}{630 a^{7}}+\frac {d^{2} \left (378 a^{4} c^{2}+180 a^{2} c d +35 d^{2}\right ) x^{4}}{1260 a^{5}}+\frac {d^{3} \left (36 a^{2} c +7 d \right ) x^{6}}{378 a^{3}}+\frac {d^{4} x^{8}}{72 a}+c^{4} x \,\mathrm {arccoth}\left (a x \right )+\frac {4 c^{3} d \,x^{3} \mathrm {arccoth}\left (a x \right )}{3}+\frac {6 c^{2} d^{2} x^{5} \mathrm {arccoth}\left (a x \right )}{5}+\frac {4 c \,d^{3} x^{7} \mathrm {arccoth}\left (a x \right )}{7}+\frac {d^{4} x^{9} \mathrm {arccoth}\left (a x \right )}{9}+\frac {\left (315 a^{8} c^{4}+420 a^{6} c^{3} d +378 a^{4} c^{2} d^{2}+180 a^{2} c \,d^{3}+35 d^{4}\right ) \ln \left (-a^{2} x^{2}+1\right )}{630 a^{9}} \]
command
integrate((d*x^2+c)^4*arccoth(a*x),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \text {output too large to display} \]
Giac 1.7.0 via sagemath 9.3 output \[ \int {\left (d x^{2} + c\right )}^{4} \operatorname {arcoth}\left (a x\right )\,{d x} \]_______________________________________________________