\[ \int \left (c+d x^2\right )^2 \coth ^{-1}(a x) \, dx \]
Optimal antiderivative \[ \frac {d \left (10 a^{2} c +3 d \right ) x^{2}}{30 a^{3}}+\frac {d^{2} x^{4}}{20 a}+c^{2} x \,\mathrm {arccoth}\left (a x \right )+\frac {2 c d \,x^{3} \mathrm {arccoth}\left (a x \right )}{3}+\frac {d^{2} x^{5} \mathrm {arccoth}\left (a x \right )}{5}+\frac {\left (15 a^{4} c^{2}+10 a^{2} c d +3 d^{2}\right ) \ln \left (-a^{2} x^{2}+1\right )}{30 a^{5}} \]
command
integrate((d*x^2+c)^2*arccoth(a*x),x, algorithm="giac")
Giac 1.9.0-11 via sagemath 9.6 output
\[ \frac {1}{15} \, a {\left (\frac {{\left (15 \, a^{4} c^{2} + 10 \, a^{2} c d + 3 \, d^{2}\right )} \log \left (\frac {{\left | a x + 1 \right |}}{{\left | a x - 1 \right |}}\right )}{a^{6}} - \frac {{\left (15 \, a^{4} c^{2} + 10 \, a^{2} c d + 3 \, d^{2}\right )} \log \left ({\left | \frac {a x + 1}{a x - 1} - 1 \right |}\right )}{a^{6}} + \frac {4 \, {\left (\frac {{\left (5 \, a^{2} c d + 3 \, d^{2}\right )} {\left (a x + 1\right )}^{3}}{{\left (a x - 1\right )}^{3}} - \frac {{\left (10 \, a^{2} c d + 3 \, d^{2}\right )} {\left (a x + 1\right )}^{2}}{{\left (a x - 1\right )}^{2}} + \frac {{\left (5 \, a^{2} c d + 3 \, d^{2}\right )} {\left (a x + 1\right )}}{a x - 1}\right )}}{a^{6} {\left (\frac {a x + 1}{a x - 1} - 1\right )}^{4}} + \frac {{\left (\frac {15 \, {\left (a x + 1\right )}^{4} a^{4} c^{2}}{{\left (a x - 1\right )}^{4}} - \frac {60 \, {\left (a x + 1\right )}^{3} a^{4} c^{2}}{{\left (a x - 1\right )}^{3}} + \frac {90 \, {\left (a x + 1\right )}^{2} a^{4} c^{2}}{{\left (a x - 1\right )}^{2}} - \frac {60 \, {\left (a x + 1\right )} a^{4} c^{2}}{a x - 1} + 15 \, a^{4} c^{2} + \frac {30 \, {\left (a x + 1\right )}^{4} a^{2} c d}{{\left (a x - 1\right )}^{4}} - \frac {60 \, {\left (a x + 1\right )}^{3} a^{2} c d}{{\left (a x - 1\right )}^{3}} + \frac {40 \, {\left (a x + 1\right )}^{2} a^{2} c d}{{\left (a x - 1\right )}^{2}} - \frac {20 \, {\left (a x + 1\right )} a^{2} c d}{a x - 1} + 10 \, a^{2} c d + \frac {15 \, {\left (a x + 1\right )}^{4} d^{2}}{{\left (a x - 1\right )}^{4}} + \frac {30 \, {\left (a x + 1\right )}^{2} d^{2}}{{\left (a x - 1\right )}^{2}} + 3 \, d^{2}\right )} \log \left (-\frac {\frac {\frac {{\left (a x + 1\right )} a}{a x - 1} - a}{a {\left (\frac {a x + 1}{a x - 1} + 1\right )}} + 1}{\frac {\frac {{\left (a x + 1\right )} a}{a x - 1} - a}{a {\left (\frac {a x + 1}{a x - 1} + 1\right )}} - 1}\right )}{a^{6} {\left (\frac {a x + 1}{a x - 1} - 1\right )}^{5}}\right )} \]
Giac 1.7.0 via sagemath 9.3 output
\[ \int {\left (d x^{2} + c\right )}^{2} \operatorname {arcoth}\left (a x\right )\,{d x} \]________________________________________________________________________________________