\[ \int \coth ^2(x) \left (a+b \coth ^2(x)\right )^{3/2} \, dx \]
Optimal antiderivative \[ \left (a +b \right )^{\frac {3}{2}} \arctanh \left (\frac {\coth \left (x \right ) \sqrt {a +b}}{\sqrt {a +b \left (\coth ^{2}\left (x \right )\right )}}\right )-\frac {\left (3 a^{2}+12 a b +8 b^{2}\right ) \arctanh \left (\frac {\coth \left (x \right ) \sqrt {b}}{\sqrt {a +b \left (\coth ^{2}\left (x \right )\right )}}\right )}{8 \sqrt {b}}-\frac {\left (5 a +4 b \right ) \coth \left (x \right ) \sqrt {a +b \left (\coth ^{2}\left (x \right )\right )}}{8}-\frac {b \left (\coth ^{3}\left (x \right )\right ) \sqrt {a +b \left (\coth ^{2}\left (x \right )\right )}}{4} \]
command
integrate(coth(x)^2*(a+b*coth(x)^2)^(3/2),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \text {output too large to display} \]
Fricas 1.3.7 via sagemath 9.3 output \[ \text {Timed out} \]