\[ \int \frac {\cot ^3(c+d x) (A+B \tan (c+d x))}{(a+b \tan (c+d x))^{3/2}} \, dx \]
Optimal antiderivative \[ \frac {\left (8 a^{2} A -15 A \,b^{2}+12 a b B \right ) \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {a}}\right )}{4 a^{\frac {7}{2}} d}-\frac {\left (-i B +A \right ) \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {-i b +a}}\right )}{\left (-i b +a \right )^{\frac {3}{2}} d}-\frac {\left (i B +A \right ) \arctanh \left (\frac {\sqrt {a +b \tan \left (d x +c \right )}}{\sqrt {i b +a}}\right )}{\left (i b +a \right )^{\frac {3}{2}} d}+\frac {b \left (7 A \,a^{2} b +15 A \,b^{3}-4 a^{3} B -12 B a \,b^{2}\right )}{4 a^{3} \left (a^{2}+b^{2}\right ) d \sqrt {a +b \tan \left (d x +c \right )}}+\frac {\left (5 A b -4 B a \right ) \cot \left (d x +c \right )}{4 a^{2} d \sqrt {a +b \tan \left (d x +c \right )}}-\frac {A \left (\cot ^{2}\left (d x +c \right )\right )}{2 a d \sqrt {a +b \tan \left (d x +c \right )}} \]
command
integrate(cot(d*x+c)^3*(A+B*tan(d*x+c))/(a+b*tan(d*x+c))^(3/2),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \text {output too large to display} \]
Fricas 1.3.7 via sagemath 9.3 output \[ \text {Timed out} \]