\[ \int (d \sec (e+f x))^{5/2} (a+b \tan (e+f x))^2 \, dx \]
Optimal antiderivative \[ \frac {18 a b \left (d \sec \left (f x +e \right )\right )^{\frac {5}{2}}}{35 f}+\frac {2 \left (7 a^{2}-2 b^{2}\right ) d \left (d \sec \left (f x +e \right )\right )^{\frac {3}{2}} \sin \left (f x +e \right )}{21 f}+\frac {2 \left (7 a^{2}-2 b^{2}\right ) d^{2} \sqrt {\frac {\cos \left (f x +e \right )}{2}+\frac {1}{2}}\, \EllipticF \left (\sin \left (\frac {f x}{2}+\frac {e}{2}\right ), \sqrt {2}\right ) \left (\sqrt {\cos }\left (f x +e \right )\right ) \sqrt {d \sec \left (f x +e \right )}}{21 \cos \left (\frac {f x}{2}+\frac {e}{2}\right ) f}+\frac {2 b \left (d \sec \left (f x +e \right )\right )^{\frac {5}{2}} \left (a +b \tan \left (f x +e \right )\right )}{7 f} \]
command
integrate((d*sec(f*x+e))^(5/2)*(a+b*tan(f*x+e))^2,x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \frac {-5 i \, \sqrt {2} {\left (7 \, a^{2} - 2 \, b^{2}\right )} d^{\frac {5}{2}} \cos \left (f x + e\right )^{3} {\rm weierstrassPInverse}\left (-4, 0, \cos \left (f x + e\right ) + i \, \sin \left (f x + e\right )\right ) + 5 i \, \sqrt {2} {\left (7 \, a^{2} - 2 \, b^{2}\right )} d^{\frac {5}{2}} \cos \left (f x + e\right )^{3} {\rm weierstrassPInverse}\left (-4, 0, \cos \left (f x + e\right ) - i \, \sin \left (f x + e\right )\right ) + 2 \, {\left (42 \, a b d^{2} \cos \left (f x + e\right ) + 5 \, {\left ({\left (7 \, a^{2} - 2 \, b^{2}\right )} d^{2} \cos \left (f x + e\right )^{2} + 3 \, b^{2} d^{2}\right )} \sin \left (f x + e\right )\right )} \sqrt {\frac {d}{\cos \left (f x + e\right )}}}{105 \, f \cos \left (f x + e\right )^{3}} \]
Fricas 1.3.7 via sagemath 9.3 output
\[ {\rm integral}\left ({\left (b^{2} d^{2} \sec \left (f x + e\right )^{2} \tan \left (f x + e\right )^{2} + 2 \, a b d^{2} \sec \left (f x + e\right )^{2} \tan \left (f x + e\right ) + a^{2} d^{2} \sec \left (f x + e\right )^{2}\right )} \sqrt {d \sec \left (f x + e\right )}, x\right ) \]