\[ \int \frac {\left (a x+\sqrt {-b x+a^2 x^2}\right )^{3/4}}{\sqrt {-b x+a^2 x^2}} \, dx \]
Optimal antiderivative \[ \frac {4 \left (a x +\sqrt {a^{2} x^{2}-b x}\right )^{\frac {3}{4}}}{3 a}+\frac {2^{\frac {1}{4}} b^{\frac {3}{4}} \arctan \! \left (\frac {2^{\frac {1}{4}} a^{\frac {1}{4}} \left (a x +\sqrt {a^{2} x^{2}-b x}\right )^{\frac {1}{4}}}{b^{\frac {1}{4}}}\right )}{a^{\frac {7}{4}}}-\frac {2^{\frac {1}{4}} b^{\frac {3}{4}} \operatorname {arctanh}\! \left (\frac {2^{\frac {1}{4}} a^{\frac {1}{4}} \left (a x +\sqrt {a^{2} x^{2}-b x}\right )^{\frac {1}{4}}}{b^{\frac {1}{4}}}\right )}{a^{\frac {7}{4}}} \]
command
Int[(a*x + Sqrt[-(b*x) + a^2*x^2])^(3/4)/Sqrt[-(b*x) + a^2*x^2],x]
Rubi 4.17.3 under Mathematica 13.3.1 output
\[ \int \frac {\left (a x+\sqrt {-b x+a^2 x^2}\right )^{3/4}}{\sqrt {-b x+a^2 x^2}} \, dx \]
Rubi 4.16.1 under Mathematica 13.3.1 output
\[ \frac {4 \left (\sqrt {a^2 x^2-b x}+a x\right )^{3/4}}{3 a}+\frac {\sqrt [4]{2} b^{3/4} \arctan \left (\frac {\sqrt [4]{2} \sqrt [4]{a} \sqrt [4]{\sqrt {a^2 x^2-b x}+a x}}{\sqrt [4]{b}}\right )}{a^{7/4}}-\frac {\sqrt [4]{2} b^{3/4} \text {arctanh}\left (\frac {\sqrt [4]{2} \sqrt [4]{a} \sqrt [4]{\sqrt {a^2 x^2-b x}+a x}}{\sqrt [4]{b}}\right )}{a^{7/4}} \]