\[ \int x^5 \cos \left (\frac {1}{2} b^2 \pi x^2\right ) \text {FresnelC}(b x) \, dx \]
Optimal antiderivative \[ -\frac {2 x^{3}}{3 b^{3} \pi ^{2}}+\frac {x^{3} \cos \left (b^{2} \pi \,x^{2}\right )}{4 b^{3} \pi ^{2}}+\frac {4 x^{2} \cos \left (\frac {b^{2} \pi \,x^{2}}{2}\right ) \FresnelC \left (b x \right )}{b^{4} \pi ^{2}}-\frac {8 \FresnelC \left (b x \right ) \sin \left (\frac {b^{2} \pi \,x^{2}}{2}\right )}{b^{6} \pi ^{3}}+\frac {x^{4} \FresnelC \left (b x \right ) \sin \left (\frac {b^{2} \pi \,x^{2}}{2}\right )}{b^{2} \pi }-\frac {11 x \sin \left (b^{2} \pi \,x^{2}\right )}{8 b^{5} \pi ^{3}}+\frac {43 \,\mathrm {S}\left (b x \sqrt {2}\right ) \sqrt {2}}{16 b^{6} \pi ^{3}} \]
command
integrate(x^5*cos(1/2*b^2*pi*x^2)*fresnel_cos(b*x),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \frac {24 \, \pi b^{4} x^{3} \cos \left (\frac {1}{2} \, \pi b^{2} x^{2}\right )^{2} - 44 \, \pi b^{4} x^{3} + 192 \, \pi b^{3} x^{2} \cos \left (\frac {1}{2} \, \pi b^{2} x^{2}\right ) \operatorname {C}\left (b x\right ) + 129 \, \sqrt {2} \sqrt {b^{2}} \operatorname {S}\left (\sqrt {2} \sqrt {b^{2}} x\right ) - 12 \, {\left (11 \, b^{2} x \cos \left (\frac {1}{2} \, \pi b^{2} x^{2}\right ) - 4 \, {\left (\pi ^{2} b^{5} x^{4} - 8 \, b\right )} \operatorname {C}\left (b x\right )\right )} \sin \left (\frac {1}{2} \, \pi b^{2} x^{2}\right )}{48 \, \pi ^{3} b^{7}} \]
Fricas 1.3.7 via sagemath 9.3 output
\[ {\rm integral}\left (x^{5} \cos \left (\frac {1}{2} \, \pi b^{2} x^{2}\right ) {\rm fresnelc}\left (b x\right ), x\right ) \]