\[ \int x^5 \text {FresnelC}(b x) \sin \left (\frac {1}{2} b^2 \pi x^2\right ) \, dx \]
Optimal antiderivative \[ -\frac {4 x}{b^{5} \pi ^{3}}+\frac {x^{5}}{10 b \pi }+\frac {11 x \cos \left (b^{2} \pi \,x^{2}\right )}{8 b^{5} \pi ^{3}}+\frac {8 \cos \left (\frac {b^{2} \pi \,x^{2}}{2}\right ) \FresnelC \left (b x \right )}{b^{6} \pi ^{3}}-\frac {x^{4} \cos \left (\frac {b^{2} \pi \,x^{2}}{2}\right ) \FresnelC \left (b x \right )}{b^{2} \pi }+\frac {4 x^{2} \FresnelC \left (b x \right ) \sin \left (\frac {b^{2} \pi \,x^{2}}{2}\right )}{b^{4} \pi ^{2}}+\frac {x^{3} \sin \left (b^{2} \pi \,x^{2}\right )}{4 b^{3} \pi ^{2}}-\frac {43 \FresnelC \left (b x \sqrt {2}\right ) \sqrt {2}}{16 b^{6} \pi ^{3}} \]
command
integrate(x^5*fresnel_cos(b*x)*sin(1/2*b^2*pi*x^2),x, algorithm="fricas")
Fricas 1.3.8 (sbcl 2.2.11.debian) via sagemath 9.6 output
\[ \frac {8 \, \pi ^{2} b^{6} x^{5} + 220 \, b^{2} x \cos \left (\frac {1}{2} \, \pi b^{2} x^{2}\right )^{2} - 430 \, b^{2} x - 80 \, {\left (\pi ^{2} b^{5} x^{4} - 8 \, b\right )} \cos \left (\frac {1}{2} \, \pi b^{2} x^{2}\right ) \operatorname {C}\left (b x\right ) - 215 \, \sqrt {2} \sqrt {b^{2}} \operatorname {C}\left (\sqrt {2} \sqrt {b^{2}} x\right ) + 40 \, {\left (\pi b^{4} x^{3} \cos \left (\frac {1}{2} \, \pi b^{2} x^{2}\right ) + 8 \, \pi b^{3} x^{2} \operatorname {C}\left (b x\right )\right )} \sin \left (\frac {1}{2} \, \pi b^{2} x^{2}\right )}{80 \, \pi ^{3} b^{7}} \]
Fricas 1.3.7 via sagemath 9.3 output
\[ {\rm integral}\left (x^{5} {\rm fresnelc}\left (b x\right ) \sin \left (\frac {1}{2} \, \pi b^{2} x^{2}\right ), x\right ) \]