\[ -\sin (a x) \sin (b x)+y''(x)+y(x)=0 \] ✓ Mathematica : cpu = 0.590877 (sec), leaf count = 159
\[\left \{\left \{y(x)\to \frac {a^4 c_2 \sin (x)-2 a^2 b^2 c_2 \sin (x)-a^2 \sin (a x) \sin (b x)-2 a^2 c_2 \sin (x)+c_1 \left (a^4-2 a^2 \left (b^2+1\right )+\left (b^2-1\right )^2\right ) \cos (x)-b^2 \sin (a x) \sin (b x)+\sin (a x) \sin (b x)-2 a b \cos (a x) \cos (b x)+b^4 c_2 \sin (x)-2 b^2 c_2 \sin (x)+c_2 \sin (x)}{(a-b-1) (a-b+1) (a+b-1) (a+b+1)}\right \}\right \}\]
✓ Maple : cpu = 0.115 (sec), leaf count = 82
\[ \left \{ y \left ( x \right ) =\sin \left ( x \right ) {\it \_C2}+\cos \left ( x \right ) {\it \_C1}+{\frac {- \left ( a+b+1 \right ) \left ( a+b-1 \right ) \cos \left ( x \left ( a-b \right ) \right ) +\cos \left ( \left ( a+b \right ) x \right ) \left ( a-b+1 \right ) \left ( a-b-1 \right ) }{2\,{a}^{4}+ \left ( -4\,{b}^{2}-4 \right ) {a}^{2}+2\,{b}^{4}-4\,{b}^{2}+2}} \right \} \]
\begin {equation} y^{\prime \prime }+y=\sin ax\sin bx\tag {1} \end {equation} We start by solving the homogeneous equation \[ y^{\prime \prime }+y=0 \] Let \(y=e^{\lambda x}\), substitution in above gives\begin {align*} \lambda ^{2}e^{\lambda x}+e^{\lambda x} & =0\\ \lambda ^{2}+1 & =0 \end {align*}
Hence \(\lambda =\pm i\), therefore the solution is\begin {align*} y_{h} & =Ae^{ix}+Be^{-ix}\\ & =A\left ( \cos x+i\sin x\right ) +B\left ( \cos x-i\sin x\right ) \\ & =\cos x\left ( A+B\right ) +\sin x\left ( Ai-iB\right ) \\ & =\cos x\left ( A+B\right ) +\sin x\left ( i\left ( A-B\right ) \right ) \end {align*}
Let \(A+B=c_{1},i\left ( A-B\right ) =c_{2}\) hence\begin {align*} y_{h} & =c_{1}\cos x+c_{2}\sin x\\ & =y_{h}=c_{1}y_{1}+c_{2}y_{2} \end {align*}
Now we solve for the particular solution using variation of parameters.
\[ y_{p}=u_{1}y_{1}+u_{2}y_{2}\]
Wonskian is
\[ W=\begin {vmatrix} y_{1} & y_{2}\\ y_{1}^{\prime } & y_{2}^{\prime }\end {vmatrix} =\begin {vmatrix} \cos x & \sin x\\ -\sin x & \cos x \end {vmatrix} =\cos ^{2}x+\sin ^{2}x=1 \]
Hence, using \(f=\sin ax\sin bx\), which is the RHS of the ODE, and noting that \(a_{0}\) is the coefficient of \(y^{\prime \prime }\) which is one here, then
\begin {align*} u_{1} & =\int \frac {-y_{2}}{W}\frac {f}{a_{0}}dx=-\int \sin x\sin \left ( ax\right ) \sin \left ( bx\right ) dx\\ & =-\frac {1}{4}\left ( \frac {\cos \left ( \left ( a-b-1\right ) x\right ) }{a-b-1}-\frac {\cos \left ( \left ( a-b+1\right ) x\right ) }{a-b+1}-\frac {\cos \left ( \left ( a+b-1\right ) x\right ) }{a+b-1}+\frac {\cos \left ( \left ( a+b+1\right ) x\right ) }{a+b+1}\right ) \end {align*}
And
\begin {align*} u_{2} & =\int \frac {y_{1}}{W}\frac {f}{a_{0}}dx=-\int \cos x\sin \left ( ax\right ) \sin \left ( bx\right ) dx\\ & =\frac {1}{4}\left ( \frac {\sin \left ( \left ( a-b-1\right ) x\right ) }{a-b-1}+\frac {\sin \left ( \left ( a-b+1\right ) x\right ) }{a-b+1}-\frac {\sin \left ( \left ( a+b-1\right ) x\right ) }{a+b-1}-\frac {\sin \left ( \left ( a+b+1\right ) x\right ) }{a+b+1}\right ) \end {align*}
Since \(y_{p}=u_{1}\left ( x\right ) \cos x+u_{2}\left ( x\right ) \sin x\) then\begin {align*} y_{p} & =-\frac {1}{4}\left ( \frac {\cos \left ( \left ( a-b-1\right ) x\right ) }{a-b-1}-\frac {\cos \left ( \left ( a-b+1\right ) x\right ) }{a-b+1}-\frac {\cos \left ( \left ( a+b-1\right ) x\right ) }{a+b-1}+\frac {\cos \left ( \left ( a+b+1\right ) x\right ) }{a+b+1}\right ) \cos x\\ & +\frac {1}{4}\left ( \frac {\sin \left ( \left ( a-b-1\right ) x\right ) }{a-b-1}+\frac {\sin \left ( \left ( a-b+1\right ) x\right ) }{a-b+1}-\frac {\sin \left ( \left ( a+b-1\right ) x\right ) }{a+b-1}-\frac {\sin \left ( \left ( a+b+1\right ) x\right ) }{a+b+1}\right ) \sin x\\ & \\ & =\frac {1}{4}\left ( -\frac {\cos ^{2}\left ( \left ( a-b-1\right ) x\right ) }{a-b-1}+\frac {\sin ^{2}\left ( \left ( a-b-1\right ) x\right ) }{a-b-1}\right ) +\frac {1}{4}\left ( \frac {\cos ^{2}\left ( \left ( a-b+1\right ) x\right ) }{a-b+1}+\frac {\sin ^{2}\left ( \left ( a-b+1\right ) x\right ) }{a-b+1}\right ) \\ & +\frac {1}{4}\left ( \frac {\cos ^{2}\left ( \left ( a+b-1\right ) x\right ) }{a+b-1}-\frac {\sin ^{2}\left ( \left ( a+b-1\right ) x\right ) }{a+b-1}\right ) +\frac {1}{4}\left ( -\frac {\cos ^{2}\left ( \left ( a+b+1\right ) x\right ) }{a+b+1}-\frac {\sin ^{2}\left ( \left ( a+b+1\right ) x\right ) }{a+b+1}\right ) \\ & \\ & =\frac {1}{4}\left ( -\frac {\cos ^{2}\left ( \left ( a-b-1\right ) x\right ) }{a-b-1}+\frac {\sin ^{2}\left ( \left ( a-b-1\right ) x\right ) }{a-b-1}\right ) +\frac {1}{4}\left ( \frac {1}{a-b+1}\right ) \\ & +\frac {1}{4}\left ( \frac {\cos ^{2}\left ( \left ( a+b-1\right ) x\right ) }{a+b-1}-\frac {\sin ^{2}\left ( \left ( a+b-1\right ) x\right ) }{a+b-1}\right ) -\frac {1}{4}\left ( \frac {1}{a+b+1}\right ) \end {align*}
Let \(a-b-1=\alpha ,a+b-1=\beta \) then
\begin {align*} y_{p} & =\frac {1}{4}\left ( \frac {\sin ^{2}\left ( \alpha x\right ) -\cos ^{2}\left ( \alpha x\right ) }{\alpha }\right ) +\frac {1}{4}\left ( \frac {\cos ^{2}\left ( \beta x\right ) -\sin ^{2}\left ( \beta x\right ) }{\beta }\right ) +\frac {1}{4}\left ( \frac {1}{\alpha +2}\right ) -\frac {1}{4}\left ( \frac {1}{\beta +2}\right ) \\ & =\frac {1}{4}\left ( \frac {\sin ^{2}\left ( \alpha x\right ) -\cos ^{2}\left ( \alpha x\right ) }{\alpha }+\frac {\cos ^{2}\left ( \beta x\right ) -\sin ^{2}\left ( \beta x\right ) }{\beta }+\frac {1}{\alpha +2}-\frac {1}{\beta +2}\right ) \end {align*}
Therefore, the full solution is \begin {align*} y & =y_{h}+y_{p}\\ & =c_{1}\cos x+c_{2}\sin x+\frac {1}{4}\left ( \frac {\sin ^{2}\left ( \alpha x\right ) -\cos ^{2}\left ( \alpha x\right ) }{\alpha }+\frac {\cos ^{2}\left ( \beta x\right ) -\sin ^{2}\left ( \beta x\right ) }{\beta }+\frac {1}{\alpha +2}-\frac {1}{\beta +2}\right ) \end {align*}
I made mistake. Need to go over it again. I do not see it now. Maple does not verify.
restart; ode:=diff(diff(y(x),x),x)+y(x)-sin(a*x)*sin(b*x)=0; alpha:=a-b-1; beta:=a+b-1; yp:=1/4*(1/alpha* (sin(alpha*x)^2-cos(alpha*x)^2)+1/beta* (cos(beta*x)^2-sin(beta*x)^2))+(1/4)*(1/(alpha+2)-1/(beta+2)); y0:=yp+_C1*sin(x)+_C2*cos(x); not zero