\[ y''(x)=y(x) \left (-\csc ^2(x)\right ) \left (-\left (a^2 b^2-(a+1)^2\right ) \sin ^2(x)-a (a+1) b \sin (2 x)-(a-1) a\right ) \] ✗ Mathematica : cpu = 200.569 (sec), leaf count = 0 , could not solve
DSolve[Derivative[2][y][x] == -(Csc[x]^2*(-((-1 + a)*a) - (-(1 + a)^2 + a^2*b^2)*Sin[x]^2 - a*(1 + a)*b*Sin[2*x])*y[x]), y[x], x]
✓ Maple : cpu = 1.767 (sec), leaf count = 203
\[ \left \{ y \left ( x \right ) ={1{{\rm e}^{\int \!{\frac { \left ( a{b}^{2}-a-2 \right ) \left ( \cos \left ( 2\,x \right ) \right ) ^{2}+ \left ( -2\,b \left ( a+1 \right ) \sin \left ( 2\,x \right ) -2\,a-1 \right ) \cos \left ( 2\,x \right ) + \left ( -2\,a-1 \right ) b\sin \left ( 2\,x \right ) -a{b}^{2}-a+1}{ \left ( \cos \left ( 2\,x \right ) +1 \right ) \left ( b\cos \left ( 2\,x \right ) -\sin \left ( 2\,x \right ) -b \right ) }}\,{\rm d}x}} \left ( \int \!-2\,{{\rm e}^{-2\,\int \!{\frac { \left ( a{b}^{2}-a-2 \right ) \left ( \cos \left ( 2\,x \right ) \right ) ^{2}+ \left ( -2\,b \left ( a+1 \right ) \sin \left ( 2\,x \right ) -2\,a-1 \right ) \cos \left ( 2\,x \right ) + \left ( -2\,a-1 \right ) b\sin \left ( 2\,x \right ) -a{b}^{2}-a+1}{ \left ( \cos \left ( 2\,x \right ) +1 \right ) \left ( b\cos \left ( 2\,x \right ) -\sin \left ( 2\,x \right ) -b \right ) }}\,{\rm d}x}}\sin \left ( 2\,x \right ) \,{\rm d}x{\it \_C2}+{\it \_C1} \right ) {\frac {1}{\sqrt {\sin \left ( 2\,x \right ) }}}} \right \} \]