\[ y''(x)=y(x) \left (-\csc ^2(x)\right ) \left (-\left (a^2 b^2-(a+1)^2\right ) \sin ^2(x)-a (a+1) b \sin (2 x)-(a-1) a\right ) \] ✗ Mathematica : cpu = 201.201 (sec), leaf count = 0 , could not solve
DSolve[Derivative[2][y][x] == -(Csc[x]^2*(-((-1 + a)*a) - (-(1 + a)^2 + a^2*b^2)*Sin[x]^2 - a*(1 + a)*b*Sin[2*x])*y[x]), y[x], x]
✓ Maple : cpu = 1.825 (sec), leaf count = 179
\[ \left \{ y \left ( x \right ) ={1{{\rm e}^{\int \!{\frac {1}{\sin \left ( 2\,x \right ) \left ( b\sin \left ( 2\,x \right ) +\cos \left ( 2\,x \right ) +1 \right ) } \left ( 2\, \left ( \left ( a+1 \right ) \cos \left ( 2\,x \right ) +a+1/2 \right ) b\sin \left ( 2\,x \right ) - \left ( \cos \left ( 2\,x \right ) +1 \right ) \left ( \left ( a{b}^{2}-a-2 \right ) \cos \left ( 2\,x \right ) -a{b}^{2}-a+1 \right ) \right ) }\,{\rm d}x}} \left ( \int \!-2\,{{\rm e}^{-2\,\int \!{\frac {2\, \left ( \left ( a+1 \right ) \cos \left ( 2\,x \right ) +a+1/2 \right ) b\sin \left ( 2\,x \right ) - \left ( \cos \left ( 2\,x \right ) +1 \right ) \left ( \left ( a{b}^{2}-a-2 \right ) \cos \left ( 2\,x \right ) -a{b}^{2}-a+1 \right ) }{\sin \left ( 2\,x \right ) \left ( b\sin \left ( 2\,x \right ) +\cos \left ( 2\,x \right ) +1 \right ) }}\,{\rm d}x}}\sin \left ( 2\,x \right ) \,{\rm d}x{\it \_C2}+{\it \_C1} \right ) {\frac {1}{\sqrt {\sin \left ( 2\,x \right ) }}}} \right \} \]