\[ f(x) \left (2 a y(x)+b+y(x)^2\right )+y'(x)=0 \] ✓ Mathematica : cpu = 0.0624036 (sec), leaf count = 48
\[\left \{\left \{y(x)\to \sqrt {b-a^2} \tan \left (\sqrt {b-a^2} \left (\int _1^x -f(K[1]) \, dK[1]+c_1\right )\right )-a\right \}\right \}\]
✓ Maple : cpu = 0.073 (sec), leaf count = 35
\[ \left \{ y \left ( x \right ) =\tanh \left ( \sqrt {{a}^{2}-b} \left ( {\it \_C1}+\int \!f \left ( x \right ) \,{\rm d}x \right ) \right ) \sqrt {{a}^{2}-b}-a \right \} \]
\begin {align} y^{\prime }\left ( x\right ) +f\left ( x\right ) \left ( 2ay\left ( x\right ) +b+y^{2}\left ( x\right ) \right ) & =0\nonumber \\ y^{\prime }\left ( x\right ) & =-2af\left ( x\right ) y\left ( x\right ) -bf\left ( x\right ) -f\left ( x\right ) y^{2}\left ( x\right ) \nonumber \\ & =P\left ( x\right ) +Q\left ( x\right ) y+R\left ( x\right ) y^{2} \tag {1} \end {align}
This is Riccati first order non-linear ODE. \(P\left ( x\right ) =-bf\left ( x\right ) ,Q\left ( x\right ) =-2af\left ( x\right ) ,R\left ( x\right ) =-f\left ( x\right ) \).
Let \[ y\left ( x\right ) =-\frac {u^{\prime }\left ( x\right ) }{u\left ( x\right ) R\left ( x\right ) }=\frac {u^{\prime }\left ( x\right ) }{u\left ( x\right ) f\left ( x\right ) }\]
Hence
\[ y^{\prime }\left ( x\right ) =\frac {u^{\prime \prime }\left ( x\right ) }{u\left ( x\right ) f\left ( x\right ) }-\frac {\left ( u^{\prime }\left ( x\right ) \right ) ^{2}}{u^{2}\left ( x\right ) f\left ( x\right ) }-\frac {u^{\prime }\left ( x\right ) f^{\prime }\left ( x\right ) }{u\left ( x\right ) f^{2}\left ( x\right ) }\]
Equating this to RHS of (1) gives
\begin {align*} \frac {u^{\prime \prime }\left ( x\right ) }{u\left ( x\right ) f\left ( x\right ) }-\frac {\left ( u^{\prime }\left ( x\right ) \right ) ^{2}}{u^{2}\left ( x\right ) f\left ( x\right ) }-\frac {u^{\prime }\left ( x\right ) f^{\prime }\left ( x\right ) }{u\left ( x\right ) f^{2}\left ( x\right ) } & =-2af\left ( x\right ) y\left ( x\right ) -bf\left ( x\right ) -f\left ( x\right ) y^{2}\left ( x\right ) \\ & =-2af\left ( x\right ) \left [ \frac {u^{\prime }\left ( x\right ) }{u\left ( x\right ) f\left ( x\right ) }\right ] -bf\left ( x\right ) -f\left ( x\right ) \left [ \frac {u^{\prime }\left ( x\right ) }{u\left ( x\right ) f\left ( x\right ) }\right ] ^{2}\\ & =-2a\frac {u^{\prime }\left ( x\right ) }{u\left ( x\right ) }-bf\left ( x\right ) -\frac {u^{\prime }\left ( x\right ) ^{2}}{u^{2}\left ( x\right ) f\left ( x\right ) } \end {align*}
Simplifying
\begin {align*} u^{\prime \prime }\left ( x\right ) -\frac {\left ( u^{\prime }\left ( x\right ) \right ) ^{2}}{u\left ( x\right ) }-\frac {u^{\prime }\left ( x\right ) f^{\prime }\left ( x\right ) }{f\left ( x\right ) } & =-2au^{\prime }\left ( x\right ) f\left ( x\right ) -u\left ( x\right ) bf^{2}\left ( x\right ) -\frac {u^{\prime }\left ( x\right ) ^{2}}{u\left ( x\right ) }\\ u^{\prime \prime }\left ( x\right ) -\frac {u^{\prime }\left ( x\right ) f^{\prime }\left ( x\right ) }{f\left ( x\right ) } & =-2au^{\prime }\left ( x\right ) f\left ( x\right ) -u\left ( x\right ) bf^{2}\left ( x\right ) \\ u^{\prime \prime }\left ( x\right ) +u^{\prime }\left ( x\right ) \left ( -\frac {f^{\prime }\left ( x\right ) }{f\left ( x\right ) }+2af\left ( x\right ) \right ) +u\left ( x\right ) bf^{2}\left ( x\right ) & =0 \end {align*}
Second order ODE with variable coefficients. Since coefficients are variables and not constants, a power series method is the standard way to continue. When I tried solving this now pretending the coefficients are constants in time, using the standard auxiliary equation method, the solution did verify OK. I need to look more into this. For now, this is solved using standard method for solving second order ODE with constant coefficients.
\[ u\left ( x\right ) =C_{1}\exp \left ( \frac {\int f\left ( x\right ) \sqrt {-b}dx\left ( \sqrt {\frac {b-a^{2}}{b}}b+a\sqrt {-b}\right ) }{b}\right ) +C_{2}\exp \left ( \frac {\int f\left ( x\right ) \sqrt {-b}dx\left ( -\sqrt {\frac {b-a^{2}}{b}}b+a\sqrt {-b}\right ) }{b}\right ) \]
Hence
\begin {align*} u^{\prime }\left ( x\right ) & ={\frac {C_{1}\,f\left ( x\right ) \sqrt {-b}}{b}\left ( \sqrt {{\frac {-{a}^{2}+b}{b}}}b+\sqrt {-b}a\right ) e{^{{\frac {\int \!f\left ( x\right ) \sqrt {-b}\,\mathrm {d}x}{b}\left ( \sqrt {{\frac {-{a}^{2}+b}{b}}}b+\sqrt {-b}a\right ) }}}}\\ & +{\frac {C_{2}\,f\left ( x\right ) \sqrt {-b}}{b}\left ( -\sqrt {{\frac {-{a}^{2}+b}{b}}}b+\sqrt {-b}a\right ) e{^{{\frac {\int \!f\left ( x\right ) \sqrt {-b}\,\mathrm {d}x}{b}\left ( -\sqrt {{\frac {-{a}^{2}+b}{b}}}b+\sqrt {-b}a\right ) }}}} \end {align*}
Therefore
\begin {align*} y & =\frac {u^{\prime }\left ( x\right ) }{u\left ( x\right ) f\left ( x\right ) }\\ & =\frac {{\frac {C_{1}\,f\left ( x\right ) \sqrt {-b}}{b}\left ( \sqrt {{\frac {-{a}^{2}+b}{b}}}b+\sqrt {-b}a\right ) e{^{{\frac {\int \!f\left ( x\right ) \sqrt {-b}\,\mathrm {d}x}{b}\left ( \sqrt {{\frac {-{a}^{2}+b}{b}}}b+\sqrt {-b}a\right ) }}+}\frac {C_{2}\,f\left ( x\right ) \sqrt {-b}}{b}\left ( -\sqrt {{\frac {-{a}^{2}+b}{b}}}b+\sqrt {-b}a\right ) e{^{{\frac {\int \!f\left ( x\right ) \sqrt {-b}\,\mathrm {d}x}{b}\left ( -\sqrt {{\frac {-{a}^{2}+b}{b}}}b+\sqrt {-b}a\right ) }}}}}{f\left ( x\right ) \left [ C_{1}\exp \left ( \frac {\int f\left ( x\right ) \sqrt {-b}dx\left ( \sqrt {\frac {b-a^{2}}{b}}b+a\sqrt {-b}\right ) }{b}\right ) +C_{2}\exp \left ( \frac {\int f\left ( x\right ) \sqrt {-b}dx\left ( -\sqrt {\frac {b-a^{2}}{b}}b+a\sqrt {-b}\right ) }{b}\right ) \right ] } \end {align*}
Verification
restart; book:=diff(y(x),x)+f(x)*(2*a*y(x)+b+y(x)^2)=0; eqU:=diff(u(x),x$2)+diff(u(x),x)*(- diff(f(x),x)/f(x)+2*a*f(x))+u(x)*f(x)^2*b=0; solU:=dsolve(eqU,u(x)); my_sol:=diff(rhs(solU),x)/(rhs(solU)*f(x)); odetest(y(x)=my_sol,book); 0