Any idea what is the problem?
Yes. The equivalence of your formula 1 and formula 2 is based on the assumption that both peaks are Gaussian*, which your second peak clearly is not. I don't know how MassHunter is making its calculations, but I suspect that it's actually measuring the dispersion (σ) in both distributions and using that as the basis. No matter which way you do it, the results will be virtually meaningless and can only be improved by fixing the peak shape problem. You can't compute your way around bad chromatography.
*To be pedantic about it: the "ideal" shape of a chromatographic peak is approximated by the normal distribution ("Gaussian" or "bell-shaped-curve"). The dispersion in that distribution is measured by the variance, σ^2 (f this were a statistics course, we would be talking about the standard deviation, σ). For the normal distribution, what we think of as "baseline" width is equal to 4*σ, and the width-at-half-height is equal to 2.35*σ. In fact, I could look up the width at any fraction of the maximum height in a table of the normal distribution.
A quick look at your figure shows that the second peak is far from being Gaussian, and I suspect that the first peak also is significantly non-Gaussian. As I said, I don't know the details of how MassHunter does its calculation, but since it has access to the raw data, in principle the dispersion can be measured instead of estimated. Given the actual dispersions (σ1 and σ2) and the retention times (tr1 and tr2), the resolution can be calculated as {(tr1 - tr2)/[2*(σ1 + σ2)]}.