As soon as you choose photoshop to refine the edge, a window will enable the option’s preview mode.
At first glance, you might become overwhelmed by the options the window contains but don’t be. You need to change the value of a couple of sliders (Smooth, Feather, Contrast) from these options and nothing else.
To remove the rough edges increase the value of the Smooth slider and see the changes. When you get the smoothness you have wanted, increase the Feather slider to smooth transition.
You must have got the smooth edge you are looking for, but making the edge blend with its background, uses the Contrast slider.