I may be wrong about the difference here but it's my understanding that a pedestrian always has the right of way in the sense that you must stop if one is in front of you, and cannot nudge or harass them. What's different is that that at a legal crosswalk, at least in Vermont, you must not only yield to a pedestrian but must stop to let him or her pass if they step off the curb. At a marked crosswalk, a pedestrian stepping onto the highway is equivalent to a red light.
That may bring a difference in liability if you hit one. A pedestrian stepping out into traffic outside a crosswalk runs some risk of failure to see them, or perhaps of a car not judging clearance correctly, and some excuse might prevail for the driver. But on a marked crosswalk, a pedestrian is expected to be able to step out and cross without having to dodge traffic, and a driver is expected to keep an eye out for anyone stepping off the curb at either side. Prudent pedestrians still take some care, of course, as some drivers are...well, you know, like the wrong end of the horse...but a commanding and imperious attitude is entirely acceptable when you motion them to stop or yell nasty things when they don't.