I haven't had a problem removing drums. Yet...The bearing seems to be fairly tight to the spindle. I'd hope the center of the bearing isn't stuck or wedged on the spindle. Obviously you have to pull the drum off straight. Worst case get a slide hammer if you're really stuck. I've never tried backing the brakes off through the backing plate.
Good luck!