Pack the hole with grease. Find something that is a close fit (almost same diameter as the hole) and place it at the start of the hole. HIT it hard with the hammer. This will act like a hydraulic press and it should pop out.
This method is commonly used to get the pilot bearings out of the rear of the crankshaft. It works really well, but can me a little messy.
Good luck!! If this works I like my coffee with cream and sugar