Congrats on returning from the dark side. I can't speak to the merits of the Z4, but have experience with the Boxster and Porsche's generally.
Back in the day, our garage contained an E36 M3, and a Boxter S. Both different, (to be honest I preferred the M3 as an all rounder), but for a fun summer car, the Boxster is much sharper and more responsive (though no faster). The mid-engined layout meant extraordinary balance, and the engine loves to rev. More importantly, the Porsche was bulletproof - through 5 years and about 75000 kms, there was not one issue - regular services and tires - that's it - and its not as though it wasn't being driven hard (two kids actually learned to drive stick on that car, so the clutch endured a lot).
Currently the cars have been upgraded to an E46 M3 (ZHP) and a 911 Turbo, and again, the Porsche has not presented a single issue (despite a few track days) beyond regular service and tires (though its only run about 70 000 kms). Porsche builds cars to last, and more than that, are built to be thrashed, regularly, and soldier on. (Just make sure you don't miss a service - then all bets are off).
Either choice would be fine I'm sure, but if it were my money, I'd be heading straight to Porsche.
Hope this helps.
Edit: none of the above comments apply to the Cayenne - step mother had one a few years back and other than the Mercedes ML, it was the worst car I've ever come across.