My father has an '04 BMW M5, and my brother has a '00 M3, I drive a '00 Z3 2.3 , my dad also drives a '05 X5 4.4 and my mom an '01 525it.
After driving all of these cars, my favorite is the M5.
There is just nothing like driving something that looks like an average 5 series, but packs such a punch. The M3 is a blast to drive, but the 04 M5 looked better than the 00 M3.
As for the "bangle butt" on the E 60 M5, I think it's ugly as hell.
If I were in your place, I would either get an 05 M5 or an 00 or later M3.
The M3 is still an excellent car, it's just not quite as refined as the M5. Two door cars are easier to park and such, but rear passengers will not be happy in an M3.
Here's a pic of our collection of Ms :-)