From personal experience, the 4.4 or greater is the best choice in an X5. There are body changes from 2003 to 2004. I would suggest getting an 04. I found an 04 X5 4.4 for 24K (all packages). I'm sure if you offered 20K, they would sell it. I know you can find one.
Now, the Z4 is a totally different vehicle. You are comparing an SUV to a sporty 2 door sports car. First obvious observation...winter/bad weather. I know from experience that the Z4 is the worst handling when it comes to bad weather. The X5 is AWD and the best choice when it comes to bad weather. 2. Space. If you don't need much space...a Z4 is fine. X5 has more space. 3. Safety, I know all BMWs are safe, but realistically you are more safe in an SUV than a small sports car. 4. Insurance. An SUV is less expensive to insure than a small sports car. 5. Style. If you are a convertible person and love the outdoors, a Z4 has that option. Much better and stylish. Yes, an X5 has 4+1 wide sun roof, but hands down nothing compared to a convertible.
And, I would get the 2004 3.0 Sports package bare minimum. I personally love heated seats, so Cold Weather for me. And HK sound system for the Premium package for me too! Yes, you can find these for around 20K also.
Thats all I have for right now. Let me know if you need any other additional information.