There is lots that can be done in terms of performance, although it depends on what and where you want performance. If your talking hp, then undoubtably the best for the buck would have to be an engine swap of some sort.
How is the condition of the car? (post some pics) If it's far from pristine original condition, then "I" say modify it as much as you want. I love e30's and I still want one, but theirs enough out there. With that said, the saturation of e30 & e36's only makes your e21 standout. Just don't expect to have a screamer unless you have the wallet to build one. An e21 is a fun car with or without a ton of hp, as it has it's own classic charm and quirks.
Modifying a car is not all about hp and I'm not even sure if that's where your focus is. You may have the budget to build a rocket, but theirs a few things that can be done if you are on one. You can definitely make the car run smooth, efficient and quickly with little money involved, but yet have a fun enjoyable ride.
Anyhow, checkout the e21 FAQ section on bimmerforums, theirs a lot of useful information to help you get started or plan your direction. But support yor own and keep active and updated on maxbimmer as well.
Good luck and congrats of the acquisition!