Driving used to always make me anxious, to the point I'd feel nauseous and I'd tremble. I hated it. So much that I didn't even bother practicing up until I was 18, almost 19, and even then I was often forced by my mother because I needed my license. You may not see it now, but having a license is incredibly useful (and important), especially the older you get.
I've had my license now for 2 years and I'm happy to have it. Driving still makes me a little anxious, I think it always will, but it can also be relaxing when you get to driving in rural areas where you're the only one on the road. The key to getting over that anxiety is to practice, practice, practice. You've probably heard that before, but it's true.