My personal response is, I don't care. What you guys do is irrelevant to me, do what you want.
Howeevveerr... when people mention age differences, the issue isn't years. Its emotional maturity, sexual maturity and life experience. There's laws preventing these things in most states and countries for a reason - they don't just make them up for fun.
If you were 20, and he was 24... there'd be no issue.
But the difference between a 16 yr old and a 20 year old is pretty big.
This type of thing has happened where Ive lived before and the people are always viewed badly - you'd be viewed as pretty immature (or poorly educated/ from a lower socioeconomic background, usually) to not be able to see there's an issue (and possibly a bit 'promiscuous', because in terms of development he *should* be well and truly a much more sexual being than yourself at the moment, and yet, you're dating him), and he as a sex offender.
If it was love, neither of you would have a problem waiting. If its not, then you'd have to wonder why he is dating you, and vice versa.
Again, I'm a big believer in do whatever you want... but the above is something to think about - the laws aren't there for nothing, and people don't form these views from thin air.
Also, there's nothing wrong with telling people how you feel... they should be supportive of *you*, though they may not be supportive of taking your feelings any further. You should still be able to talk about it.