I have a 2 and a half year old. I can honestly say I don't think my SA affects him one little bit. He spends 2 days a week at daycare (strictly because he needs interaction with kids), he is well behaved, well looked after, well loved. I have barely any friends, so we don't go out and socialise much. I do have one best friend who adores him and who I feel totally comfortable with.
When it comes time for birthday parties and whatnot.. Well like you said, I'll just grin and bear it. Although I would hope my therapy sessions would have significantly improved my SA by then.