I failed my first year of university because of exactly that
So I quit, and did it externally (correspondence / online / distance ed, whatever you want to call it), and have now have a few degrees and have "attended" some of the best uni's in the world.
Its ok now, but at the time it was pretty horrible, not just the social situation but failing because of it too.
You don't need to be on campus, there's more than one way to skin a cat, as they say. this isn't the 1980's anymore. I stupidly listened to other people at the time who didnt know any better.
If it works, great, but if not, dont worry too much, you can still do it other ways.