I know what you mean. I think the dilemma of "doing and not doing" is what really damages bully victims. Should I hit them or let them go like Jesus Christ? I don't think that there is an easy answer. The ideal situation would have been to have hit them then, when they did those things. But maybe you weren't confident enough or strong enough. Maybe there was a whole group attacking you. So you didn't do anything.
All I can say is whatever you do, don't take out all your frustration on innocent people. That would just be terrible.