Yeah, I don't understand the difference between 'nice' and 'polite' either - maybe you could ask her to elaborate? (both could be true: either she could see you as too 'friendly' or 'too cold/disinterested', or she could be just a bit 'loony' or after some extra attention herself, hmm..)
It could also be about tone of voice or body proximity... Or maybe she was from a different culture with different expectations...?
If you were bored there (or shown any 'negative' emotions or eg thought she was a ninny and it showed on your face) it could also maybe come across as 'not polite'..
So maybe it's good to have a 'shema' for challenging people too?
About learning 'small talk':
You might try to 'mimic' what some other successful people there do, who get along with customers well? You could even maybe say 'sorry, I'm a little bit shy, and I'm still learning, I am trying to improve' or something like that? (?) (You could maybe even ask others to give you tips or how they approach situations or how they see things or think of customers?)
If you have trouble ad libbing, maybe you could write down a list of some potential stuff to say that seems to come across well with most (or also observe who they use it on - younger or older folk, men or women, etc).
Some 'shop speak' is very boring if you're used to intellectual stimulation and 'real talk' - there's stuff about weather, or observations about unnecessary/obvious things (if you read intro of Hitchhiker's Guide to the Galaxy you'll laugh at what is said there, so true) or 'description of situation' or small tiny compliments (if they look good or have made an effort/new haircut/new clothes and they are a repeat customer etc - this depends somwehat on the shop too, does it sell clothes or other things, or you can talk about new stuff in shop etc. or connect a few of those (for example, 'Wow, great outfit, we just got new jackets in, the black one would go together great with this.' (you bring the jacket) 'What do you think?' or something like that? though this would be for a repeat customer maybe, and more 'friendly' than 'polite')
It's important to focus on what the person wants, how fast, etc. (eg a young girl all made up in glamorous way will probably want to look good and be admired etc..) someone who's in a hurry will just want to grab what they need and run.. If it's raining outside and they buy an umbrella, you can say different things than if it's sunny and they buy sunglasses.. adapt to outside circumstances a bit..
When you see a customer, maybe it can help to imagine they're a new cool person and you'd like to make them feel good/better, so that they'll be happy to come again - and even get to know them a little bit so you can alert them to cool and interesting new things (for them!) or just build rapport or tell'em what you can genuinely admire about them.. maybe they have SA or other problems or family/life problems and maybe you can brighten up their day a little bit by giving a small (genuine!) compliment or observation.. maybe they weren't anywhere nice and coming to you is the highlight of their day?
so the schema can maybe be a bit different than existing one: observe person, see and find out what they want, if you and they have time and are not in a rush, maybe see if they are cheerful or in bad mood (angry or down), if cheerful or neutral, just observe on the weather or such, or maybe try to joke a bit if they seem friendly/receptful to that, if they seem down you can maybe give a nice compliment or sympathize, if angry - I don't know what to do with those, if they look like they are in a hurry, maybe just give'em what they want asap and be really respectful?
I usually chatted with those who liked to chat and if someone looked in a hurry just gave'em what they wanted asap. You can even adapt tone of voice and speed of talking to their speed a bit.. (don't exaggerate it tho)
You can get to know them a bit and what exactly they wish, if/when there's time.. so you can consult them better on what to buy.. Of course you can offer any items on sale etc too and inform them of new opportunities.. (this will be easier if you know what they are looking for and have observed them well..)
Many shy people can observe really well, so something like that can 'come naturally'... or it may be a learnt effort..
It also depends what kind of a shop it is: a lot of people and grocery store= different than a high scale boutique or electronics shop. And how many people are in the shop.. You do need some distance with people, it's probably best learnt from other people who work there.. again, different shops have different 'feel' and distance to customers too..
not sure if any of this helps, there are probably manuals for people who work in shops out there!
or probably some tips online too!
I used to not be able to make small talk or talk to people at all, when I was a teen, then I observed other people a bit, how they did it, and when I worked in a shop/with people for a bit, observing other people was helpful too..
When it comes to small talk, it's not so much about what you say, it's more about that you say it.. and the tone, that you keep it polite and respectful, 'just say something'... (that is still polite & well-meaning)
I'm not sure if this is something that can be helpful in a case like you stated (maybe there it's better to pretend you work in a posh store and be 'extra polite' or...?) But when just doing small talk, keep in mind it doesn't have to be inventing a rocket, you can say the same thing to a hundred people every day (okay exaggerating a bit) and they won't know.. (as long as they're not in the store the same time, so maybe just find a few things you could say..)
also, if it's a sports store or someone looks like a sports fan, you could comment on the sports, etc. whatever they might be interested in.. There are books on selling and marketing out there, maybe they could be helpful too?