I think technically they can choose to retract the job offer depending on what your references say. However, depending on where you live, what your prospective employer can ask your references is very limited by law. So this doesn't give them a whole lot of information anyway and in the rare case your references say something bad, they definitely have the right to take that into consideration. But then again, why would you list a reference who wouldn't say good things about you?
Hopefully, since it looks like you're at the final stages of accepting the position, the process of talking to your references (if they even decide to contact them) should go over smoothly. I wouldn't worry too much about it. Good luck!