first thing I would do is take the switch out and clean the contacts on the connector may not be it but its worth a shot if they are loose beyond there being a problem with wiring I would point to the switch or the connection
Is it only when you bring the window down you have the problem? There is a switch for up, down, lock and unlock. If none of these others have problems, the actual copper switch may be off track or bad. If bad you will need a whole new assembly. They should firmly click in and out of place.
This is what the switch looks like, the copper parts are the bounce back switches.