No, you would both be able to still play. My wife is left handed and I am right handed. We each use one controller and it works fine. It mainly focuses on the movement of the controller, not which hand it’s in. Just make sure you both stand in front of the sensor! Hope this helps.