Camera gear is meant to be abused HOWEVER, have you ever actually taken one out in the winter in actual cold Thomas? The problem is, if you want to come in from the cold temporarily and then go back out, like when you ski, that condensation and moisture will mean it will be very difficult to continue taking photographs. The interior of the lens will fog up destroying your chances to take photos for a while, so yes, it is a very real problem and if it is cold that fog and moisture will freeze. You can get a new one, but in a lot of cases it can end your shoot.
Good solutions mentioned. I just leave mine in my pack for a while and let it gradually warm up and then put it straight in to a sealed plastic bag for a while. When it's no longer cold to the touch you are good to go. When tenting in winter it is very important to place it straight in to a sealed bag with 0 delay.