I don't know about physical laborers being fatter, but our bodies are horribly/wonderfully efficient when it comes to work. The amount eaten will be the larger factor.
See:
http://en.wikipedia.org/wiki/Basal_metabolic_rate#Biochemist.... About 70% of a human's total energy expenditure is due to the basal life processes within the organs of the body (see table). About 20% of one's energy expenditure comes from physical activity and another 10% from thermogenesis, or digestion of food (postprandial thermogenesis).
The reference to the wikipedia article is misleading, and practically false.
Practicing physical activity indirectly increases a series the expenditure for the basal life processes, so it increases the calories consumption more than the raw amount of energy spent doing it.
See:
http://en.wikipedia.org/wiki/Basal_metabolic_rate#Biochemist.... About 70% of a human's total energy expenditure is due to the basal life processes within the organs of the body (see table). About 20% of one's energy expenditure comes from physical activity and another 10% from thermogenesis, or digestion of food (postprandial thermogenesis).