First things first: !!!! What kind of AU?

Second things second: black_hound is, as usual, the expert on the matter, but I think I've seen references to shirts made from cotton/muslin, linen, and wool, with linen usually being the material for young gentlemen like Mr. Hammond. I don't want to think about how much the wool shirts itched.

(And a quick google suggests those materials, as well as something called osnaburg.)