Besides being some of the biggest manufacturers of swimwear today, Catalina, Cole of California, Jantzen and Speedo have another striking thing in common: They all got their start as wool knitting mills.