A physical characteristic of an organism's sex that is not directly related to its REPRODUCTIVE SYSTEM.