It might be true that leather feels colder than fabric at first contact. But this impression disappears within a few seconds after the air between the fibres and the individual cells has warmed up.
Leather adapts quickly and individually to the ambient room temperature and the body temperature of its owner, hence becoming just as pleasant as fabric within a few seconds.