He once told Vogue that he wanted to be the Irish Ralph Lauren