Mine is camelhair, but I do the same. Wore it a ton last year. Just pulled it out this year. Still looking great.
I’ve done two jumpers in alafoss lopi - one 25 years ago - going strong. Also a poncho in Lett Lopi. They are not tightly spun yarns but the long Icelandic sheep hair holds the fluffier parts together. If a strand seems a bit fragile, I just felted it back together. Don’t pull the tension too tight but once knit, they kind of felt into a cohesive whole (without the adverse attributes of felting by accident). I agree they’re not good next to the skin (itchy) but are fabulously warm, lightweight and sort of water resistant too. You would want to be able to take them off indoors as they’d be too warm so useful to have something on underneath!! I’ve not tried the lighter weight Icelandic yarns. I suppose you could line the band of a hat or scarf with microfleece?