I haven't had that experience with Morrissey or The Smiths. I devoured every album and dwelt upon them for days, months, years...
(Well, not so much the last few Morrissey albums because I haven't had the time to do that.)
I did have an experience like yours with Joy Division's "Unknown Pleasures". At first I didn't like it at all, couldn't get used to Curtis's voice. After I bought "Closer", though, which I loved almost instantly, I went back and I couldn't believe what I had missed. It's definitely a "grower".
Whenever I play "Kill Uncle", I am always mildly surprised because it's better than I remember it being. I more or less gave up on it after the tour because the rockier stage versions were so superior, in my opinion, that I couldn't go back and play the songs which you aptly describe as delicate. I find Morrissey's songs to be hard and tough-- beautiful but bruising, with cutting humor and crackling with a lot of fight. For that reason the precious (read: wimpy) "Kill Uncle" was and remains an unusual record in my view. I love side one but side two loses steam fast-- although the sublime "There's A Place In Hell" redeems the entire thing.