Sounds odd, but look in your local record store. When I first moved to London, I bought 3 bootlegs from Tower Records in Piccadilly Circus, and two Rough Trade 7" singles from HMV Oxford Street - Shoplifters... and That Joke Isn't Funny Anymore. This was in 1996.
If you go into HMV Oxford Street today, they have something like 30 copies of a Smiths interview on 7", which they're selling for £3.99.
Can never hurt to be nice to the old dears at your local charity shops either - it's amazing the records that some people give away to such places. Over the years 4 excellent condition copies of This Charming Man (2 each of 1st and 2nd pressings) have cost me less than a fiver in total.
So yeah - get your first few bits, keep buying new releases as they emerge, and suddenly it'll start racking up. The most straightforward thing to do from the off is gradually start getting the old, official releases (the Rough Trade singles and albums). Once you get all these on vinyl - which is pretty straightforward, even 20 years on - then it's time to turn to the rarities.
The key is to not get too excited and pay well over the odds for things. There are so many different items that you could get your hands on that even the deepest pockets won't suffice. Decide if travelling 2 hours on a train to go to a second hand record shop, only to emerge with nothing but a 1 track CD promo in a white paper sleeve is a worthwhile way to spend a day.
One method I'd suggest is to go to record fairs, see what's out there, but limit your horizons to a specific area. For instance, I started out trying to complete my collection of German coloured vinyl, and foreign editions of singles with alternate artwork, and unusual items like the Aussie Education In Reverse (aka Viva Hate) LP. Some people are into test pressings.
Just don't buy anything just for the sake of it - you'll end up with a bitty collection of things you don't actually care about that much if you do. Half the fun is formulating a mental idea of items you dearly want, and then getting out there to find it. It won't happen in a day, but when you eventually find what you're looking for, and then manage to haggle the owner down by a tenner, it's well, well worth it.
.
