Yeah, it is a copyright issue for sure. But zines are such small run and zinesters are (usually) not millionaires, so the chances that someone would come after you is really slight. I would say clip up those magazines and get to pasting.
*** Although, if there is anyone on this site who is a zinester AND a lawyer I would love to ask you a few questions!***
I dig collage, and while I have some public domain material, I wind up using mostly my own images and drawings as they are the only materials I can be 100% sure are safe to use.