Rob Lowe on Taylor Swift’s Fame Compared to His ’80s Heyday (Exclusive)

At his peak Hollywood It Boy fame during the ’80s, Rob Lowe navigated a superstar life and daily frenzied fandom.

While shooting St. Elmo’s Firethe Brat Pack era’s magnum opus, he says, “I remember them having to bring me on and off the set in a police car, and that wasn’t the first or only time. People breaking into my grandparents’ house in Ohio thinking I was going to be there.”

The list of “crazy stuff” goes on: a fan broke into his house and nabbed his underwear; cops called after screaming throngs of people waited in the snow all night while he was on a ski vacation; folks dressed as him for Halloween. And on and on.

“It’s the kind of stuff you look back on and go, did that really happen?” Lowe tells PEOPLE in this week’s cover story that celebrates transformative moments in the Hollywood icon’s life. “The stories I have are mental, they’re nuts.”

Now 60 and enjoying an enduring career that’s spanned decades and genres, Lowe has a special appreciation for those glory days. “It was an incremental process to occupying that place in the culture that I did in the ’80s, and it was a lot,” says Lowe, who made his film debut in Francis Ford Coppola’s 1983 coming-of-age classic The Outsiders, followed by St. Elmo’s Fire.

“I’m super grateful that I can say that I had that in my life, because very few people get to be that person. Every decade there’s a new crop and society demands it. It’s fun to watch that unfold having been there.”

He adds: “Today it would be (Justin) Bieber, Taylor Swift, I don’t know if it’s Austin Butler, whoever it is today, there’s always going to be somebody living that life.”

While he remembers the ’80s with a certain fondness, Lowe says he grappled at the time with how to make sense of the white-hot spotlight.

“The unease I felt with it I could never put my finger on until many years later and a lot of self-reflection, which came with getting into recovery,” says Lowe, who is 34 years sober.

“I was intuitive enough in those days to sense the disconnect between me, who I was, the work I was doing, that was out there in the public and making this phenomenon, the hysteria, happen.”

Still, Lowe says his peak celebrity pales in comparison to the stratospheric fame juggled by the likes of Swift. He says, “I watch that, and it’s what I went through on a gazillion steroids.”

For more of Rob Lowe’s PEOPLE interview, please pick up this week’s cover, on newsstands Friday.