James Caan and Hugh Hefner in 2003.Photo: Donald Miralle/Getty

James Caanonce had an extended stay at thePlayboy Mansion.
Back in 2003, the actor — whodied Wednesday at the age of 82— toldEsquire, “To get over my divorce, I got a prescription to live at the Playboy Mansion for a while.”
James was previously married four times: first to Dee Jay Mattis, from 1961 to 1966, then to Sheila Ryan in 1976. He later tied the knot with Ingrid Hajek in 1990. The two divorced in 1995, and he wed Linda Stokes later that year.
The actorfiled for divorce from Stokesin 2005, butthe two later reconciled. James and Stokes were in the middle of a divorce as of December 2016, according toTMZ.
“I have four wives and five kids. I apparently don’t know the difference [between sex and love],” James joked in his 2003Esquireinterview.
Never miss a story — sign up forPEOPLE’s free daily newsletterto stay up-to-date on the best of what PEOPLE has to offer, from juicy celebrity news to compelling human-interest stories.
James Caan and family at the premiere ofMiseryin 1990.Ron Galella Collection via Getty

James' sonScott Caanspent time at the Mansion with his father when James briefly lived there during Scott’s childhood.
In a2010 interview with PEOPLE, James recalled with a laugh, “I’d have Scott bring me girls.”
For more on James Caan, listen below to our daily podcast PEOPLE Every Day.
“No one could say no to a cute little boy,” he added of Scott, now 45.
Scott’s mother is Ryan, who died in 2012 at age 60.
Hugh Hefner and James Caan in 1983.Bei/Shutterstock

Atweet from James' official accountThursday read, “It is with great sadness that we inform you of the passing of Jimmy on the evening of July 6.”
“The family appreciates the outpouring of love and heartfelt condolences and asks that you continue to respect their privacy during this difficult time,” the tweet added.
A cause of death was not made immediately available.
source: people.com