I don't think it's as cut and dry as just taking a week off from golfing. Keep in mind, acting was his life, and he made what was probably a difficult decision to retire from that. Think of the psychological impact and transformation that kind of decision would have on your identity, after being an actor for decades and decades. He decided to say goodbye to that world, and enjoy the rest of his days in leisure.
My guess is, had the movie gotten going even a year or two earlier, he might have joined the cast. But after a few years of retirement, that mindset, the decision he made, would be pretty well set in stone.
Which is why I understand why he said to Lucas, "Look, I've done it, I've done it." He couldn't go home again, couldn't reverse that difficult decision, even for a week, as tempted as he might have been.