Baker, 73, said he never felt bad and had only mild symptoms that he attributed to seasonal allergies before testing positive.
"I'm very fortunate that I had both my shots and the two boosters," he said. "So, I'm really hoping that everybody goes and gets the shots and boosters, because if you do get [COVID-19] hopefully it's mild like mine was."
Baker tested positive on Friday when the team was on a road trip in Cleveland. He isolated in his hotel room all weekend and did not return to Houston with the team on Sunday night.
He returned to Houston on a private jet Monday but wasn't cleared to return to the dugout until Thursday.
At first, Baker welcomed the rest amid a hectic season in which he didn't get any time off during the All-Star break because he was the manager of the AL squad. But after a couple of days, boredom set in.
"I listened to music all day, watched TV, made phone calls and caught up on things," he said. "I don't get bored, but after five or six days you want to get back out here."
He kept up with the team by watching the games on television and staying in constant contact with bench coach Joe Espada, who managed the team in his absence. He didn't watch one game -- Sunday's 1-0 loss to Cleveland -- because it was broadcast on Peacock, NBC's streaming service, and he couldn't figure out how to watch it in his hotel room.
He wasn't that mad about it, though, because he used the extra time to catch a Bob Marley documentary on Netflix while keeping up with the game on his phone.
"That was a pretty good afternoon," he said, adding that he saw the late reggae great in concert twice, including once in Jamaica.
Baker said he was never that concerned about testing positive despite being in a high-risk age bracket.
"Not really because I didn't feel anything," he said. "And I kept waiting to feel something."
The Astros, who at 71-41 began Thursday tied with the Yankees for the best record in the American League, went 2-3 while Baker was out.