TORONTO -- Before the Raptors faced the San Antonio Spurs on Sunday at Scotiabank Arena, Raptors coach Nick Nurse was concerned about Pascal Siakam needing to shake off several weeks' worth of rust in his first game back from a groin injury.

"I think it'll be a challenge, to be honest with you," Nurse said.

Unfortunately for Nurse -- and the Raptors -- he was proved right.

Siakam got off to a hot start, scoring 12 points in the first quarter and 14 in the first half, before fading in the second half. He missed all five of his shots after the break -- including a possible game-tying layup with 19.2 seconds remaining and a potential winning 3-pointer at the buzzer -- as the Raptors blew an 18-point second-half lead and lost 105-104 to the Spurs.

"I feel like I kind of lost my rhythm a little bit," Siakam said after going 6-for-17 from the field and finishing with 15 points, 5 rebounds and 4 assists in 30 minutes. "Eventually, I'll be better with that, but first time back out there, I think I did a decent job. I felt I could have been way more aggressive, so I'll be waiting to be more aggressive and just try to find my rhythm in a better way [and] learn from it."

Siakam and Norman Powell, who also made his return from a lengthy absence due to a shoulder injury, each were on strict minutes limits on Sunday. Although both said after the game that they felt good, there was nothing good about how the Raptors played in the second half.

For the first 30 minutes or so, it looked as if the Raptors wouldn't have a problem with the Spurs, who have beaten the Milwaukee Bucks, Boston Celtics and Toronto in the span of a week -- with the latter two victories coming on the road -- and are percentage points behind the Memphis Grizzlies for the eighth and final playoff spot in the Western Conference.

But after Toronto took a 74-56 lead with 4:16 to go in the third quarter, the Spurs outscored the Raptors 49-30 the rest of the way, including a 27-4 run in the fourth, to stun Toronto three weeks after the Raptors came back from 30 points down to beat the Dallas Mavericks.

That this loss comes just five days after the Raptors blew a 14-point lead at home against the Portland Trail Blazers only made it sting more.

"We stopped playing," said Kyle Lowry, who had 16 points and 15 assists. "We stopped being aggressive. We stopped being assertive. They got more aggressive. They got faster, and we missed shots too. A lot happened, and it was all spiraling downhill.

"[We were] just not playing with the extreme confidence that we should have been. Not getting defensive stops and kind of playing lackadaisical."

One person who wasn't lackadaisical in that second half was former Raptor DeMar DeRozan, playing in his second game back in Toronto since he was traded in exchange for Kawhi Leonard 18 months ago. According to ESPN Stats & Information research, the last time the Raptors lost a game after leading by as many as 18 points was last season against the Detroit Pistons in Toronto, which marked ex-Raptors coach Dwane Casey's first game back against his former team.

DeRozan scored 22 of his game-high 25 points in the second half, with 18 of them coming after the Spurs fell behind by 18 in the third, including a slam dunk right over Raptors forward Chris Boucher.

"I played against [Boucher] this summer in the gym, and I didn't know who he was, and he was blocking every shot," DeRozan said. "So I told myself, once I seen him down there, I was going to go to the basket, be aggressive and at least try to get fouled.

"I did and dunked it."

For his part, Boucher said plays such as that are part of the business of being a shot-blocker.

"I was just trying to block the shot, really," Boucher said, adding that those pickup games DeRozan referenced were at UCLA. "I'm a shot-blocker, so you get some, and you miss some. That's one I was trying to get. I was trying to block it, and I didn't.

"I've blocked a thousand shots. It's going to happen. I think everybody knows that's what I do. He got me this time. I might get him the next time."

The same could be said of the Raptors as a whole after yet another tough loss -- Toronto's sixth defeat in its past 10 games after a 21-8 start. Now that the Raptors are beginning to get healthy -- including the possibility of Marc Gasol coming back from a hamstring strain on Wednesday at the Oklahoma City Thunder -- they hope that the problems they saw this week against Portland and San Antonio will be remedied soon.

"I think that normally our fourth-quarter defense is great," Nurse said, "but it wasn't tonight, and it wasn't against Portland the other night, and we kind of let the star guys do the damage, and that's not really what was happening throughout the game, right? So we've just got to be able to sustain that over some extended time, and that's probably a little harder to do with a lot of people in and out of the lineup.

"The consistency thing gets jolted every time two guys come in and out, I think, and that will even itself out here."