NEW YORK -- Gerrit Cole shook his head, disgusted. Aaron Hicks put hands on hips and bowed his head. By the end of the afternoon, pretty much all of the New York Yankees were appalled, along with their loudly booing fans.

Yoshi Tsutsugo hit a tiebreaking double off Cole in the seventh inning that lifted the Tampa Bay Rays to a 4-2 win on Sunday and a three-game sweep that extended the Yankees' losing streak to five games. New York fell to an American League-worst 5-10, its poorest start since 1997.

"I'm frustrated, personally frustrated," said reigning AL batting champion DJ LeMahieu, who concluded the Yankees were "tight and pressing."

"No one's going to feel sorry for us," he said. "No one's going to throw softer or throw easier for us. We've got to find it within ourselves to continue to get better and play the way we're capable of."

Opener Andrew Kittredge, Ryan Yarbrough (1-2), Diego Castillo and Jeffrey Springs (first big league save) combined on the Rays' second three-hitter of a series in which New York managed 11 hits in all. Tampa Bay outscored the Yankees 17-7, and New York pushed across just three runs other than on homers.

"We got great players in that room," Yankees manager Aaron Boone insisted. "They do know that at their core, obviously, we're getting punched in the mouth right now."

Cole (2-1) was hurt by slipshod defense that made three mistakes in the Rays' two-run third inning alone, two by Hicks in center and one by Clint Frazier in left. One of the three runs off Cole was unearned, giving the Yankees a major league-high 10.

Tampa Bay's Joey Wendle added a ninth-inning homer off Darren O'Day, prompting boos from the crowd of 10,606, who saved their loudest jeers for the final out.

New York has lost five straight for the first time since Sept. 4-8, getting outscored 30-14.

"You have to strap your boots on and wade through the mud a little bit," Cole said. "If we stay in that mindset and we leave the field every day exhausted, good things are on the horizon. But I don't have the magic wand; I don't know when it's going to turn."

Aaron Judge struck out three times, and he is four for his past 28. Hicks has one hit in his past 15 at-bats, Gleyber Torres three in his past 24 and Giancarlo Stanton, despite a second-inning home run, is three for his past 26. Frazier is one for his past 24 and does not have an RBI in 40 plate appearances this season.

New York's batting average is down to .210, one point above the AL-worst Cleveland Indians.

Boone said his hitters have to guard against "being obsessed with chasing a result, which is hard to do when you get off to a slow start individually. You want to get a hit so bad that that can sometimes work against you."

Yarbrough allowed one run and two hits in five innings, and the Rays beat the Yankees for the 16th time in 21 meetings, including in last year's AL Division Series. The Rays swept a series for the second time in their past three visits to Yankee Stadium after sweeping just two of their previous 51 series of three games or more in the Bronx.

"It's just not an easy thing to do. It doesn't happen very often," Rays manager Kevin Cash said. "So you've got to enjoy it when it does."

Cole retired 13 in a row before Wendle singled with one out in the seventh. Tsutsugo, his average down to .146, followed by driving a changeup to right-center for an RBI double and a 3-2 lead.

Tampa Bay had gone ahead 2-1 in the third after Mike Zunino's leadoff single. Hicks got a late break on Kevin Kiermaier's bloop to short center, then bobbled the ball and lost a chance for a forceout at second. Yandy Diaz followed with an RBI single, as Hicks allowed the ball to kick off his glove for an error that let Kiermaier take third.

Frazier heaved the ball past second, as Diaz advanced on Manuel Margot's go-ahead sacrifice fly.

LeMahieu hit a tying single in the fifth, when Yarbrough struck out Judge with two on and two out.

Tampa Bay, off to a slow start in the defense of its AL pennant, got back to .500 at 8-8.

"We just wanted to kind of get back on track," Yarbrough said.