England coach Eddie Jones was involved in two altercations with Australia fans who called him a "traitor" after Saturday's win in Sydney.
The Australian, who coached his national team from 2001 to 2005, took the England job in 2015.
Jones passed home fans on the way and from the dressing room following a series-clinching 21-17 win.
"Clowns think they have full go to abuse coaches," the 62-year-old told the Sydney Morning Herald.
Footage posted on social media shows Jones being baited in two incidents at the Sydney Cricket Ground (SCG).
One clip shows a man pointing a finger at Jones and twice saying "you're a traitor". Jones approached the man and "come here and say it" three times while being ushered away by a member of security staff.
In another incident that takes place as he walks through the SCG members area, Jones is called a traitor and his reply contained an expletive.
"That fan's behaviour is totally unacceptable and is not welcome at a Wallabies Test if he carries on like that," said Rugby Australia chairman Hamish McLennan.
"We should be acknowledging Eddie's achievements as a great Aussie export. There's too much tall poppy syndrome in that comment."
The Rugby Football Union (RFU) has been approached for comment.
Last week Jones said he took satisfaction from silencing Australia fans courtesy of England's 25-17 win in the second Test in Brisbane last weekend.
On England's 2018 tour to South Africa Jones was involved in a confrontation with spectators at Ellis Park.
Earlier that year three Scotland supporters were fined for verbally abusing Jones outside Manchester's Oxford Road station following England's 25-15 defeat at Murrayfield.