Kellie Petersen, who was on the cruise with her husband and their three children, said trouble on board had been "brewing for days".
Police say they have not been contacted by Carnival.
On Saturday, Carnival said, the Legend docked back in Melbourne, 10 days after leaving port.
The company said it was fully cooperating with NSW Police and conducting its own investigation into what happened.
Fourteen other members of the family also left the ship in what Carnival has described as an unprecedented action. "We can't wait to get off", Peterson told radio station 3AW.
The cruise company's executives later confirmed that one family had been involved in ongoing incidents.
As the family was offloaded on to police boats on Friday morning, passengers rained down a barrage of insults from above on deck, yelling "Get off" and "You're not heroes, you're deros".
Ms Bolitho was critical of how the ship's management handled the family, saying the captain told concerned passengers, "What do you want me to do about it - throw them overboard?"
"Three of the gang members came down and started attacking him for no reason, one was from behind strangling him, the guy on the left was punching him in the head and the right was just trying to rip his shirt off". Lisa Bolitho said she and her son simply locked themselves inside their cabin, according to the Australian Associated Press.
"They called up the mechanical people from down the bottom, the big boys, to intimidate and basically do what they want".
Ryan Mason: 'I could go into coaching'
Mason clashed heads with Cahill in a match at Stamford Bridge early previous year and fractured his skull in the incident. If we were lucky to play on Friday, we were very unlucky to have this schedule.
The days of fighting on board revealed in videos and passenger accounts, include guests saying they were "scared for their safety" in the eight days leading up to the family's eviction.
Jarrah Bolitho said the family had been "roaming the decks looking for people to fight" and that security staff did their best under hard circumstances.
Before the video ends, a security guard balls his fist and draws his arm back as if to punch a woman in the face, but he lowers his fist at the last moment.
"My husband said to take it away, because there's kids here, and five of them surrounded my husband".
Several of the family members were kicked off the ship and taken by local authorities.
"Our security team responded in several instances to extremely unruly behavior by these guests while the ship was at sea", the statement reads.
In a statement to USA TODAY, the cruise line said it is investigating. "We are conducting a full investigation and will take appropriate corrective action as necessary".
Arriving in Melbourne, passengers aboard the ship described hiding in their cabins to escape threats of violence.
Carnival general manager Jennifer Vandekreek said: "We sincerely apologise to our guests who were impacted by the disruptive behaviour of the group removed from the ship".