Bobby Paul Edwards, 52, has been accused of abusing 39-year-old John Christopher Smith at his fast food restaurant J&J Cafeteria in Conway, South Carolina, over a period of five years.
It carries a 20 year sentence and maximum fine of $250,000.
Edwards was released on bail pending further court action.
Like Smith, waitresses at the J&J Cafeteria were reluctant to come forward because they feared Bobby Paul Edwards, according to Geneane Caines, an advocate for Smith who said her daughter-in-law worked at the restaurant. Smith was rescued during a field visit by the workers who received the information.
Edwards allegedly forced Smith to work 18-hour shifts six days a week.
According to The Courier, lawyers claimed that Bobby would also beat Smith with frying pans and butcher knives all "while being called the N-word repeatedly".
Though the indictment remains sealed, numerous details surrounding Smith's alleged enslavement were revealed in a 2015 lawsuit against Edwards; the man's brother and restaurant owner, Ernest J. Edwards; and the establishment, The Washington Post reported.
Edwards had already been arrested on a state charge of second-degree assault and battery, which remains pending, but the Federal Bureau of Investigation and civil rights prosecutors from the Department of Justice also started probing the case. Edwards will also have to pay restitution to Smith if convicted.
In a lawsuit which Smith filed against the brothers in 2015 but dropped in favor of having the case pursued criminally, his attorneys told how he was often taken to the industrial freezer to be beaten.
Shiv Sena leader Pappu Mane's vehicle mows down 2 girls in Maharashtra
A group of angry locals set it on fire and resorted to a "rasta roko" (road block) to demand the driver's arrest, he said. Speaking to DNA, additional superintendent of police Sandeep Pakhale said, "We have summoned Mane for questioning".
Edwards' attorney didn't respond to outlets' requests for comment.
Smith described Edwards like a slave driver.
"Plaintiff was heard crying like a child and yelling, 'No, Bobby, please!'" according to the suit, which accused the cafeteria's owner of knowing about the alleged abuse but doing nothing to stop it.
Edwards allegedly told Smith that he had a bank account with more than $30,000 of his earnings, but Smith said he was never paid any of that money or given access to the account.
Smith has become a "very happy, very outgoing" person since leaving the restaurant in late 2014, according to Caines.
"Costumers that were going in there would hear stuff and they didn't know what was going on, and they would ask the waitresses, and the waitresses were so scared of Bobby they wouldn't tell them then what it was", Caines told WMBF past year. Workers allegedly discovered scars on Smith's back and placed him into Adult Protective Services' (APS) custody.
In late 2014 Edwards was captured on a state charge of second-degree ambush and battering, which is as yet pending, the Post and Courier revealed.
In an interview with WMBF in 2016, Smith said Edwards began abusing him in 2010. "And he believes that ultimately, justice will be served".