Sullivan is scheduled to teach a daily course on September afternoons and evenings. That, coupled with preparation in the morning, would make his participation in the Weinstein case "unworkable," and it's too late to arrange a sabbatical or alternative teaching schedule, the letter reads.
The judge accepted Sullivan's withdrawal Monday.
Though the withdrawal request pointed to his teaching schedule, Sullivan's statement referenced the controversy that swirled at Harvard this year over his decision to represent Weinstein.
"My decision to represent Mr. Weinstein sparked considerable discussion and activism around issues of sexual violence, the appropriate role and responsibilities of Harvard and its faculty in addressing those issues, and the tension between protecting the rights of those criminally accused and validating the experience of those who are survivors of sexual violence," Sullivan said.
"My representation of those accused of sexual assault does not speak to my personal views on any of these matters," he added.
Sullivan remains available to the Weinstein trial team for consultation.
"Mr. Weinstein is extremely grateful to Ronald Sullivan for his work with him until now, and for Ron's offer to advise where he can going forward," a spokesman for Weinstein said Tuesday, adding that "Sullivan believed that Mr. Weinstein deserved a vigorous defense, and it is a sad moment for us all right now."
"We, as a country, have now reached the point when a Harvard lawyer and professor cannot serve his duty to, and belief in, the law and defend a person who may be deemed unpopular or unworthy of a legal defense by segments of the public," the spokesman said.
Sullivan had told students that people deemed 'vile' deserve defense
"It is particularly important for this category of unpopular defendant to receive the same process as everyone else -- perhaps even more important," Sullivan wrote.
"To the degree we deny unpopular defendants basic due process rights we cease to be the country we imagine ourselves to be. In fact, most of the due process rights we hold dearest derive from lawyers who represented unpopular defendants."
The decision to represent a high-profile defendant in an assault case was fraught on Harvard's campus, where sexual assault and harassment have been a major issue.
Over the weekend, Harvard announced that Sullivan and his wife, Harvard law lecturer Stephanie Robinson, will no longer be faculty deans of Winthrop House after their term ends June 30.
Harvard College Dean Rakesh Khurana, in a letter explaining his decision not to renew the couple's deanships, wrote generally about a deteriorating climate in the 400-student house but did not specify the underlying issue.
"Over the last few weeks, students and staff have continued to communicate concerns about the climate in Winthrop House to the college. The concerns expressed have been serious and numerous," Khurana wrote.
"This decision in no way lessens my gratitude to them for their contributions to the college."
In an interview with the school's newspaper in February, Khurana said that faculty members are given "academic freedom to make decisions that are right for them," and that "every individual is entitled to a vigorous defense."
Over the weekend, Sullivan and Robinson said they were "surprised and dismayed" by Khurana's decision not to renew them as faculty deans.
"We believed the discussions we were having with high-level university representatives were progressing in a positive manner, but Harvard unilaterally ended those talks," the couple said in a joint statement. "We will now take some time to process Harvard's actions and consider our options.
"We are sorry that Harvard's actions and the controversy surrounding us has contributed to the stress on Winthrop students at this already stressful time."
Weinstein's trial begins later this year
Weinstein is accused of raping a woman in a New York hotel room in 2013 and forcibly performing oral sex on another woman at his Manhattan apartment in 2006.
The former film executive faces five felony charges: two counts of predatory sexual assault, one count of criminal sexual act in the first degree, and one count each of first-degree rape and third-degree rape. He has pleaded not guilty to all charges.
Weinstein has denied all accusations of nonconsensual sex.
The trial is set to begin September 9.
CNN's Eric Levenson and Augusta Anthony contributed to this report.