For the first time in years, there are signs that liberals are ready to give up their infatuation with the Supreme Court. While it has not come easy, it is long past time. And there are promising avenues for reform on the horizon that could wrest control from right-wing ideologues on the bench.

Liberals’ past resistance to reform has seemed connected to their belief in an old story about the Court. Mainstream Democrats love to remember Brown v. Board of Education and celebrate the friendship between the liberal Ruth Bader Ginsburg and the conservative Antonin Scalia. The idea is of a Court beyond politics, upholding Americans’ most fundamental rights.

But this story has never been true, and the images it relies on are highly misleading. Brown didn’t end segregation, which persists in many areas of American life; what real progress we have made towards integration owes more to organizing, legislation, and administrative action than to the Court. The obsession with Ginsburg and Scalia’s friendship betrays a similar ignorance of underlying social realities. They were two elites with elite friends who accepted sketchy trips from rich people, didn’t hire black clerks often, liked to go to the opera together, and were so convinced of their own brilliance they died in office. To canonize either, we have to forget the actual consequences of their time on the bench — for Ginsburg, the way her stubbornness contributed to the gutting of equal protection law and the right to abortion; for Scalia, the way his bluster enshrined bigotry in the pages of the U.S. Reports.

A better figure for grasping the tension between the Court’s promises and its reality might be David Souter. A rock-ribbed Republican from New Hampshire, he became disgusted with the disingenuous mudslinging of his conservative colleagues and veered toward the center. He was a judge’s judge, believing the Court had a narrow role to play, and he never hid from the complexity of the problems he was asked to address. This occasionally led him to vote against preferred conservative outcomes. The lesson Republicans drew was that judiciousness was a liability. They rallied around a promise to appoint “no more Souters.” Each of their nominees since has been carefully vetted for ideological conformity. Their views are so out of step with those of regular people they can only be imposed through fiat. That, of course, is why they are there.

If liberals are finally ready to deal with the rogue judiciary, they have many tools ready to hand, borne of earlier fights between conservative courts and progressive advocates. After the Civil War, when the Court threatened to invalidate Congressional Reconstruction, Radical Republicans used “jurisdiction stripping” to eliminate its power to hear the case. In the Progressive Era, when courts colluded with employers to crush labor strikes, legislators fought to limit the ability of the judiciary to end strikes. And during the New Deal, when a conservative Court struck down social welfare programs, Franklin Roosevelt threatened to add new justices to the Court unless it changed course. The deeper past offers more extreme reform examples. When Thomas Jefferson and his party came to power after the election of 1800, for instance, they repealed an act that had created new Federalist judgeships and canceled the Court’s term.

In recent years, liberals, centrists, and even some conservatives have begun to build consensus around comparatively moderate reforms. In the legal academy, many scholars now agree that lower courts should have greater restrictions on their ability to issue certain kinds of nationwide injunctions and that Supreme Court justices should have term limits. (I still hold a torch for a panel rotation system, in which many judges would be cross-appointed “Supreme Court justices” and randomly selected in panels to hear cases.) Versions of these and other proposals are legally and administratively feasible with extant powers. But we should not lose sight of the ultimate goal: we have to understand that the Court is a political actor and implement reforms to take political power back. Our gerontocracy and its enablers have been in the grip of a powerful, comforting fantasy. But we can no longer afford such illusions. The current Supreme Court is a redoubt of conservative power, committed to a rebarbative social order. It will pull us down, if we don’t stop it first.