EFF: EFF to HUD: Algorithms Are No Excuse for Discrimination

EFF to HUD: Algorithms Are No Excuse for Discrimination

The U.S. Department of Housing and Urban Development (HUD) is considering adopting new rules that would effectively insulate landlords, banks, and insurance companies that use algorithmic models from lawsuits that claim their practices have an unjustified discriminatory effect. HUD’s proposal is flawed, and suggests that the agency doesn’t understand how machine learning and other algorithmic tools work in practice. Algorithmic tools are increasingly relied upon to make assessments of tenants’ creditworthiness and risk, and HUD’s proposed rules will make it all but impossible to enforce the Fair Housing Act into the future.

What Is a Disparate Impact Claim?

The Fair Housing Act prohibits discrimination on the basis of seven protected classes: race, color, national origin, religion, sex, disability, or familial status. The Act is one of several civil rights laws passed in the 1960s to counteract decades of government and private policies that promoted segregation—including Jim Crow laws, redlining, and racial covenants. Under current law, plaintiffs can bring claims under the Act not only when there is direct evidence of intentional discrimination, but also when they can show that a facially-neutral practice or policy actually or predictably has a disproportionate discriminatory effect, or “disparate impact.” Disparate impact lawsuits have been a critical tool for fighting housing discrimination and ensuring equal housing opportunity for decades. As the Supreme Court has stated, recognizing disparate impact liability “permits plaintiffs to counteract unconscious prejudices and disguised animus” and helps prevent discrimination “that might otherwise result from covert and illicit stereotyping.”

What Would HUD’s Proposed Rules Do?

The defendant’s use of an algorithm wouldn’t merely be a factor the court would consider; it would kill the lawsuit entirely.

HUD’s proposed rules do a few things. They would make it much harder for plaintiffs to prove a disparate impact claim. They would also create three complete defenses related to the use of algorithms that a housing provider, mortgage lender, or insurance company could rely on to defeat disparate impact lawsuits. That means that even after a plaintiff has successfully alleged a disparate impact claim, a defendant could still get off the hook for any legal liability by applying one of these defenses. The defendant’s use of an algorithm wouldn’t merely be a factor the court would consider; it would kill the lawsuit entirely.

These affirmative defenses, if adopted, would effectively insulate those using algorithmic models from disparate impact lawsuits—even if the algorithmic model produced blatantly discriminatory outcomes.

Let’s take a look at each of the three affirmative defenses, and their flaws.

The first defense a defendant could raise under the new HUD rules is that the inputs used in the algorithmic model are not themselves “substitutes or close proxies” for protected classes, and that the model is predictive of risk or some other valid objective. The problem? The whole point of sophisticated machine-learning algorithms is that they can learn how combinations of different inputs might predict something that any individual variable might not predict on its own. And these combinations of different variables could be close proxies for protected classes, even if the original input variables are not.

For example, say you were training an AI to distinguish between penguins and other birds. You could tell it things like whether a particular bird was flightless, where it lived, what it ate, etc. Being flightless isn’t a close proxy for being a penguin, because lots of other birds are flightless (ostriches, kiwis, etc.). And living in Antarctica isn’t a close proxy for being a penguin, because lots of other birds live in Antarctica. But the combination of being flightless and living in Antarctica is a close proxy for being a penguin because penguins are the only flightless birds that live in Antarctica.

In other words, while the individual inputs weren’t close proxies for being a penguin, their combination was. The same thing can happen with any characteristics, including protected classes that you wouldn’t want a model to take into account.

Apart from combinations of inputs, other factors, such as how an AI has been trained, can also lead to a model having a discriminatory effect. For example, if a face recognition technology is trained by using many pictures of men, when deployed the technology may produce more accurate results for men than women. Thus, whether a model is discriminatory as a whole depends on far more than just the express inputs.

HUD says its proxy defense allows a defendant to avoid liability when the model is “not the actual cause of the disparate impact alleged.” But showing that the express inputs used in the model are not close proxies for protected characteristics does not mean that the model is incapable of discriminatory outcomes. HUD’s inclusion of this defense shows that the agency doesn’t actually understand how machine learning works.

The second defense a defendant could raise under HUD’s proposed rules has a similar flaw. This defense shields a housing provider, bank, or insurance company if a neutral third-party analyzed the model in question and determined—just as in the first defense—that the model’s inputs are not close proxies for protected characteristics and is predictive of credit risk or another valid objective. This has the very same problem as the first defense: proving that the express inputs used in an algorithm are not close proxies for one of the protected characteristics—even when analyzed by  a “qualified expert”—does not mean that the model itself is incapable of having a discriminatory impact.

The third defense a defendant could raise under the proposed rules is that a third party created the algorithm. This situation will apply in many cases, as most defendants—i.e.,the landlord, bank, or insurance company—will use a model created by someone else. This defense would protect them even if an algorithm they used had a demonstrably discriminatory impact—and even if they knew it was having such an impact.

There are several problems with this affirmative defense. For one, it gets rid of any incentive for landlords, banks, and insurance companies to make sure that the algorithms they choose to use do not have discriminatory impacts—or to put pressure on those who make the models to work actively to try to avoid discriminatory outcomes. Research has shown that some of the models being used in this space discriminate on the basis of protected classes, like race. One recent study of algorithmic discrimination in mortgage rates, for example, found that Black and Latinx borrowers paid around 5.3 basis points more in interest with online mortgage applications when purchasing homes than similarly situated non-minority borrowers. Given this pervasive discrimination, we need to be creating more incentives to address and root out systemic discrimination embedded in mortgage and risk assessment algorithms, not getting rid of them. 

In addition, it is unclear whether aggrieved parties can get relief under the Fair Housing Act by suing the creator of the algorithm instead, as HUD suggests in its proposal. In disparate impact cases, plaintiffs are required under law to point to a specific policy and show how that policy (usually with statistical evidence) results in a discriminatory effect. In a case decided earlier this year, a federal judge in Connecticut held that a third-party screening company could be held liable for a criminal history screening tool that was relied upon by a landlord and led to discriminatory outcomes. However, disparate impact case law around third-party algorithm creators is sparse. If HUD’s proposed rules are implemented, courts first must decide whether third-party algorithm creators can be held liable under the Fair Housing Act for disparate impact discrimination before they can even reach the merits of a case.  

Even if a plaintiff would be able to bring a lawsuit against the creator of an algorithmic model, the model maker would likely attempt to rely on trade secrets law to resist disclosing any information about how its algorithm was designed or functioned. The likely result would be that plaintiffs and their legal teams would only be allowed to inspect and criticize these systems subject to a nondisclosure order, meaning that it would be difficult to share information about their flaws and marshal public pressure to change the ways the algorithms work. Many of these algorithms are black boxes, and their creators want to keep it that way. That’s part of why it’s so important for plaintiffs to be able to sue the landlord, bank, or insurance company implementing the model: to ensure that these entities have an incentive to stop using algorithmic models with discriminatory effects, even if the model maker may try to hide behind trade secrets law to avoid disclosing how the algorithm in question operates. If HUD’s third-party defense is adopted, the public will effectively be walled off from information about how and why algorithmic models are resulting in discriminatory outcomes—both from the entity that implemented the model and from the creator of the model. Algorithms that affect our rights should be well-known, well-understood, and subject to robust scrutiny, not secretive and proprietary.

HUD claims that its proposed affirmative defenses are not meant to create a “special exemption for parties using algorithmic models” and thereby insulate them from disparate impact lawsuits. But that’s exactly what the proposal will do. HUD says it just wants to make it easier for companies to make “practical business choices and profit-related decisions.” But these three complete defenses will make it all but impossible to enforce the Fair Housing Act against any party that uses algorithmic models going forward. Today, a defendant’s use of an algorithmic model in a disparate impact case would be considered on a case-by-case basis, with careful attention paid to the particular facts at issue. That’s exactly how it should work. HUD’s proposed affirmative defenses are dangerous, inconsistent with how machine learning actually works, and will upend enforcement of the Fair Housing Act going forward.

What is EFF Doing, and What Can You Do?

HUD is currently accepting comments on its proposed rules, due October 18, 2019. EFF will be submitting comments opposing HUD’s proposal and urging the agency to drop these misguided and dangerous affirmative defenses. We hope other groups make their voices heard, too.