Researcher Bias undercuts New York’s attempt to assess Kendra’s Law

This post is about New York State’s attempt to assess Kendra’s Law, also known as its court-ordered Assisted Outpatient Treatment (AOT) program. This program mandates treatment of individuals with severe persistent mental illness who meet criteria and who have refused treatment and refused voluntary services. In its evaluation of Kendra’s Law (you too can read all 389 pages here), the PathLab at the Pitt (University of Pittsburgh) collected a tiny sample of lived experience qualitative data and used that data as the basis for sweeping policy criticisms of Kendra’s Law which they allege leads to far more harm than good. In important areas such as housing stability, the quantitative data fails to support the criticisms that PathLab levels in its report. But some of their observations hit home.

I am a bit late to the party when it comes to reviewing the recent report on New York State’s Kendra’s Law, better known as AOT. In the past few days, I read the lengthy report along with defenses of the findings provided by the PathLab (you can read that here) and analysis of the findings from the Treatment Advocacy Center (you can find that here) and public comments made by a NAMI New York State representative (see the final paragraph here). I do not have an advanced degree in statistics and have relied on a variety of sources to help me evaluate the findings and recommendations in this report. Mostly I have relied on my college level training in research and statistics, my common sense, and my years of experience working with individuals and families of individuals with serious mental illness.

Share

First, some background for anyone who doesn’t know much about this program intended to intervene in cases of untreated very severe persistent mental illness. The Assisted Outpatient Treatment program in New York State differs significantly from the program in California (you can read my post about California’s program here). In New York State, there is no guardianship that permits involuntary psychiatric treatment under the direction of the guardian. Here in California we have LPS conservatorships which do allow for this. (You can read a recent post of mine about LPS conservatorships here.) Since they don’t have any way to mandate treatment for someone with untreated severe persistent mental illness who is unable to survive safely in the community and who may be a danger to themselves or the community, New York created Kendra’s Law or Assisted Outpatient Treatment. AOT in New York is a civil court process that results in mandated treatment for certain individuals. Its criteria include the requirement that the person be “unlikely to survive safely in the community” and “unlikely to engage in voluntary care.” Furthermore, under New York State’s AOT criteria (Kendra’s Law), an individual must have a documented history of treatment non-compliance that has directly resulted in acts, threats, attempts of serious violent behavior, or resulted in incarceration within the preceding 48 months.

Individuals who are ordered into treatment through Assisted Outpatient Treatment in New York must report to their provider weekly, comply with treatment requirements including taking medication (long acting injectables are often prescribed), and work towards developing the life skills needed to live independently. If they fail to engage and/or take their medications or begin to decompensate, a doctor can order them picked up on a 9.60 order and taken to a hospital for a psychiatric evaluation. AOT orders in New York are generally for about one year and can be renewed indefinitely. An alternative to AOT is the Enhanced Voluntary Agreement or EVA. This allows someone who is eligible for AOT to agree to accept high intensity services voluntarily rather than be formally ordered through AOT. The extent to which EVAs are utilized by New York counties varies considerably. I will talk about the importance of this fact a little later.

It is clear (starting with the Executive Summary and continuing through the entire report) that PathLab had philosophical objections to Assisted Outpatient Treatment going in and that they had happily found a way to claim that their data supported their opposition to AOT. Their position? They contend that New York State is forcing people into a coercive program rather than offering intensive levels of service on a voluntary basis. Further, they contend that court-ordered treatment is always coercive and harmful to the people who are receiving mandated services, which seems to them to be the most important reason to reject the need for AOT. In other words, if the system would just do its job and offer everyone intensive treatment, there would be no need for AOT which only ever harms the people who end up trapped in the program. They specifically assert that the coercive elements of AOT are not justified because a clinically and socio-economically similar voluntary treatment cohort had outcomes that were just as good as the AOT cohort they were being compared to.

PathLab asserts in their findings that the voluntary treatment cohort performed as well as the AOT cohort and that there is a boatload of harms associated with AOT which make voluntary treatment better than AOT. In fact, the report found that AOT outperformed the voluntary treatment cohort (all assigned to an Assertive Community Treatment program or ACT) in four of ten areas (housing, risk of harm to others, hospitalization, and number of inpatient nights in those with at least one hospitalization), performed the same as the voluntary cohort in another four areas (life skills, employment, number of hospitalizations greater than one, and average number of inpatient nights per hospitalization) and performed less well in two areas (arrests and number of arrests among those with at least one arrest).

Does that mean that the voluntary treatment cohort did as well as the AOT cohort? For me, the first question that must be answered is are these two cohorts really comparable. And the answer is no. The voluntary treatment cohort was able to agree to treatment in the face of severe persistent symptoms that had resulted in massive destabilization of their lives. The AOT cohort, by definition since it is a criterion of AOT, was not. This despite the fact that the AOT cohort had all had an episode within the past four years that included a danger to self, danger to others, or incarceration (or all of the above) and had been determined to be unlikely to survive safely in the community (also known as gravely disabled). PathLab contended that the AOT cohort would have agreed to voluntary services if they had been offered (they assert that voluntary services were not usually offered prior to referring a person to AOT) and therefore the two groups were the same.

I think it’s really unfortunate that the PathLab took this important opportunity to evaluate Kendra’s Law outcomes independently and turned it into a soap box for their biased position opposing all mandated treatment. That’s because there are some troubling issues that are buried in the report that a fair-minded examiner would have identified and that might have supported some of their criticisms better than their blatant biased position that AOT is all bad.

There are a number of reasons to criticize the conclusions that the researchers drew in the report. For example, they make the case that the improvement in housing stability of the AOT cohort is essentially a statistical artifact and that actually the voluntary treatment cohort’s absolute greater improvements in housing (they had worse housing scores to begin with) show that voluntary treatment was better at improving housing stability. Yes, there were improvements in housing stability in both groups. But which cohort attained and/or sustained greater absolute stability over time as supported by the raw quantitative data? That data shows that AOT participants’ housing was much more stable over time compared to the voluntary treatment cohort. The problem? The voluntary treatment cohort’s housing stability was still a lot worse than the AOT cohorts had been to begin with at the assessment point. So the AOT group sustained consistent housing while the voluntary cohort improved their housing stability but did not achieve the level of stability that the AOT cohort had to begin with. Overall, the AOT cohort attained and sustained greater stability across more domains than the voluntary cohort did. But PathLab did their best to bury that finding under pages and pages of qualitative data from forty-six current or former AOT participants.

When it comes to the lived experience qualitative data which the researchers use to support their allegation that the harm to participants far outweighs any good that might have come of their AOT participation, even I, with my BA in Psychology and one published research paper to my credit, was astounded at what seemed to me to be brazen manipulation. They asserted that their interviews of forty-six participants should be used to make sweeping generalizations about the experiences of more than 10,000 AOT participants. And those forty-six participants were a select group who could meet the study requirements that they be able to consent to be research subjects. This alone made them unusual representatives of the entire population of people on AOT who, as a group, are likely to be experiencing significant psychosis and severe anosognosia (you can read more about this lack of insight here) making them ineligible to participate. And, even in the interview group, more than ten percent asserted they were glad to have been forced into treatment because they would never have agreed to it and they had achieved stability and a greatly improved quality of life thanks to their treatment. It seems to me to be expected that someone whose functioning has improved to the point that they could participate in a research study like this one might resent the continuing supervision that AOT requires even if they are doing well for a period of time. It makes sense to me that they might be eager to complain about it to researchers who were looking for data to support their position and might be asking leading questions. I can’t say for sure that happened but some of the quotes provided in the report suggest as much to me.

I want to be clear that I am not trying to say that AOT is not coercive. It is. The question isn’t is it coercive but is it better than the alternative. The outcomes I have seen through the eyes of families desperate to help their loved ones and through reports from law enforcement, public health, and public behavioral health agencies, is often a miserable life punctuated by repeated rounds of incarceration, hospitalizations, and homelessness followed by an early death due to disease, malnutrition, self-harm, and suicide. Is Assisted Outpatient Treatment worse than that? Is it wrong to attempt to save someone’s life who doesn’t have the ability to accurately evaluate reality? We do it all the time with people who have dementia. How is this different?

It’s not hard to set aside the researcher’s contention that the two groups are comparable when, by definition, one cohort has refused to accept voluntary treatment and the other cohort has accepted treatment, making any assertions about their clinical similarities moot. On that one issue they differ as much as it is possible to differ. There were other differences too. I already mentioned the housing issue. But the AOT cohort was more diverse and had a higher representation of people of color than the voluntary treatment cohort had.

Share

This brings up the fact that racism is clearly a factor in this equation. The presence of high numbers of Black/African Americans in the AOT cohort substantiates a concern that Black/African American individuals are more likely to be funneled into a coercive program than a voluntary one. The AOT criteria that includes a history of incarceration sets Black/African Americans up for a higher representation in AOT since more Black/African Americans are incarcerated to begin with. And data shows that Black/African Americans in AOT are more likely to be hospitalized and more likely to be subject to a 9.60 order. Clearly, greater efforts must be made to ensure that Black/African American individuals are offered voluntary services by culturally trustworthy providers and for providers to increase their efforts to provide equitable, culturally sensitive care to those who are receiving AOT services. On this score, the researchers are right to criticize the process that lands a person of color in AOT.

The researchers also criticize the apparent failure of the system to protect the due process rights of the individuals referred for Assisted Outpatient Treatment. For example, they contend that renewals are pro forma and that participants are not included in decisions made about renewals and imply that people are held under AOT orders indefinitely. Yet they found that 46% of the cohort had only one court order (court orders lasted an average of eleven months in the AOT cohort) and that the average number of court orders (including renewals) was only 1.6. So most of the AOT cohort were under AOT orders for less than two years. Yes, some individuals were under repeated court orders (9% had six or more orders) but almost half of the AOT cohort was under AOT orders for less than a year.

An enormous flaw in the basic set of assumptions that underlie this report is the assumption that people in AOT would have accepted services if they’d been offered them. This completely ignores one of AOT’s requirements—that the person has refused to accept services voluntarily. I agree that services may not have been offered in an optimal manner every time but my personal experiences lead me to believe that many of the people who end up on AOT are not able to volunteer due to severe psychosis, paranoia, and anosognosia. That leads me back to the Enhanced Voluntary Agreements or EVAs I mentioned earlier.

Some New York counties had high numbers of Enhanced Voluntary Agreements prior to initiating an Assisted Outpatient Treatment order and others had few or even none. New York City had none yet it has the largest number of AOT participants in the state. The fact that even high EVA counties continued to have a cohort of individuals who met criteria for AOT and were ordered into treatment shows that there are a significant group of individuals who will benefit from an involuntary treatment model like AOT. Enhanced Voluntary Agreements were more frequently used to step someone down from AOT in most counties including in New York City.

New York has taken steps to encourage low Enhanced Voluntary Agreement counties to offer more EVAs through favorable funding mechanisms and greater guidance on their use. Hopefully, they will take steps to collect data that can be used to fine tune when, where, and how voluntary services are offered to people who meet criteria for AOT. It would also be really helpful if New York started substantiating every effort made to offer voluntary services to people who are eligible for them. This would allow researchers to answer the question of whether people who have been mandated into Assisted Outpatient Treatment have repeatedly refused services in the past the next time an attempt is made to evaluate the program in New York.

There is a lot more in this very long report that I could comment on but this is already much, much longer than my usual posts so I will leave it to you to look up the report and review its recommendations and justifications in more detail. It’s a mixed bag of findings and the bias of the researchers shows throughout the document, obscuring some findings that demonstrate the value of Assisted Outpatient Treatment. On the other hand, they raise valid concerns about racism, due process, and the role Enhanced Voluntary Agreements could play in diverting individuals away from an inherently coercive program into voluntary ones where there is less risk of collateral harm to the individuals enrolled in those programs.

In the future, I’ll tackle what takeaways there might be in this report for California. If you’ve gotten this far, please comment and let me know what you think of today’s post. I appreciate and respond to all comments. They make my day. 😊