SB 1120 (the “Bill”), which takes effect on January 1, 2025, amends existing California law to adopt guardrails around the use of artificial intelligence tools for the purpose of utilization management.[1] As discussed in a prior Proskauer alert, the Bill represents one of the latest attempts by the California legislature to regulate the use of AI in the health care industry.

Overview of the Bill’s Requirements

The Bill addresses utilization management and utilization review (“UM/UR”) requirements for “health care service plans”, “disability insurers”, “specialized health insurers,” and their contractors (“Covered Plans”). In addition, the UM/UR requirements apply to Medi-Cal managed care plans, so long as federal funding is not jeopardized by the rules. Notably, these requirements apply to UM/UR functions regardless of whether the functions occur prospectively, retrospectively, or concurrently.[2]

Pursuant to the Bill, Covered Plans that utilize an “artificial intelligence, algorithm, or other software tool”[3] (“AI Tools”) for UM/UR must ensure that the tool’s determination(s) are based on personalized enrollee information. Specifically, prior to making a UM/UR determination, AI Tools must consider (i) an enrollee’s medical history or other clinical history, (ii) individual clinical circumstances as presented by a health care provider, and (iii) other clinical information contained in the enrollee’s medical records.[4] Consistent with the foregoing, AI Tools utilized for UM/UR purposes must not base determinations “solely on a group dataset.”[5]

The Bill Restricts AI-Based Medical Necessity Determinations

At its core, the Bill imposes guardrails on the use of AI Tools in the managed care industry. However, the Bill also expressly restricts the use of AI Tools in making medical necessity determinations. In particular, AI Tools may not be used to “deny, delay or modify health care services based, in whole or in part, on medical necessity” or to supplant a provider’s decision-making.[6]

The Bill underscores that “a determination of medical necessity shall be made only by a licensed physician or a licensed health care professional competent to evaluate the specific clinical issues involved in the health care services requested by the provider…by reviewing and considering the requesting provider’s recommendation, the enrollee’s medical or other clinical history, as applicable, and individual clinical circumstances.”[7] The foregoing requirement aligns with recent guidance issued by the Centers for Medicare and Medicaid Services (“CMS”), which clarifies that AI and algorithms can assist Medicare Advantage organizations (“MAOs”) in making coverage decisions, but that such decisions must be reviewed by a physician or other health care professional with appropriate expertise before the MAO may make a determination.

 The Bill Adopts a Number of Other Restrictions, Which Impose Ambiguous Requirements on a Nascent Industry

In addition to the above, the Bill sets forth that AI tools: (i) may not “discriminate, directly or indirectly, against enrollees”; (ii) must be “fairly and equitably applied”; (iii) must not “directly or indirectly cause harm” to enrollees; (iv) must be disclosed in written policies and procedures; and (v) must be periodically reviewed and revised to ensure accuracy and reliability.

Although positive and focused on protecting enrollees, the foregoing requirements lack specificity, which hopefully will be further clarified by California regulators through issued guidance.

Finally, the Bill sets forth that AI Tools may not use patient data “beyond its intended and stated purpose, consistent with the Confidentiality of Medical Information Act [(“CMIA”)]… and the federal Health Insurance Portability and Accountability Act [(“HIPAA”)].”[8] The foregoing restriction, on its face, appears to unnecessarily incorporate the requirements of the CMIA and HIPAA, as compliance with the CMIA and HIPAA is already required by law. However, the Bill’s prohibition on using patient data “beyond its intended and stated purposes” consistent with these laws creates ambiguity and opens the door for novel enforcement actions under California’s Health and Safety Code and the Insurance Code.

Audit Requirement

Of note, and as is relatively common in the health care sector given its highly-regulated nature, the Bill requires that AI Tools be “open to inspection for audit or compliance reviews” pursuant to applicable state or federal law.[9]

Implications

The Bill is a first step at providing a regulatory framework to payors, contractors, and IT developers that are using, or intend to use, AI Tools for UM/UR functions. However, more guidance and specificity is likely warranted to provide certainty in a highly regulated and fiercely-competitive period of time in the industry.

[1] The Bill amends Section 1367.01 of the Health and Safety Code, and Section 10123.135 of the Insurance Code.

[2] See Cal. Health & Safety Code § 1367.01(k)(1)(A) (eff. Jan. 1, 2025); Cal. Insur. Code § 10123.135(j)(4) (eff. Jan. 1, 2025).

[3] The Bill defines “artificial intelligence” as “an engineered or machine-based system that varies in its level of autonomy and that can, for explicit or implicit objectives, infer from the input it receives how to generate outputs that can influence physical or virtual environments.” See Cal. Health & Safety Code § 1367.01(k)(3) (eff. Jan. 1, 2025) and Cal. Insur. Code § 10123.135(j)(3) (eff. Jan. 1, 2025). However, it does not define “algorithm” or “other software tool,” which are words that can be broadly construed.

[4] See Cal. Health & Safety Code § 1367.01(k)(1)(A) (eff. Jan. 1, 2025); Cal. Insur. Code § 10123.135(j)(1)(A) (eff. Jan. 1, 2025).

[5] Cal. Health & Safety Code § 1367.01(k)(1)(B) (eff. Jan. 1, 2025). Cal. Insur. Code § 10123.135(j)(1)(B) (eff. Jan. 1, 2025).

[6] Cal. Health & Safety Code § 1367.01(k)(2) (eff. Jan. 1, 2025); Cal. Insur. Code § 10123.135(j)(2) (eff. Jan. 1, 2025).

[7] Id. (emphasis added).

[8] Cal. Health & Safety Code § 1367.01(k)(1)(J) (eff. Jan. 1, 2025); Cal. Insur. Code § 10123.135(j)(1)(J) (eff. Jan. 1, 2025).

[9] See Cal. Health & Safety Code § 1367.01(k)(1)(G) (eff. Jan. 1, 2025); Cal. Insur. Code § 10123.135(j)(1)(J) (eff. Jan. 1, 2025).