As the implementation and integration of artificial intelligence and machine learning tools (AI) continue to affect nearly every industry, concerns over AI’s potentially discriminatory effects in the use of these tools continue to grow. The need for ethical, trustworthy, explainable, and transparent AI systems is gaining momentum and recognition among state and local regulatory agencies—and the insurance industry has not escaped their notice.

On January 17, 2024, the New York State Department of Financial Services (“NYSDFS”) took a further step towards imposing substantial obligations on insurers with its Proposed Insurance Circular Letter, entitled Use of Artificial Intelligence Systems and External Consumer Data and Information Sources in Insurance Underwriting and Pricing (“Proposed AI Letter”).[1]

The Proposed AI Letter—aimed at all insurers authorized to write insurance in the state of New York, Licensed Fraternal Benefit Societies, and the New York State Insurance Fund—calls on those using external consumer data and information sources (“ECDIS”) and Artificial Intelligence Systems (“AIS”) to establish organizational governance and risk management frameworks to assess and mitigate systemic bias, inequality, and discriminatory decision making or other adverse effects attendant with underwriting and pricing of insurance policies.

The Proposed AI Letter comes on the heels of a Model Bulletin of the National Association of Insurance Commissioners (NAIC) on the use of artificial intelligence systems, adopted in December 2023, wherein NAIC reminds all insurers that those using AI systems must comply with all applicable federal and state insurance laws and regulations, including those addressing unfair discrimination.

It is now clear with the publication of the Proposed AI Letter that NYSDFS may audit and examine insurers’ use of ECDIS and AIS— “including within the scope of regular or targeted examinations pursuant to New York Insurance Law §309, or a request for a special report pursuant to §308.” Thus, insurers should be prepared to thoroughly assess whether the use of AIS/ ECDIS in underwriting or pricing is unfairly discriminatory.

The 2024 Proposed AI Letter expands upon a 2019 Insurance Circular Letter No. 1 (“2019 Letter”) issued by NYSDFS that was the result of an investigation of insurers’ underwriting guidelines and practices relating to the use of external data in underwriting for life insurance. Specifically, the investigation found that insurers’ use of external data sources “has the strong potential to mask” discrimination prohibited by New York’s insurance laws.

The 2024 Proposed AI Letter adds greater specificity to an insurer’s obligations, while also clarifying a point regarding the disclosure of objective threshold criteria for an “accelerated” underwriting process. The 2019 Letter provides guidance on the use of “unconventional sources or types of external data available to insurers, including within algorithms and predictive models.” NYSDFS’s 2024 Proposed AI Letter specifically targets AIS.

AIS is defined as “any machine-based system designed to perform functions normally associated with human intelligence, such as reasoning, learning, and self-improvement, that is used—in whole or in part—to supplement traditional medical, property or casualty underwriting or pricing, as a proxy for traditional medical, property or casualty underwriting or pricing, or to establish ‘lifestyle indicators’ that may contribute to an underwriting or pricing assessment of an applicant for insurance coverage.” Similarly, ECDIS would include data or information used, in whole or in part, for the same purpose.

Proposed Guidance

While recognizing the benefits of AIS and ECDIS in the underwriting and pricing processes, the Proposed AI Letter directs insurers to develop and manage their use of these tools with an approach that is “reasonable and appropriate to each insurer’s business model.” Thus, as set forth in the proposed letter, insurers should be prepared to do the following (not an exhaustive list):

1. Establish that data sources or models that use AIS and ECDIS for underwriting and pricing purposes are not unfairly or unlawfully discriminatory or harmful to any protected class under New York’s insurance laws. (9) Additionally, insurers should be able to demonstrate that the ECDIS employed for these tasks do not serve as a proxy for any protected classes that may result in unfair or unlawful discrimination. (11)
2. Demonstrate that the ECDIS are supported by generally accepted standards of practice and are based on actual or reasonably anticipated experience (e.g., statistical studies, predictive modelling, risk assessments). The analyses should demonstrate a clear, empirical, statistically significant, rational, and non-discriminatory relationship between the variables used and the relevant risk of the insured. (10)
3. Document the processes and reasoning behind their methodologies and analysis of AIS/ECDIS to address unfair or unlawful discrimination with an expectation that they may be called upon to supply this documentation to NYSDFS. (15)
4. Establish a corporate governance framework to provide appropriate oversight of the use of AIS/ECDIS to ensure compliance with New York’s Insurance Law, specifically inclusive of Board of Director oversight and Senior Management responsibility for day-to-day implementation of AIS. (19-23)
5. Formalize the development and management of AIS/ECDIS in written policies and procedures that should include clearly defined roles and responsibilities, training, and more. (24)
6. In addition to 3 above, maintain comprehensive documentation regarding the use of AIS/ECDIS itself, whether developed internally or supplied by third parties. Documentation may include, for example, descriptions of processes for identifying and assessing risks associated with AIS/ECDIS; an inventory of AIS implemented for use, a description of how each AIS/ECDIS operates, a description of the process for monitoring usage and performance, etc. (28)
7. Implement procedures to respond to consumer complaints about the use of AIS/ECDIS. (29)
8. Implement Risk Management and Internal Controls regarding the use of AIS/ECDIS. This includes an internal audit function to assess the overall effectiveness of the AIS/ECDIS risk management framework, which may include verification of proper adherence to acceptable policies and procedures, verifying records of use and validation, assessing the accuracy and completeness of documentation, assessing potential biases, and more. (30-33)
9. Ensure the compliance of all third-party vendors with respect to AIS/ECDIS (34-35) and
10. Be prepared to disclose, in a transparent manner, the reasoning behind an adverse decision with respect to coverage, and to provide proper notice, when using AIS/ECDIS. (36-40)

Particularly with “1” above, the Proposed AI Letter sets forth quantitative and qualitive benchmarks for assessing whether the use of AIS/ECDIS tools produces disproportionate adverse effects in similarly situated insureds or members of a protected class. This adverse impact analysis should be administered before using AIS/ECDIS and “on a regular cadence thereafter,” as well as whenever “material updates or changes are made.” (16) The Proposed AI Letter provides some flexible guidance in assessing discriminatory effect and encourages the use of multiple statistical metrics.

The Proposed AI Letter notes that insurers should be prepared to explain, “at all times, how the insurer’s AIS operates and to articulate the intuitive logical relationship between ECDIS and other model variables with an insured or potential insured individual’s risk.” (18) (emphasis added). Of note, a footnote in the Proposed AI Letter references New York Insurance Laws, including Section 2303 that specifically prohibits unlawful discrimination in pricing/rates for property and casualty policies. However, it is not clear whether the Proposed AI Letter is limited to individual consumer coverage. Arguably, the comment period that ends mid-March will help to clarify this point in the final NYSDFS iteration. We will certainly be following developments and reporting further thereon.

Takeaways:

• Compliance rests squarely on the shoulders of the insurer and remains a nondelegable duty. Thus, third-party vendors contracted for AI design, deployment, maintenance, and auditing remain the responsibility of the insurer. Written standards, contractual obligations, timely reporting, and vendor auditing should be developed and reviewed on an ongoing basis.
• Risk management should include the development of standards and protocols for the design, development, validation, and deployment of AIS/ECDIS.A specific risk assessment team dedicated to oversight of AIS should be considered.
• Documentation of the insurer’s efforts in the development, execution, and evaluation of internal processes and monitoring of the reliability and accuracy of the AIS/ECDIS will be critical in defending actions related to underwriting and pricing.
• Insurers should be prepared to adequately and transparently explain all negative decisions that may arise from the use of AIS/ECDIS. An important aspect of the Proposed AI Letter is the obligation to provide notice to the insured when adverse action is taken that arises from the application of AIS/ECDIS, which includes the right of the affected insured to obtain information about the underlying data that resulted in the adverse decision.
• Consumer complaints will be reportable to the NYSDFS; thus, adequate and reasonable measures should be undertaken by the insurer to ensure a viable complaint procedure and complaint documentation.

Epstein Becker Green Staff Attorney Ann W. Parks contributed to the preparation of this post.

ENDNOTES

[1] New York State Department of Financial Services, Proposed Insurance Circular Letter (Jan. 17, 2024) at https://www.dfs.ny.gov/industry_guidance/circular_letters/cl2024_nn_proposed