POMs and Circumstance at Policy2Code Demo Day at BenCon 2024
The team explored using LLMs to interpret the Program Operations Manual System (POMS) into plain language logic models and flowcharts as educational resources for SSI and SSDI eligibility, benchmarking LLMs in RAG methods for reliability in answering queries and providing useful instructions to users.
Determining eligibility for Supplemental Security Income (SSI) and Social Security Disability Insurance (SSDI) can be complex, often requiring court adjudication in appeals process based on various evaluations outlined in the Program Operations Manual System (POMS). This team was interested in using LLMs to interpret POMS into plain language logic models and flowcharts that can provide educational resources for people with disabilities. They used PolicyEngine as a baseline and benchmarked LLMs in RAG methods for the reliability in answering queries and the capability of giving more elaboration and helpful instructions to users. They used their test dataset of question/answer pairs to calculate the performance of different methods and compared the score.
Share this Resource:
