Context and my contribution

Spark provides software that Medicare agents and agencies use to manage client relationships and run their businesses. Plan Fit applies the beneficiary information available within that environment to help agents evaluate plans.

I initiated the project to investigate whether publicly available CMS data could support better plan-selection tools. As Spark’s founding product designer, I built the plan database, developed the filtering and comparison logic, and designed the tool. I used AI as a development partner and tested the model against historical enrollment data supplied by our data team.

That testing changed the solution substantially: an approach that looked mathematically sound was favoring plans associated with poor enrollment outcomes.

From available plans to relevant choices

An agent begins with a beneficiary’s ZIP code and county, establishing the available plans. Plan Fit then applies what is known about that person—their doctors, prescriptions, health needs, and coverage circumstances—to narrow the options.

Plan Fit uses information collected through Spark’s broader workflows. An agent might access a client-update workflow while using the tool, but collecting that information is separate from the plan-selection logic.

The concept combines two kinds of narrowing:

• Filter for fit. Remove plans that don’t meet the beneficiary’s requirements.
• Compare the remaining options. Use Pareto elimination to remove plans that are worse than another option on the dimensions being compared.

The result is a shortlist of plans with meaningful tradeoffs. The agent and beneficiary still decide which tradeoffs make sense.

Building a reliable foundation

CMS publishes extensive Medicare Advantage plan data, but it doesn’t arrive as a single, usable plan database. Benefits, service areas, star ratings, and formularies live in separate files, with identifiers and structures that require careful reconciliation.

I built a SQLite database consolidating 8,000+ plan segments across 3,000+ counties, along with star ratings and drug-tier information. The work involved building data pipelines, validating joins, and investigating discrepancies before the data could support meaningful comparisons.

This also exposed the limits of the source material. Exact prescription costs and provider-network checks required information beyond the CMS datasets I had assembled. The database provided a foundation for comparison, with further integrations needed to support the complete plan-selection workflow.

On top of that foundation, I built a Python library for querying, filtering, and comparing plans, with Streamlit interfaces for exploration and testing. Working directly with the data let me investigate product questions and test approaches without waiting for a separate implementation cycle.

Where AI helped and where it didn’t belong

I used AI as a development partner to build pipelines, implement business logic, and create validation tools. The plan comparisons themselves followed explicit, deterministic rules.

That distinction was essential: AI could help me build the system, but the system needed to produce repeatable results grounded in plan data. Its reasoning had to be inspectable so I could test it, identify failures, and revise the logic.

Testing changed the model

The first model compared plans across three dimensions: net cost, financial risk exposure, and supplemental benefits. Pareto elimination removed plans that were outperformed on all three, leaving options that represented different tradeoffs.

I also introduced tolerance bands so small differences wouldn’t create false distinctions between otherwise comparable plans.

A promising result and a reason to question it

In tests across six counties, the model eliminated 88–98% of plans. In Los Angeles, it reduced more than 120 options to just 7.

That showed the approach could simplify a large choice set. It didn’t establish that the remaining plans were good choices.

To investigate that question, I tested the model against 152,000 historical Medicare Advantage enrollments. Plans retained for their low cost were associated with substantially poorer enrollment outcomes than those retained for lower financial risk.

The model was following its rules. The results made me question whether those rules represented what mattered.

What the first model missed

The cost calculation favored $0-premium plans with large Part B givebacks. Those plans looked attractive on the net-cost axis, but that number alone couldn’t establish whether they would serve a beneficiary well.

Two other weaknesses emerged. Supplemental benefits added complexity without providing a useful independent distinction in the model. And the comparison did not account for CMS star ratings, allowing poorly rated plans to compete on the same terms as highly rated ones.

The lesson was consequential: reducing the number of choices wasn’t enough. I needed to examine which choices survived and why.

Revising the comparison

I explored several responses, including adjusting tolerances and adding carrier-aware scoring. The revised model made two principal changes:

• Simplify the comparison to cost and risk protection. Supplemental benefits remained available as information and filters, rather than forming a third optimization axis.
• Apply a quality threshold before Pareto elimination. Plans initially needed a CMS star rating of at least 4.0. In markets where that excluded more than 90% of options, the threshold relaxed in half-star increments.

This preserved the core product concept: help agents work with a smaller set of meaningful tradeoffs. The change was in how the model determined which plans entered that comparison.

Test the revision, too

I evaluated the revised model against the historical enrollment data. The results improved, with the revised model approaching the performance of the strongest field-agent segment in that comparison.

This was retrospective evidence, not a demonstrated improvement from agents using Plan Fit in practice. It supported further development while leaving prospective testing as an essential next step.

The most important design change came from testing the assumptions beneath the interface. A shortlist can look clear and useful while systematically preserving the wrong options. Building the model myself let me investigate that failure and change the logic behind the experience.

Validating against historical enrollment outcomes

Spark’s data team tracked enrollment outcomes primarily to understand agent behavior. Because those records linked applications to both agents and plans, I used the same data to investigate a different question: were particular plans or plan types associated with higher losses?

The dataset contained 152,000 historical enrollments. Some plans performed substantially worse on average, and outcomes also varied across agent groups. Our strongest field agents achieved quality-enrollment rates above 80%, while some call-center agents retained compensation on roughly half of their submitted enrollments.

That gave Plan Fit a practical target: identify a shortlist of plans whose observed enrollment quality could match the performance of our best agents.

Defining enrollment quality

For this analysis, quality-enrollment rate meant the percentage of submitted applications that became active and survived the first 90 days. It is a metric used in this analysis, not an industry-standard measure.

Quality-enrollment rate = accretion rate × (1 − rapid-disenrollment rate)

Accretion measures how many applications became active. Rapid-disenrollment rate measures the proportion of those active enrollments lost within 90 days.

For example, out of 100 submitted applications, 80 might become active. If 10% of those rapidly disenrolled, 8 would be lost, leaving 72 quality enrollments.

These surviving enrollments represent business for which the agent and Spark retain carrier compensation. I also used retention as a proxy for beneficiary satisfaction, while recognizing that it does not directly measure satisfaction or explain an individual’s reason for leaving.

Historical comparison results

The table compares observed historical enrollment outcomes for the plans identified by the original and revised models. These are retrospective comparisons, not results from agents using Plan Fit.

Quality-enrollment rates use submitted applications as the denominator. Rapid-disenrollment rates use enrollments that became active.

What the results establish

The evaluation used observed historical outcomes, not outcomes generated by deploying Plan Fit. It helped me identify patterns across plans and plan types, challenge the first model’s assumptions, and evaluate the revised shortlist against an existing business benchmark.

Those associations supported further development. They did not establish that plan selection alone caused the differences between agents, or that deploying Plan Fit would produce the same results. The analysis also did not measure longer-term retention or lifetime value.

Project status as of October 2026

When I left Spark, Plan Fit was being handed off to engineering. The screens shown here represent my design concepts at that point.

The team subsequently implemented the work as a recommendation feature rather than the shortlist interface shown in this case study. In a recent conversation, the PM reported that it had entered beta with a small group of agents.

The upcoming Annual Enrollment Period will provide the first substantial opportunity to evaluate its use in practice. The historical analysis informed the model, but the results presented here are not outcomes from the deployed feature.

I plan to follow up in early 2027 to learn how the feature performed during enrollment and, as follow-up data becomes available, whether those enrollments became active and remained active through their first 90 days.

The project demonstrates how I approach an unfamiliar product problem: investigate the underlying data, build something testable, challenge its assumptions, and revise the solution when the evidence points elsewhere.