RDEL #72: how can engineering teams resolve challenges with integrating LLMs?

The top challenges, related distractions, and solutions for engineering teams working with LLMs

Welcome back to Research-Driven Engineering Leadership. Each week, we pose an interesting topic in engineering leadership and apply the latest research in the field to drive to an answer.

Large Language Models (LLMs) are transforming software products across industries, offering new capabilities but also introducing significant engineering challenges. As these models disrupt traditional workflows, engineers must navigate uncharted territory, from changing quality standards to evolving testing practices. This week we ask: How can engineering leaders resolve the common challenges with integrating LLMs into products?

The context

The integration of LLMs into software products has become a cornerstone for innovation in various fields, including healthcare, legal services, and enterprise software. Unlike traditional software, LLM-based features demand new engineering paradigms. Their non-deterministic nature, reliance on subjective quality metrics, and operational costs challenge established engineering norms.

Teams are now faced with reconciling these unique demands with existing software engineering workflows, which often lack the tools or processes to address these complexities. This gap has spurred the need for innovative solutions to ensure quality, fairness, and reliability in LLM-enabled systems.

The research

A recent mixed-method study from Carnegie Mellon University and Microsoft Research investigated these challenges by interviewing 26 practitioners and surveying 332 developers involved in integrating LLMs. The research identified the top challenges, and categorized their impact into disruptions and solutions to quality assurance in LLM-based products.

First, researchers organized the most common challenges into the following nine categories:

1. Lack of specification: A bug is ambiguously defined.

2. Subjectivity: there are not clear expectations of what a correct answer is.

3. Metrics dilemma: developing the right metrics is complicated.

4. LLM properties: because LLMs are made non-deterministic, teams struggle with testing outputs that aren’t consistently reproducible.

5. Evaluation: lack of robust evaluation methods and pipeline.

6. Manual efforts: manual testing is common, but labor intensive and inefficient

7. Infrastructure constraints: there is an over-dependence on the current evaluation infrastructure that lacks flexibility for all use cases.

8. Model migration issues: minor tweaks can cause substantial changes.

9. Compliance: processes are lengthy, manual, and often bureaucratic.

A sample of known challenges and the qualitative responses around those challenges

Next, researchers reviewed the impact of those challenges into the top disruptions, the challenges most likely to cause those disruptions, and the top reported solutions to resolve them. (We will be reporting the top few solutions for each disruption - for more inspiration, check out the full paper).

1. Evaluation metrics for LLMs change frequently.

   1. Caused by challenge: #1, #2, #3, #4, #6

   2. Top solutions

      1. Define custom metrics through iterative collaboration and expert consultations (57% adopted, 55% satisfied, 2% did not find useful).

      1. Combine qualitative and quantitative metrics to evaluate the multifaceted outputs effectively (54.4% adopted, 53.7% satisfied, 0.7% did not find useful)

      2. Evaluate subjective metrics using LLM validators (50.6% adopted, 47.3% satisfied, 3.3% did not find useful).

      3. Establish clear rubrics and scoring mechanisms (33.8% adopted, 33.1% found useful, 0.7% did not find useful)
         

2. Common assumptions about test processes and environments break.

   1. Caused by challenge: #5, #6, #9

   2. Top solutions

      1. Establish internal team standards for evaluation processes and pipelines (48.7% adopted, 48% satisfied, 0.7% did not find useful)

      2. Automate offline evaluation to run periodically on a schedule (29.6% adopted, 28.9% satisfied, 0.7% did not find useful)
         

3. Engineers require new skills to handle LLM evaluations.

   1. Caused by challenge: #4

   2. Top solutions:

      1. Involve data scientists in authoring tests, as they understand the limitations of the model better (33.5% adopted, 32.1% satisfied, 1.4% did not find useful)

4. Even with extensive evaluations, LLM solutions remain unreliable and developers have difficulty establishing trust.

   1. Caused by challenge: #4, #5

   2. Top solutions:

      1. Employ canary release strategy to enhance confidence in LLM outputs (standardized practice, no adoption stats available)

      2. Use A/B testing to track chanegs in different versions (ie prompt updates) (36.7% adopted, 36.7% satisfied, 0% did not find useful)

      3. Establish extensive guardrails (44.2% adopted, 41.3% satisfied, 2.9% did not find useful)

5. Existing approaches to telemetry and monitoring need to be revised.

   1. Caused by challenge: #3, #9

   2. Top solutions:

      1. Develop new and multiple telemetry metrics that may better suit LLM solutions (58.1% adopted, 57.4% satisfied, 0.7% did not find useful).

      2. Use the LLM-as-a-validator strategy to gain granular insight in production behavior without revealing private data (24.8% adopted, 24.1% satisfied, 0.7% did not find useful)

6. Lack of focus on focus on system-wide evaluation of LLM based products.

   1. Caused by challenge: #5

   2. Top solutions:

      1. Set up an end-to-end test automation infrastructure (38.2% adopted, 37.6% satisfied, 0.6% did not find useful)

      2. Conduct comprehensive tests beyond unit tests, including reliability and availability (61% adopted, 60.3% satisfied, 0.7% did not find useful)

7. Responsible AI is a relatively new concept, has a steep learning curve, and might be bureaucratic.

   1. Caused by challenge: #9

   2. Top solutions:

      1. Standardize Responsible AI (RAI) evaluation and practices. The most common were bias and fairness evaluation (39.7% adopted), safety evaluation (55.9% adopted), privacy compliance (55.8% adoption), and security and vulnerability adoption (59.8% adoption)

      2. Apply RAI red teaming strategies (48% adoption)

      3. Follow a robust RAI audit process (mandatory for adoption at Microsoft)

The application

The findings of this paper point out just how new and nascent this technology is, and how teams are adapting their software development processes to manage these new integrations. Even still, there is no clear best practice for developing products with LLMs, which is why it can feel so experimental to build a development pipeline.

For engineering teams who are integrating with LLMs within their core offering, consider the top challenges you are currently encountering, and use this guide to discover a new solution that you can use to adjust your existing processes. If you are just getting started, consider choosing the most popular strategy as a baseline, and to adapt to the development practices, security standards, and culture of your own engineering team.

—

Thats it for this week’s RDEL. We hope you enjoyed this week’s dose of research!

Lizzie