LLM Responses: How to Change, Control,
and Improve Them (Healthcare Case Study)

Large Language Models (LLMs) have changed how organizations handle interactions, share information, and assist users. But straight out of the box, even the top LLMs often fall short in high-stakes areas, like healthcare. They hallucinate facts, provide generic or unsafe recommendations, or miss local context entirely.

This article explores how LLM responses can be changed, controlled, and improved. We illustrate practical approaches tested across multiple methods: Prompt Engineering, Retrieval-Augmented Generation (RAG), and Parameter-Efficient Fine-Tuning (PEFT). While the examples are drawn from a healthcare-focused case study, the principles apply to any domain that requires precise, reliable, and context-aware AI outputs.

LLM outputs don’t always line up with what a user needs or what an organization expects. Sometimes the answers are a bit off — inaccurate, biased, too general, or just not consistent. That can lower trust and make responses less useful.

Changing how models behave can make their responses more on point, more relevant, and better fit for what’s really needed. It helps people, supports decisions, and prevents wrong or awkward answers.

Understanding how LLMs produce output helps explain why interventions are necessary.

LLMs generate responses by combining:

• Patterns learned from massive text corpora during pretraining
• Context from the user’s input and prompt
• Decoding strategies (like temperature or top-k) that influence creativity versus precision
• Their internal world model, which is static unless adapted

By default, LLMs are generalists. They do not inherently know your domain, rules, or preferred style unless you provide that information explicitly through prompts or adapt the model’s knowledge.

This is why techniques like Prompt Engineering, RAG, and PEFT exist — to control, customize, and enhance responses.

Base models fall short for several reasons:

1. Limited Domain Knowledge

Models don’t know your internal documentation, product specifics, or up-to-date local data. They cannot answer location-specific or industry-specific questions reliably without external context.

2. Inconsistent Tone & Formatting

LLMs often drift away from brand voice, change structure unexpectedly, or fail to maintain strict output formats like JSON, tables, or templates.

3. Hallucinations

LLMs may confidently invent facts — a critical failure in domains like healthcare, compliance, or law.

4. Lack of Specialized Skills

They can perform many tasks passably, but struggle with niche, complex workflows that require deep expertise.

Because of these limitations, changing and controlling LLM responses becomes essential.

Healthcare forces an LLM to demonstrate:

• Accuracy: Wrong facts can cause harm.
• Sensitivity: Tone matters in emotional or clinical conversations.
• Local Knowledge: Advice must reflect local laws, clinics, and insurance systems.
• Safety Compliance: Models must handle suicide/self-harm queries responsibly.
• Consistency: No hallucinations, no improvisation, no ambiguity.

In our test, we simulated a patient persona:

• Alex, recently diagnosed with Type 2 Diabetes, asking real questions about life, care, costs, and emotional support.

There are three dominant strategies for controlling LLM responses. Each solves different problems and requires different effort levels.

Method 1 — Prompt Engineering

Prompt engineering is the lowest-cost, fastest way to influence LLM behavior.

It changes responses without training the model.

Concept: Prompt Engineering is the discipline of crafting detailed inputs to guide the model’s response. It’s the fastest way to influence behavior without changing the model itself. We’ll look at two key techniques.

Key Technique 1: Conditional Directives (Safety Overrides)

What it is: Embedding explicit IF-THEN style rules into the prompt to handle high-stakes topics. This forces the model to abandon its normal response generation and follow a strict, predefined script when specific keywords or themes are detected.

What it solves: Enforces critical safety procedures. For example, it can override a generic, non-local response to a self-harm query with a specific, regional helpline, ensuring the user receives the most appropriate information.

IF the user mentions self-harm, THEN your entire response must be ONLY this…

This forces the LLM to use a predefined, safe, region-specific response.

For example, the model was instructed to say:

Please know there is immediate help available.

You can call SAMHSA National Helpline at 1-800-662-HELP (4357)…

The override replaced the model’s incorrect U.S. hotline numbers.

Key Technique 2: Few-Shot / One-Shot Examples

What it is: Including examples of the desired input and output directly within the user’s prompt.

What it solves: Teaches the model the exact format you want for the output. It’s incredibly effective for fixing unreliable formatting issues.

This teaches the LLM the desired output format and local accuracy by providing example Q&A pairs.

Example:

When asked about support groups in Orlando, the prompt’s examples instructed the model to mention organizations such as

• Orlando Health
• AdventHealth
• American Diabetes Association (ADA)

Results

Prompt engineering transformed vague answers into highly specific, structured ones without modifying the model.

Method 2 — Retrieval-Augmented Generation (RAG)

RAG solves the “knowledge gap” by connecting the LLM to an external, trusted data source. Instead of relying on the model’s static, pre-trained memory, we give it the exact information it needs to answer a question, just in time.

How RAG Works (The 3-Step Process):

• Retrieve: When a user asks a question, the system first searches a private knowledge base (like a collection of documents or a database) for relevant information. This is often done using a Vector Database.
• Augment: The relevant text snippets found in the search are then combined with the user’s original question to create a new, much more detailed prompt.
• Generate: This “augmented” prompt is sent to the LLM. The model now has all the necessary context to generate a factual, accurate answer based on the provided data.

Analogy:

You’re not asking the model to remember the answer to a test question. You’re giving it the textbook and telling it which page to read.

Results

RAG cut hallucinations dramatically and supported accurate, current responses for local healthcare situations.

Method 3 — Parameter-Efficient Fine-Tuning (PEFT)

PEFT teaches the model new skills by training only a small set of additional parameters (e.g., LoRA, QLoRA).

It is ideal for:

• Adding medical writing style
• Teaching local terminology
• Reducing generic responses
• Creating clinical persona consistency

Why PEFT?

• PEFT (Parameter-Efficient Fine-Tuning) is the modern, efficient way to do fine-tuning.
• Instead of retraining the entire model (which is slow and expensive), PEFT freezes the original model and only trains a tiny set of new parameters, like a small plugin or “adapter.”
• This is much faster, requires less data, and results in a tiny adapter file, not a whole new multi-gigabyte model.

What is QLoRA?

QLoRA (Quantized Low-Rank Adaptation) is a technique that makes fine-tuning more accessible. It follows a simple but powerful two-step process:

1. Quantize: It takes a large, pre-trained model and shrinks it down to a 4-bit version.
2. Adapt: It then performs the standard LoRA fine-tuning on this smaller, quantized model.

By training on the compressed model, QLoRA dramatically reduces the memory needed for fine-tuning, making it possible to adapt huge models on a single, consumer-grade GPU.

Example Results

After QLoRA fine-tuning:

Fine-Tuning (PEFT) pros and cons

 Each method progressively enhances control, accuracy, and user trust.

Whether you’re building a medical chatbot, a triage assistant, or a patient support system, improving each LLM response requires a layered strategy:

1. Start with Prompt Engineering

Set personality rules, add safety instructions, provide examples, and define structure, tone, and disclaimers.

2. Add RAG for Accuracy

Use embeddings and a vector database to pull in verified medical policies, clinic details, insurance info, and an always-updated knowledge base.

3. Use PEFT for Domain Behavior

Train the model on your organization’s communication style and real conversations to enhance accuracy in specialized areas like diabetes, oncology, or mental health.

The best systems combine all three layers.

Evaluation must be systematic, especially in healthcare.

Healthcare systems usually require a hybrid stack: Prompt Engineering + RAG + PEFT.

Changing and controlling LLM responses is not optional—it’s essential for safe, accurate, and trustworthy AI in healthcare.

  Our case study shows that:

• Prompt Engineering offers instant control
• RAG provides factual grounding
• PEFT delivers long-term behavioral consistency

Together, these methods form the foundation for AI systems in high-stakes environments. At DevCom, we combine structured methods, domain datasets, and robust evaluation to turn LLMs into specialized, reliable assistants. For support designing controlled LLM systems in healthcare, finance, or regulated industries, our team is ready to help.