LLMs and RAG in Agentforce: How Agents Understand Your Data

The Core Problem

An AI agent based on a Large Language Model (LLM) alone is unreliable for business. Here’s why:

LLMs are trained on public internet data (cutoff dates in 2023-2024). They don’t know:

• Your customer information
• Your internal documentation
• Your product specifications
• Your company policies
• Your recent market data

So when your customer asks “How do I reset my password?”, the LLM doesn’t have access to your knowledge base. It makes a plausible-sounding guess. Sometimes it’s right. Sometimes it’s wrong—and you lose the customer.

This is hallucination. And it’s why most AI chatbots fail in production.

The solution: Retrieval-Augmented Generation (RAG).

What Is RAG?

RAG is deceptively simple: Before the LLM answers a question, retrieve the relevant documents from your business data. Then feed those documents to the LLM as context.

Without RAG:

1. User asks: “What’s your refund policy?”
2. LLM guesses based on general training data
3. Agent gives a wrong answer

With RAG:

1. User asks: “What’s your refund policy?”
2. RAG searches your documentation for “refund policy”
3. RAG retrieves your actual refund policy document
4. LLM reads the document and answers based on your actual policy
5. Agent gives a correct answer with a source citation

How RAG Works in Agentforce

Agentforce makes RAG work by connecting three components:

1. The Knowledge Source

Your data: knowledge base articles, help docs, product specs, CRM data, database records.

In Agentforce, knowledge sources include:

• Salesforce Knowledge Base
• Content management systems
• Custom databases
• External documents (PDFs, web pages)

2. The Retrieval Engine

When an agent needs to answer a question, the retrieval engine:

1. Converts the question into a semantic search query
2. Finds the most relevant documents from your knowledge source
3. Ranks results by relevance
4. Returns the top 3-5 documents

3. The LLM

The LLM reads the retrieved documents, then answers the question based on them.

LLM prompt (simplified): ``` You are a helpful customer support agent. Answer the following question using ONLY the provided documents. If the answer isn’t in the documents, say “I don’t know.”

Documents: [retrieved content here]

Question: What’s your refund policy?

Answer: [generates answer from documents] ```

Why This Matters for Your Business

RAG Improves Quality

• Accuracy: Agents reference your actual data, not guesses
• Currency: Agents answer based on your latest docs (updated in real time)
• Traceability: Every agent answer comes with a source citation

RAG Reduces Risk

• Hallucinations prevented: Agent can’t answer what’s not in your docs
• Brand safety: Agents can’t make promises your company didn’t make
• Compliance: Audit trail shows which documents informed each decision

RAG Improves ROI

• Faster resolution: Agent finds answers instantly (no human search)
• Higher resolution rate: Agents resolve 50-70% of conversations without escalation
• Lower support costs: One agent replacing 2-3 humans

The Hidden Complexity: Knowledge Quality

RAG only works if your knowledge source is good. Here’s the reality:

Most companies have terrible knowledge bases:

• Outdated documentation (2+ years old)
• Contradictory information (Policy A says X, Policy B says Y)
• Missing content (gaps in coverage)
• Poor organization (can’t find anything)

A RAG system is only as good as the data it retrieves.

Preparing Your Knowledge Base for RAG

Before deploying an agent with RAG:

1. Audit your docs

   • What’s outdated? (Flag for refresh)
   • What’s contradictory? (Resolve conflicts)
   • What’s missing? (Fill gaps)

2. Organize by intent

   • Group related content (not by department)
   • Use consistent terminology
   • Link related topics

3. Test retrieval

   • Run sample questions
   • Check if retrieval returns the right docs
   • Refine search parameters

4. Monitor in production

   • Track retrieval accuracy (is the right doc returned?)
   • Track answer quality (is the agent answering well?)
   • Adjust weights and rankings over time

RAG + Prompt Engineering = Reliable Agents

RAG prevents hallucination, but LLMs are still unpredictable. That’s where prompt engineering comes in.

Your agent’s prompt needs to tell the LLM:

• What to do: “You are a customer support agent”
• What to use: “Use ONLY the provided documents”
• What to do when stuck: “If you don’t know, escalate to a human”
• How to behave: “Be friendly but professional. Never make promises about timelines”
• What boundaries exist: “Don’t discuss pricing. Refer to the sales team”

Example (simplified): ``` You are a helpful customer support agent for Acme Corp.

Your job:

1. Answer customer questions using the provided knowledge base documents
2. If the answer isn’t in the docs, tell the customer you don’t know and escalate

Tone: Friendly, professional, concise Escalation triggers:

• Angry or threatening customer
• Technical issue you can’t resolve
• Billing dispute over $100
• Request for pricing or demos

You MUST:

• Always cite the source document
• Never make promises about refunds or returns
• Never discuss competitor products
• Admit when you don’t know something ```

With good prompts + RAG, you get reliable agents.

Real-World Example: Resolve Service Agent

In Funnelists’ Resolve product, we use RAG like this:

1. Customer asks: “How do I cancel my subscription?”
2. RAG retrieves: Your subscription cancellation policy doc
3. Agent reads: The document (it says “3-day free trial, $50 cancellation fee after”)
4. Agent answers: “You can cancel anytime. If you’re in the free trial, no fee. After that, there’s a $50 cancellation fee. Want me to process that now?”
5. Result: Customer knows exactly what to expect. No hallucinations, no surprises.

This conversation happens 500+ times/month without human involvement. That’s 500 support tickets you don’t have to handle manually.

Common Mistakes with RAG

Mistake 1: Uploading Unstructured Data

RAG works best with organized, searchable data. Uploading a folder of PDFs and hoping for the best rarely works.

Fix: Structure your knowledge base. Use metadata tags (topic, audience, urgency). Make search easy.

Mistake 2: Not Monitoring Retrieval Quality

You deploy the agent and assume it’s working. But maybe it’s retrieving the wrong documents 30% of the time.

Fix: Track retrieval accuracy. Sample conversations weekly. Check if the right docs were retrieved.

Mistake 3: Ignoring Prompt Limitations

You think RAG solves everything. But a bad prompt can override good retrieval.

Fix: Treat prompts as seriously as retrieval. Refine them based on real conversations.

The Path Forward

If you’re building an agent in Agentforce:

Week 1: Audit your knowledge base. Identify gaps and outdated content.

Week 2: Organize and structure your docs. Tag by topic, audience, urgency.

Week 3: Test RAG retrieval. Sample 50 questions and check if you get the right docs.

Week 4: Engineer your prompts. Define behavior, boundaries, and escalation triggers.

Week 5: Deploy with human-in-the-loop. Monitor conversations and refine.

What’s Next?

RAG + LLMs + Agentforce = the foundation for trustworthy agent automation.

But there’s more. Add agent governance, multi-agent orchestration, and continuous monitoring—and you have a production-grade system.

Ready to build reliable AI agents for your business?

Book a consultation →