LLM prompt engineering · safety-first
LLM prompt engineering strategy
6 prompt layers diferenciados · 6 prompt injection defenses · 6 evaluation metrics con objetivo definido · 6 evolution governance rules. Prompts diseñados para safety médica + brand consistency + cost control · NO ad-hoc experimentation production.
6 prompt layers · stacked architecture
| Layer | Purpose | Content |
|---|---|---|
| System prompt (immutable per tenant) | Role definition · clinic identity · brand voice · scope constraints · NEVER user-editable | ~800 tokens · clinic name + style guide + boundaries (NO medical diagnosis · NO prices not in KB · NO insurance coverage) |
| Knowledge base context (per query) | Retrieval-augmented · clinic-specific FAQ + services + policies · grounds responses in clinic facts | RAG retrieval ~1500 tokens relevant chunks · prevents hallucination · cited responses |
| Conversation history (sliding window) | Multi-turn coherence · last 6-8 messages · trimmed if too long · summarized old context | ~500-2000 tokens dynamic · oldest dropped first · summary preserved |
| User input (sanitized) | Current message paciente · pre-filtered (prompt injection detection · jailbreak attempts blocked) | Typically 10-200 tokens · max 1000 enforced · longer = split or reject |
| Output schema enforcement (Zod) | Structured response · validated post-LLM · prevents free-form harmful content slipping | JSON schema: response_text · confidence · escalate_to_human · scheduling_intent · feedback_request |
| Safety guardrails (output filter) | Post-generation review · medical diagnosis detection · forbidden claims · auto-correct or escalate | Pattern matching + secondary LLM evaluation · borderline cases human-in-loop fallback |
6 prompt injection defenses
Input sanitization
Strip system-like prompts ('Ignore previous instructions...') · neutralize markdown injection · escape special tokens · max length enforcement
Role separation strict
System role NEVER concatenated with user input · API enforces distinct messages · prevents context confusion
Output validation post-generation
LLM cannot bypass schema · Zod validation rejects free-form · forces structured response · simpler to audit
Forbidden topic detection
Pre-LLM classifier flags: medical diagnosis · pricing absolute · insurance claims · prescription · contraindications · escalates human
Rate limiting per-tenant + per-conversation
Prevent flooding attack · cost protection · max 50 messages/conversation auto-escalates · max 1000/clinic/hour throttle
Conversation handoff triggers
Detect frustration · jailbreak attempts · complex requests · auto-escalate clinic admin · documented in handoff-policy
Evaluation framework · 6 metrics
| Metric | Target | Estado real (sin tráfico de clientes) |
|---|---|---|
| Response quality (LLM judge + human) | >85% acceptable · measured weekly sample 50 conversations · LLM judge correlates with human rating ±10% | sin tráfico de clientes · no hay muestra semanal de 50 conversaciones que juzgar |
| Handoff rate (when bot escalates) | 15-25% target healthy · too low = bot overreaching · too high = bot underperforming | sin tráfico de clientes · no medido |
| Hallucination rate (factual errors) | <2% target · measured manual review weekly · RAG grounding helps | sin tráfico de clientes · no medido · la revisión manual semanal empieza con el primer cliente |
| Prompt injection success rate | 0% objetivo · cobertura adversarial en cada merge, no semanal · un objetivo no es un resultado | suite adversarial automatizada en CI (OWASP LLM Top-10 + red-team agéntico) · sin red team humano sobre tráfico real |
| Coste LLM por respuesta | presupuesto <0,005 € con gpt-4.1 | ~0,00258 € CALCULADO con 600+200 tokens y la tarifa real · sin tráfico de clientes no hay nada medido |
| Latency p95 generation | <5s p95 target · user experience constraint | sin tráfico de clientes · no medido · la latencia de generación no se instrumenta todavía |
Evolution governance · 6 rules
- Prompt changes require ADR si pattern change · documented WHY + before/after evaluation results
- A/B testing prompts via feature flag · 10% traffic new prompt · 7 días minimum · statistical significance before rollout
- Per-tenant overrides ONLY via approved patterns · clinic-specific KB updates allowed · system prompt structure locked
- Evaluation snapshots versionados · prompt version + eval results stored · rollback capability si regression
- Weekly review founder · prompt changes proposed + evidence + decision documented
- External adversarial review · ChatGPT auditor reviews prompts trimestral · attempts jailbreak · postmortem any successful
Honest limitations · todavía no hay nada medido
Las cifras de la tabla son objetivos, no resultados: sin tráfico de clientes no hay muestra de la que salgan. La columna «Estado real» dice, métrica a métrica, qué falta para poder medirla. Cuando haya tráfico se publicará cada cifra con su método y su fecha.
Compromiso: cuando existan mediciones se publicarán con su método y su fecha · NO cherry-pick de meses buenos · mejoras y regresiones por igual.
¿Tu AI/ML team necesita prompt architecture deep-dive?
Para Enterprise procurement · sample prompts · evaluation harness · red team scenarios reports disponibles bajo NDA Enterprise.