A scheduled run processed roughly 2,000 airline support conversations in a few hours. Once processing finished, leadership could ask questions in plain language and receive answers in seconds without rerunning the analysis. That separation between evaluation and retrieval was the central architectural decision. In this customer journey intelligence project, we were working with chat logs, voice recordings, routing events, conversation attributes, and agent interaction logs. The data covered the same customer journeys, but it lived in different systems and did not share a reliable schema. Normalize the journey before evaluating it The model could not evaluate a conversation until the system knew which records belonged together. Each environment used a different name for the shared conversation identifier. Timestamps did not always align, and voice and chat records arrived at different levels of detail. We built a normalization layer that mapped each environment into one journey model, then ordered every message, transfer, hold, and resolution chronologically. This work was programmatic. The LLM received a coherent customer journey rather than being asked to infer relationships between disconnected records. That distinction matters. A capable model can still produce a confident assessment of incomplete evidence. Separate analysis from querying The platform has 2 execution paths. The scheduled path joins the source data, transcribes voice with Amazon Transcribe, removes personally identifiable information, and sends the reconstructed journey to Claude Sonnet for evaluation. The results are then stored for later use. The interactive path reads those stored results. The dashboard, data explorer, and conversational agent can search and synthesize the analysis, but they do not rescore the original conversations. This design fits reporting, incident investigation, and trend analysis, where per-second processing is unnecessary. Batch execution also makes compute usage and LLM costs easier to predict. A team can scope a run to a 50-conversation incident review or a 2,000-conversation monthly analysis. Real-time processing would introduce more infrastructure and cost without improving these workflows. Keep evaluation criteria outside the model The system evaluates intent, sentiment, urgency, resolution, compliance, and specialist performance using the client's existing quality rubrics. We considered the rubrics part of the application configuration. When the quality team changes a criterion, it can update the prompt instead of collecting new training data and fine-tuning another model. Claude Sonnet was selected after smaller models lost accuracy on nuanced criteria, including sentiment across multi-turn conversations and whether an escalation was appropriate. The cheaper model was not cheaper if analysts could not trust its judgments. This approach keeps the evaluation process adjustable, but it also requires prompt governance. Rubric changes should be versioned and tested because they change the meaning of every score produced afterward. Remove PII before inference Redaction runs after journey assembly and before the model receives any content. The redaction step is deterministic, applies to every conversation, and consumes no model tokens. Sensitive information does not enter the prompt or appear in stored evaluation results. Post-processing model output would have been too late. By that point, the sensitive data would already have crossed the inference boundary. Calibrate against human decisions An LLM evaluator needs a reference set. We compared its output with conversations the client's analysts had already scored and refined the rubrics where the two diverged. Exact agreement reached about 85%. Agreement exceeded 90% when scores within 1 point on the 1-to-10 scales counted as matches. Both measurements are useful. Exact agreement reveals consistency, while the tolerance measurement shows whether disagreements are materially different or simply adjacent values on a subjective scale. The acceptable tolerance should be defined before evaluating the model, not chosen afterward to improve the result. What to carry into your own system For similar analysis platforms: Reconstruct the complete domain object before sending it to an LLM. Use deterministic code for joins, ordering, validation, and redaction. Separate expensive evaluation from latency-sensitive retrieval. Keep changing business criteria in versioned rubrics. Test model quality against examples scored by domain experts. Choose batch or real-time execution from the user workflow, not from the novelty of the architecture. The conversational agent is the visible part of the product. Its reliability comes from the normalized, redacted, calibrated evidence underneath it.