Our logistics customer receives around three thousand WhatsApp messages and another five hundred emails each day. They contain freight demand in free text: a truck type, route, tonnage, and date expressed through abbreviations and multiple languages. The ops team was reading the messages and typing load requests into the CRM by hand.
At that volume, manual capture was slow and messages were missed. An uncaptured request could not be priced, assigned, or followed up, so the data loss had a direct commercial consequence.
An LLM extracts the fields from each message, but most of the engineering work went into moving those fields safely into the customer's system.
The pipeline we built
What we built is a queue-based ingestion pipeline. Messages come in from WhatsApp and Gmail, get queued, an LLM extracts the structured fields, the extracted entities get mapped against the customer's actual world (vehicle types, locations), and a CRM entry comes out the other end. No human in the typing loop.
Each stage earns its place:
- The queue is there because traffic is bursty and providers hiccup. Messages can't be dropped just because a downstream service had a bad minute. Ingestion and processing had to fail independently and retry independently.
- Extraction handles the messy part: free text in multiple languages, "32ft sxl needed nagpur to blr tmrw" and a hundred variations of it. This is where LLMs genuinely changed what's feasible. The old regex-and-rules approach dies a slow death against input like this.
- Mapping is the stage people underestimate. The model can tell you the message mentions a truck type and two cities. It cannot tell you which canonical vehicle code and which location records those map to in this customer's system. Extraction without resolution is trivia. Resolution is what makes a row a CRM can act on.
The prompt stabilized early. We spent more time on retries, deduplication, ordering, and deciding what to do when the same load request arrived through two channels.
The replay story matters more than the prompt story. If the model or mapper improves next month, you should be able to replay yesterday's raw messages through the new pipeline and see exactly which rows would change. Without that, every improvement is also a migration you cannot measure.
Demand lineage
One message earns its way into the CRM
Trace the article’s representative freight message through extraction, resolution and duplicate handling.
Scenario
Representative article message with invented IDs and source time. Toggle delivery and location conditions to change the route to the CRM.
Sequence
- Keep the sourceCurrent
- Queue deliveryUpcoming
- Extract candidate fieldsUpcoming
- Resolve canonical recordsUpcoming
- Check demand identityUpcoming
- Commit with lineageUpcoming
Keep the source
“32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
Read the full explanation
New delivery
Representative article message with invented IDs and source time. Toggle delivery and location conditions to change the route to the CRM.
- Keep the source. “32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
- Queue delivery. A durable queued event separates intake from extraction, so downstream processing can retry.
- Extract candidate fields. 32ft sxl → vehicle; nagpur → origin; blr → destination; 18t → tonnage; tmrw → 17 January using source time in IST. Replaying later keeps that date.
- Resolve canonical records. In this illustrative dictionary, 32ft sxl maps to vehicle-32-sxl and blr maps to loc-bengaluru. These are example IDs, not production records.
- Check demand identity. No existing provider-ID association is present in this example. Check demand identity before creating a row; similar text alone is not proof of a duplicate.
- Commit with lineage. Create one CRM row with canonical IDs, source pointer and derivation versions. The original message remains available for replay.
Duplicate delivery
Representative article message with invented IDs and source time. Toggle delivery and location conditions to change the route to the CRM.
- Keep the source. “32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
- Queue delivery. A redelivery carries the same source ID. Retrying delivery is not evidence of a second load request.
- Extract candidate fields. 32ft sxl → vehicle; nagpur → origin; blr → destination; 18t → tonnage; tmrw → 17 January using source time in IST. Replaying later keeps that date.
- Resolve canonical records. In this illustrative dictionary, 32ft sxl maps to vehicle-32-sxl and blr maps to loc-bengaluru. These are example IDs, not production records.
- Check demand identity. The provider ID already maps to example-demand-01. Preserve that association rather than creating another demand. If a new derivation is ambiguous, hold that proposed change without erasing the known link. A new message from another channel still needs a separate demand-identity check.
- Use the existing row. The duplicate creates zero new rows and retains its link to the same CRM demand. Retry-safe delivery does not imply automatic cross-channel deduplication.
Ambiguous location
Representative article message with invented IDs and source time. Toggle delivery and location conditions to change the route to the CRM.
- Keep the source. “32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
- Queue delivery. A durable queued event separates intake from extraction, so downstream processing can retry.
- Extract candidate fields. 32ft sxl → vehicle; nagpur → origin; blr → destination; 18t → tonnage; tmrw → 17 January using source time in IST. Replaying later keeps that date.
- Resolve canonical records. In this illustrative customer dictionary, blr could mean the city or a depot. Keep both candidates and leave destination ID unset.
- Check demand identity. No existing provider-ID association is present in this example. Check demand identity before creating a row; similar text alone is not proof of a duplicate.
- Hold for review. This new derivation stops before an automatic CRM write. A reviewer chooses the destination using the raw message and retained candidates. Duplicate delivery does not remove the ambiguity; any known CRM association survives, and its existing row is not overwritten.
Duplicate + ambiguous
Illustrative reprocessing of an already-linked source under an ambiguous dictionary. Preserve the existing CRM association while reviewing the new derivation.
- Keep the source. “32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
- Queue delivery. A redelivery carries the same source ID. Retrying delivery is not evidence of a second load request.
- Extract candidate fields. 32ft sxl → vehicle; nagpur → origin; blr → destination; 18t → tonnage; tmrw → 17 January using source time in IST. Replaying later keeps that date.
- Resolve canonical records. In this illustrative customer dictionary, blr could mean the city or a depot. Keep both candidates and leave destination ID unset.
- Check demand identity. The provider ID already maps to example-demand-01. Preserve that association rather than creating another demand. If a new derivation is ambiguous, hold that proposed change without erasing the known link. A new message from another channel still needs a separate demand-identity check.
- Hold for review. This new derivation stops before an automatic CRM write. A reviewer chooses the destination using the raw message and retained candidates. Duplicate delivery does not remove the ambiguity; any known CRM association survives, and its existing row is not overwritten.
New delivery
Representative article message with invented IDs and source time. Toggle delivery and location conditions to change the route to the CRM.
- Keep the source. “32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
- Queue delivery. A durable queued event separates intake from extraction, so downstream processing can retry.
- Extract candidate fields. 32ft sxl → vehicle; nagpur → origin; blr → destination; 18t → tonnage; tmrw → 17 January using source time in IST. Replaying later keeps that date.
- Resolve canonical records. In this illustrative dictionary, 32ft sxl maps to vehicle-32-sxl and blr maps to loc-bengaluru. These are example IDs, not production records.
- Check demand identity. No existing provider-ID association is present in this example. Check demand identity before creating a row; similar text alone is not proof of a duplicate.
- Commit with lineage. Create one CRM row with canonical IDs, source pointer and derivation versions. The original message remains available for replay.
Duplicate delivery
Representative article message with invented IDs and source time. Toggle delivery and location conditions to change the route to the CRM.
- Keep the source. “32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
- Queue delivery. A redelivery carries the same source ID. Retrying delivery is not evidence of a second load request.
- Extract candidate fields. 32ft sxl → vehicle; nagpur → origin; blr → destination; 18t → tonnage; tmrw → 17 January using source time in IST. Replaying later keeps that date.
- Resolve canonical records. In this illustrative dictionary, 32ft sxl maps to vehicle-32-sxl and blr maps to loc-bengaluru. These are example IDs, not production records.
- Check demand identity. The provider ID already maps to example-demand-01. Preserve that association rather than creating another demand. If a new derivation is ambiguous, hold that proposed change without erasing the known link. A new message from another channel still needs a separate demand-identity check.
- Use the existing row. The duplicate creates zero new rows and retains its link to the same CRM demand. Retry-safe delivery does not imply automatic cross-channel deduplication.
Ambiguous location
Representative article message with invented IDs and source time. Toggle delivery and location conditions to change the route to the CRM.
- Keep the source. “32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
- Queue delivery. A durable queued event separates intake from extraction, so downstream processing can retry.
- Extract candidate fields. 32ft sxl → vehicle; nagpur → origin; blr → destination; 18t → tonnage; tmrw → 17 January using source time in IST. Replaying later keeps that date.
- Resolve canonical records. In this illustrative customer dictionary, blr could mean the city or a depot. Keep both candidates and leave destination ID unset.
- Check demand identity. No existing provider-ID association is present in this example. Check demand identity before creating a row; similar text alone is not proof of a duplicate.
- Hold for review. This new derivation stops before an automatic CRM write. A reviewer chooses the destination using the raw message and retained candidates. Duplicate delivery does not remove the ambiguity; any known CRM association survives, and its existing row is not overwritten.
Duplicate + ambiguous
Illustrative reprocessing of an already-linked source under an ambiguous dictionary. Preserve the existing CRM association while reviewing the new derivation.
- Keep the source. “32ft sxl needed nagpur to blr tmrw 18t ready by evening call back urgent”. The illustrative source arrived 16 January 2026 at 16:00 IST. Keep its raw text and provider ID.
- Queue delivery. A redelivery carries the same source ID. Retrying delivery is not evidence of a second load request.
- Extract candidate fields. 32ft sxl → vehicle; nagpur → origin; blr → destination; 18t → tonnage; tmrw → 17 January using source time in IST. Replaying later keeps that date.
- Resolve canonical records. In this illustrative customer dictionary, blr could mean the city or a depot. Keep both candidates and leave destination ID unset.
- Check demand identity. The provider ID already maps to example-demand-01. Preserve that association rather than creating another demand. If a new derivation is ambiguous, hold that proposed change without erasing the known link. A new message from another channel still needs a separate demand-identity check.
- Hold for review. This new derivation stops before an automatic CRM write. A reviewer chooses the destination using the raw message and retained candidates. Duplicate delivery does not remove the ambiguity; any known CRM association survives, and its existing row is not overwritten.
Replay a burst without multiplying demand
Change delivery and recovery conditions. Follow six invented source events through every processing attempt and CRM outcome.
Columns = six source IDs. Each mark = a delivered copy. Select a mark to inspect its path. “Same row” is an idempotent no-op; “Queued” means the durable source is retained for recovery.
Inspect message 01: source time survives replay
“32ft sxl needed nagpur to blr tmrw 18t”
- Source arrival
- 16 Jan 2026, 16:00 IST
- Replay day
- 2026-01-23
- Derived pickup
- 2026-01-17. Always source + 1 day.
- Destination ID
- loc-bengaluru
- Provenance
- msg-01 → extractor-v1 → resolver-v1 → CRM outcome
Illustrative deterministic batch: six different demands, copied provider IDs, identical transient failures before any CRM write, durable intake, and one potentially ambiguous message. Exhausted copies remain queued; repeated reviews are delivery counts, not unique review tasks. Lost acknowledgements after a write need outcome reconciliation; this batch does not simulate them. Cross-channel business deduplication needs separate evidence. The article’s 97% is an observed flow average, not this simulation’s accuracy.
Trigger: missing vehicle type, ambiguous city, duplicate candidate, low extraction confidence, or a date that depends on replay time.
System action: keep the raw source message, write a pending record with provenance, and route only the uncertain fields for human correction.
Reviewer action: choose the canonical vehicle/location, mark duplicate-or-new, and leave the corrected field next to the original candidate.
Feedback: promote repeated corrections into alias dictionaries, eval cases, or validation rules. Do not let them disappear into the CRM.
Raw source: channel, sender, provider message ID, timestamp, original text, attachments, and ingestion version.
Derived row: extraction model, prompt version, resolver version, canonical IDs, confidence, and idempotency key.
Replay mode: dry-run a new extractor or resolver against old messages, diff the CRM payload, and sample the changed rows before writing.
Rollback: keep enough provenance to explain why a CRM row exists and which source event produced it.
What it does now
The flow runs live for the business at about 97% average accuracy on the demand it processes. That's an average across the flow, not a guarantee on every field. At roughly 3,000 WhatsApp messages and 500 emails a day, the math is simple: structured demand capture happens at a scale and speed no ops team could match by hand, and the misses are the exception to manage rather than the default state of the system.
Operational volume observed in the live ingestion channels.
Average across the demand-processing flow; not a guarantee on every extracted field.
Operational judgment from the business workflow: uncaptured demand cannot be priced, assigned, or followed up.
- Turning WhatsApp Into Your Document PipelineA concrete WhatsApp-to-SQLite-to-classification pipeline with schema and automation details.
- Mining WhatsApp, WeChat, Alibaba and GmailA field report on collecting demand from WhatsApp, WeChat, Alibaba, and Gmail before the data reaches an AI workflow.
- Shipping AI WhatsApp Automation to ProductionCovers WhatsApp webhooks, queues, budget caps, confidence checks, and human escalation in a production deployment.
- WhatsApp Business Platform webhooksOfficial reference for treating incoming WhatsApp messages as webhook events with JSON payloads and source metadata.
At the current volume, the model call is only one possible failure point. A better extractor cannot recover a dropped webhook, distinguish duplicate messages, or map an unknown city to the customer's canonical records. Those are ordinary data-pipeline problems, and we monitor them as such.
The LLM made the message text tractable. The queue, resolver, idempotency key, and review lane made its output safe enough to enter the CRM. That division of responsibility is the part of this system I would reuse.