You don't need AI for most of it
We built a tool that checks hundreds of hotel and flight bookings against an attendee list. Most of it isn't AI, and the bits that are needed rules around them. Here's what I learned.
A hotel sends the event organiser a rooming list. It runs to fifty pages, a few hundred names, and somebody has to check every one of them against the list of people who are actually coming, before the first guest turns up at reception and finds there’s no room. That used to be a person with a highlighter and most of a day.
Oliver and I built CrossChex for TRIBE to do that job, and the honest version of how it works is more useful than the pitch, because the pitch says “AI” and the code mostly doesn’t.
The first version sent everything to the model
That’s what everyone does and it’s what we did. Upload the PDF, hand it to Claude, ask for the bookings back as a list. For a demo it’s magic. In the first week the problems arrived in this order: the OCR service skipped pages on long documents, so we switched to Amazon’s; the model gave us example dates that were wrong, so we stopped trusting any date it produced; and on one layout it returned more bookings than were on the page, because it had worked out roughly how many we expected and filled in the gap.
That last one is the thing people call hallucination, and I think the word lets us off the hook. The model wasn’t malfunctioning. We’d asked it a question with one boring, exact answer - how many rows are in this table - and that’s the kind of question models are worst at. They’re brilliant at “what is this?” and unreliable at “how many, precisely?” We had it the wrong way round.
What the model is actually for
So the design flipped. The model now reads the document once, early, and answers one small question: what shape is this? For a spreadsheet it reads ten rows and tells us which column is the surname and which is the check-in date. For a hotel PDF it tells us the date format and what the hotel calls the check-in column. Ten tokens of answer, roughly.
Then ordinary code does every row. It doesn’t guess, it doesn’t get tired on page forty, and when it hits a row it can’t read it says so rather than inventing one. Each hotel group formats its list differently, so there’s a small rule for each: this one puts two guests in one cell with an asterisk, this one lists the sharer’s name inside the confirmation number. Dull, but dull is the point. The regression tests require those rows to match the known-good answer at 95 percent, and they do.
Honestly, we tried to undo this. Midway through last year the models had improved enough that we built a version that let one do the hotel documents end to end, and switched it on. It lasted a week. The rules went back in and they’ve stayed.
Where the model earns its keep
One hotel group’s rooming list defeated the rules. The layout changes from page to page and the columns don’t line up, and after a few attempts at parsing it we gave that one document type to the model and let it read the whole thing. That’s the right use: the shape has no shape, so you need an interpreter.
Even there, we don’t let it work unsupervised. The prompt tells it, in capitals, that ten real bookings are better than thirty with made-up ones. If it comes back with fewer than five, a plain old pattern-matcher goes over the raw text and tries again. The model is the smart reader for the odd document, and a rule stands behind it in case it gets creative.
Airline bookings went the same way for a different reason. Every airline formats a confirmation differently and there are too many to write rules for, so the whole PDF goes to the model. The cost shows up in the tests: airline fields only have to match at 80 percent, and ticket numbers at 85, because the model misreads long strings of digits. The first model we used also quietly stopped halfway through long bookings and had to be told, in the prompt, “do not stop early”. Those gaps are the price of using a model where a rule would have done, and sometimes it’s a price worth paying. But you should know you’re paying it.
The other cost is the actual cost. Every call to a model is logged now with its tokens and what it cost in US dollars. The rules cost nothing and run in milliseconds. If every document went through the model, the bill would be a line item and the processing would take minutes instead of seconds.
What this means if you’re buying, not building
When someone tells you “we’ll use AI for that”, ask one question: does the thing have a shape?
If it does - invoices, bookings, timesheets, anything that arrives in a format - you want ordinary code doing the work and a model doing the small job of working out the format. Cheap, fast, and when it’s wrong it tells you.
If it doesn’t - a customer email, a contract clause, a page that looks like nothing you’ve seen - you want the model, with a rule standing behind it to check the answer makes sense.
Most of what runs a business has a shape. That’s the bit nobody puts in the demo, because “we wrote a parser” doesn’t sell. But it’s the bit that still works at three in the morning on page forty.
Working out which of your processes has a shape and which doesn’t is most of what an AI workflow audit is - A$2,950 plus GST, fixed, and you keep the plan whatever you do next.