How We Read 25k Emails a Month to Drive Revenue with AI
Generative AI is touted as a tool for increasing revenue and reducing expenses. It is easy to get lost in the hype about the future and miss the amazing operational workflows that can be improved today.
For over a year and a half with one of our clients we’ve been reading over 25,000 inbound sales emails a month, extracting the relevant data with a fine-tuned OpenAI GPT model, and adding it to an Airtable base for their sales team to action.
The challenge was that the sales team was drowning in emails that they were managing out of a shared email inbox. Emails went unanswered and there wasn’t clarity on how many requests were actually received, so it was difficult to measure conversion.
Now emails can go from receipt to quoted in a matter of minutes. Salespeople can review and send their quotes with the click of a button in Airtable. Time that used to be spent on manual data entry (read the email, identify the request, check other similar requests, create the quote) can now be spent actually calling the customer and closing the deal.
How We Did It
We went from zero to launch in less than two months, increasing the quote volume >10x in a matter of weeks.
We did this by:
- Getting alignment across all teams,
- Starting with a thin vertical slice, and
- Improving and adjusting incrementally.
The project was spearheaded by an executive team that was excited by the opportunity of technology. They worked to ensure that all teams were on board, which is important for this type of project because of the broad impacts on the company processes and people.
For the technology, our first tests used OpenAI’s text-davinci-003 with a specified output format and only a few prompt examples. This didn’t work very well (maybe 50-50 for this application, but note: this type of few-shot learning can work really well in other applications), so we quickly pivoted to fine-tuning. We collected around 100 examples of emails that we manually parsed and used to train the model. Then we launched the system for the salespeople with a limited set of customers (YOLO, as they say).
After launching, we learned a lot really quickly. At first, some of the salespeople were concerned about their jobs being taken by robots. Our internal detractors were quick to point out all of the errors. Our internal early adopters were also able to start increasing their output. So we started the next phase of improving the system.
With another testing set, we learned that the accuracy of the model was around 70%. Not bad, but not really good enough for unsupervised usage. So we retrained the model with hundreds of new examples, focusing on some of the edge error cases that the sales team identified.
By this time, OpenAI had released upgrades to their model with gpt-3.5-turbo, which helped with accuracy, and also released Function Calling, which improved the formatting issues. With these updates and our expanded new training set, we were able to get out-of-sample accuracy up to 96%. Pretty impressive!
The system has certainly needed maintenance and additional updates since then as the business requirements have evolved. But, the core functions have been humming along in the background, churning out structured data and empowering the client to increase sales.
We’re interested in helping other clients leverage AI in useful ways, whether it is reading unstructured data or something else entirely different. Reach out to talk to us!
