Building software fast without sacrificing quality is difficult. Organizations that wish to develop and adapt technology face the “iron triangle” of balancing the trade-offs between building something good, fast, or inexpensive.

As consultants, we turn to Retool for our clients because of its powerful combination of speed and flexibility. We can use Retool’s baseline components and customize how we process and display data, which gives us the ability to “break” the iron triangle for our clients. 

In other words, with Retool, we can move fast and build quality software with limited resources. What Retool offers us is the flexibility to fit into each organization’s unique workflow.

Prototyped SaaS Applications

As one example, we recently helped a client build and launch a small SaaS app within a month. The client had an idea in the legal-tech domain and a potential customer for a demo with a tight deadline. They needed their application to function as intended and to show the potential of the idea.

We were brought in to build fast and stand up a working prototype. Retool accelerated our development by giving us the toolbox to pull together a fully-functional system. We built it from the ground up in only a few weeks, helping them hit their deadline and demo for their customer without any hand-waving. It was a huge win for all parties, and we are currently expanding to build the full consumer-facing application entirely within Retool, from interface to database.

This example has shown us that Retool can be used to prototype and power revenue-generating applications.

Supercharged Internal Operations

We’ve also used Retool for internal-facing applications. Another example is from our client flyExclusive, a private aviation company. Before using Retool, they were manually handling multiple internal workflows, from pilot flight reporting to maintenance records.

For workflows that require data transparency and customized experiences, like verifying flight logs, we’ve used Retool to create highly-specific data management interfaces. These interfaces give the right users access to the data they need in real time. And managers can edit certain fields, which updates an external database as the source of truth.

With Retool, an added bonus is that we can update the workflow and interface very quickly. In a fast-paced business environment where teams are continuously iterating and improving their operating procedures, this means that we can not only build quickly, but we can modify the application quickly too.

For example, when this client changed their procedure for managing logs, we were able to update their workflow in a matter of days instead of spending much longer developing and deploying the update. The result is that the business can evolve and the technology can keep up.

What We’ve Learned

There is certainly still a need for traditional software development, when it comes to scaling an application to support a massive user base and processing large amounts of data quickly. We haven’t hit the ceiling on either of these factors with Retool. 

Instead, we’ve found that we are able to build 90% of the necessary features with out-of-the-box Retool components and workflows. Additionally, our ability to quickly and flexibly fit into each organization’s workflow helps us delight our clients who are used to working with expensive software teams and delayed launches.

Dedicating resources to new and exploratory projects is rarely possible without a proof-of-concept. What Retool allows us to do is to help our clients achieve their goals on tight timelines and with limited budgets. This is a differentiator for our clients who need to demonstrate a return on investment before they can request additional support.

What’s Next?

While we use Retool to get our clients to outcomes fast, we don’t see it simply as a proof-of-concept system. Retool has become a key part of our toolbox because we believe that Retool can power fully-functional user applications without sacrificing quality.

We’ve found Retool to be a great data platform for internal company metrics and automation workflows. The out-of-the-box database and flexible components are the ideal foundation for us to bring disparate bits of data together and then let the end users interact with them. And after implementation, these applications are easy to maintain, update, and expand.

We are excited to be Retool Agency Partners so that we can continue to help our clients organize their data and put it to work.

We’ve done multiple projects to automatically pull structured data from text, including:

• Automatically parsing emails
• Scanning research articles
• Processing customer feedback reports

This is called natural language processing, where we use computers to process information contained in unstructured text. This is increasingly done with large language models (LLMs) because they provide amazing flexibility for processing natural language with intuitive inputs.

LLMs are next-token generators. Acting kind of like a glorified autocomplete, they generate text that is most likely to come next based on what was given as an input. To generate structured text, we need to guide the model for what output we want.

Here I'm going to show you how we do that with some basic prompts.

Structured Formats

There are a few common formats for structuring data: JSON, YAML, and XML. These are all designed so that they can be used by a computer program—whether that is sticking data into a spreadsheet or running a database operation or code function.

Luckily, most foundation LLMs have been trained on many examples of structured data, so they have an understanding of these data formats out of the box. This means we can usually get pretty good results with zero- or few-shot learning (i.e., no need for a custom-trained model).

Let’s take a look at an example and see how this works. For all of these examples, I’m using Mistral 7B Instruct (which we could hook up for you through Amazon Bedrock!), but it works similarly in other models as well. Here is the input for all of these examples, where we just substituted {format} with JSON, YAML, or XML.

Extract the name, email address, and sentiment (positive, neutral, or negative) from this EMAIL. 

Return the OUTPUT as formatted {format} with keys for sentiment and contact. 
Contact should have subkeys for name and email. 
Wrap it all in a result key.

EMAIL:
From: aaron@stradiapartners.com
Subject: Help
Body:
Hi,
I'm really happy with my product, but I need some help.
can you write me back?
Aaron

OUTPUT:

JSON – nests key-value pairs in brackets

Pros:

• Handles numbers and text differently, simplifying post-processing
• Strong support in the newer OpenAI models, with specific JSON training and built-in output validation

Cons:

• Brackets and quotations add extra tokens, increasing cost and decreasing efficiency

Output (57 tokens):

{
  "result": {
    "contact": {
      "name": "Aaron",
      "email": "aaron@stradiapartners.com"
    },
    "sentiment": "positive"
  }
}

YAML – uses indents to identify hierarchical structure

Pros:

• Easily human readable
• Without extra “fluff” it is more efficient and cheaper than other formats

Cons:

• Easy to mess up the indents, potentially increasing errors (although I haven’t quantified that)

Output (30 tokens):

result:
  contact:
    email: aaron@stradiapartners.com
    name: Aaron
  sentiment: positive

XML – wraps data in tags

Pros:

• Explicit wrappers ensure “extra” text that a model might return are ignored (sometimes models, especially “chatty” models, like to add extra context before or after the structured data like “Here is your data: … Do you need anything else?”)

Cons: 

• Opening and closing tags make it extremely verbose, increasing cost and decreasing efficiency

Output (57 tokens):

<result>
  <contact>
    <name>Aaron</name>
    <email>aaron@stradiapartners.com</email>
  </contact>
  <sentiment>positive</sentiment>
</result>

Challenges

As stochastic parrots, LLMs are not without issues.

For structured data, the first challenge is what happens if the model returns an incomplete format? In the JSON example, if the model finishes before adding the closing } bracket, then the computer won't be able to properly parse it. This can sometimes be fixed by doing code checks after running it and then rerunning if there is an error. Some models also add explicit generation checks that require valid formatted output.

The second challenge is hallucination of data. Sometimes hallucination can be fixed by providing more specifications in the input (in the sentiment example, if we don't give a specific list of the options then the model might get creative with the output) or by lowering the randomness temperature.

Other times, unclear inputs might confuse the output (e.g., if there are multiple names in the email, it could inadvertently grab the wrong one), or just simply interpolate results that aren't there (maybe based on content that the model had seen in its training set).

There are a few tricks for avoiding these shortcomings.

Adding Extra Guardrails

Specifying the Schema

This can help ensure that the output is formatted correctly. And OpenAPI specifications provide a clear way to do this. In fact, OpenAI models have the option to input a specific JSON-formatted schema in the OpenAPI format.

From the OpenAPI documentation, here is an example showing how a model specification (in this case in YAML format! The JSON format is also common) can represent data in JSON or XML format:

Model specification

components:
 schemas:
   book:
     type: object
     properties:
       id:
         type: integer
       title:
         type: string
       author:
         type: string

JSON data

{
 "id": 0,
 "title": "string",
 "author": "string"
}

XML data

<book>
 <id>0</id>
 <title>string</title>
 <author>string</author>
</book>

Including a Few Examples

This can provide additional guidance to the model through a "worked" sample. To do this, we might include a sample EMAIL and OUTPUT in the desired format before the actual EMAIL that we need to process.

Fine-Tuning

Finally, for high-risk or mission-critical cases where it is important to get it right, fine-tuning the model with lots of training samples can really lock in the output for what you need. We've found fine-tuning to increase accuracy from the low-70%s to over 90%.

As for which format to choose, this decision depends on the data you need to process and the model you plan to use. Generally, we like JSON for OpenAI models and XML for other foundation models. We find that the extra token costs and processing time are rarely a concern, and these formats allow for more explicit definition.

This kind of project is exciting for us because it allows us to tailor the output to your specific needs, which lets us dig into what you need to get out of it. We also find that this kind of project can save a lot of time for people, for example speeding up how quickly a team can work through hundreds or thousands of messages, and sometimes even increase "objectivity" if multiple people are working on the same task.

We'd love to hear how you think this can be helpful for you! Email us to talk.

A disclaimer, this post is not about best practices or even prescribing a solution. This article will outline a unique challenge and the approach we took to get past the hurdle. So let's talk about what we faced! Over the last 3 years, we helped a client transition a lot of their internal tooling over to a tool called Airtable. So you may be wondering why Airtable, and the answer is simple: speed. I mean, let's be honest, many of us have been there, a room with executives that wanted something yesterday, and your team is tasked with building a time machine.

The Challenge

Now that you understand the motivation, let's fast forward. We are fully on board with Airtable; we have architected a core set of environments that meet the needs of specific departments, and folks are off to the races! Well, not quite. The peak hours of this client are between 2 p.m. and 5 p.m., and coincidentally, that's also when we started to notice that high-traffic environments are starting to degrade. For some context, Airtable gives you the ability to define automated workflows using a mechanism in their software called automations. The automations give the user a really easy interface with which they can define the workflow along with the conditions that will trigger it. Once those conditions are met, the automation is executed, and the user-defined workflow is run. There are some limitations to be aware of, however:

• An Airtable base is limited by the number of automations you can create.
• A single automation instance has a runtime limit of 30 seconds.
• A single automation instance can make a set number of HTTP requests.
• The concurrency of these automations is handled by Airtable behind the scenes.
• An Airtable account is limited by the number of total runs a workspace can execute.
• Automation compute shares resources with the rest of the operations occurring in a base. e.g., user actions, API calls, etc.

What We Did About It

The first lever we made use of was working directly with Airtable on possible performance enhancements we could make. We did end up getting a higher compute instance provisioned for high traffic bases. We knew that the larger instance would not be enough; we needed to be able to scale past the amount of user interaction we had now, so what was left? The key is identifying that automations share compute resources with the rest of the base, so how could we offload the CPU time? Here is what we did.

• We asked Airtable for the CPU and memory metrics of all the automations in our high traffic instances.
• We used that list to identify candidates for "promotion" out of Airtable.
• We leveraged Airtable's webhooks to architect an "automation" solution from our end.
• In the event that the output of an automation had to interact with a base, we used Airtable's Rest API to push those results back.

Here Is What It Looked Like

We moved the compute over to a serverless function using AWS. This unlocks a few things for us:

• We moved valuable CPU load out of high-traffic Airtable bases.
• We gained some control over the concurrency of automation runs.
• A longer runtime for complex operations.
• Access to our own metrics over the lifecycle of any single run.

We saw immediate results, and our users also noticed. Hurray! Of course, while this approach did solve our immediate problems, it added a whole new level of complexity to managing our Airtable instance, so we carefully decided what should be "promoted.".

We learned a lot during this excursion, and we could always improve the solution. It shows that clever solutions exist; you just need a little know-how, and the tool you use needs to allow some room for extension. We love finding interesting solutions to tough problems, so if you are interested in partnering with us, reach out and let's chat.

Generative AI is touted as a tool for increasing revenue and reducing expenses. It is easy to get lost in the hype about the future and miss the amazing operational workflows that can be improved today.

For over a year and a half with one of our clients we’ve been reading over 25,000 inbound sales emails a month, extracting the relevant data with a fine-tuned OpenAI GPT model, and adding it to an Airtable base for their sales team to action.

The challenge was that the sales team was drowning in emails that they were managing out of a shared email inbox. Emails went unanswered and there wasn’t clarity on how many requests were actually received, so it was difficult to measure conversion.

Now emails can go from receipt to quoted in a matter of minutes. Salespeople can review and send their quotes with the click of a button in Airtable. Time that used to be spent on manual data entry (read the email, identify the request, check other similar requests, create the quote) can now be spent actually calling the customer and closing the deal.

How We Did It

We went from zero to launch in less than two months, increasing the quote volume >10x in a matter of weeks.

We did this by:

1. Getting alignment across all teams,
2. Starting with a thin vertical slice, and
3. Improving and adjusting incrementally.

The project was spearheaded by an executive team that was excited by the opportunity of technology. They worked to ensure that all teams were on board, which is important for this type of project because of the broad impacts on the company processes and people.

For the technology, our first tests used OpenAI’s text-davinci-003 with a specified output format and only a few prompt examples. This didn’t work very well (maybe 50-50 for this application, but note: this type of few-shot learning can work really well in other applications), so we quickly pivoted to fine-tuning. We collected around 100 examples of emails that we manually parsed and used to train the model. Then we launched the system for the salespeople with a limited set of customers (YOLO, as they say).

After launching, we learned a lot really quickly. At first, some of the salespeople were concerned about their jobs being taken by robots. Our internal detractors were quick to point out all of the errors. Our internal early adopters were also able to start increasing their output. So we started the next phase of improving the system.

With another testing set, we learned that the accuracy of the model was around 70%. Not bad, but not really good enough for unsupervised usage. So we retrained the model with hundreds of new examples, focusing on some of the edge error cases that the sales team identified. 

By this time, OpenAI had released upgrades to their model with gpt-3.5-turbo, which helped with accuracy, and also released Function Calling, which improved the formatting issues. With these updates and our expanded new training set, we were able to get out-of-sample accuracy up to 96%. Pretty impressive!

The system has certainly needed maintenance and additional updates since then as the business requirements have evolved. But, the core functions have been humming along in the background, churning out structured data and empowering the client to increase sales.

We’re interested in helping other clients leverage AI in useful ways, whether it is reading unstructured data or something else entirely different. Reach out to talk to us!

Let's imagine you're opening a restaurant. You dream of being the hardest-to-snag dinner reservation on the block.

A friend calls and says, "You have to hire Alfredo as Head Chef. His cooking is sublime."

Would you: A) Meet Alfredo, get to know him, look at his resume, and taste his famous pasta or B) Hire Alfredo as your Head Chef right away?

The success of your restaurant hinges upon this decision! So obviously you'd go with option A. But throughout my career in software development, I've seen leaders "hire Alfredo" sight unseen again and again.

Let me explain.

In a previous role, an executive came to me with an innovative idea to increase key KPIs with new technology workflows. He was ready to dedicate 3 months of our product roadmap to making the dream a reality.

I considered one week of my team's time to cost about $20,000, and that's before considering the opportunity cost of having them focus on this project instead of other valuable known-quantity projects. This project would cost a minimum of $120,000– a huge investment for an unknown ROI.

I suggested that before jumping into the project, we should try to test some of the riskiest assumptions with no-code processes. The cheapest, fastest way to learn whether our features would work was to change our services workflows to adopt them without the supporting workflow technology.

So how would we test? Someone needed to train our Services team to offer alternatives when an original customer request couldn't be accommodated. We then needed to measure how often the client took the alternative, and thus captured additional revenue. Conversations could be tracked and recorded in Airtable. The cost of this approach was a bit of Airtable configuration and training time. The outcome would be evidence our development time would be worth it.

So what happened? He didn't want to initiate the test or wait for the results. He wanted to "hire Alfredo"– a $120,000 bet.

We started Stradia Partners to help people approach problems like this differently– even if it means less money for us.

Our process would look something like this:

1. Start by calculating the size of the bet you're making with development. What will it cost to build in time, money, and opportunity?
   1. For my example, the cost is $120,000 of development time + opportunity cost of not working on previously validated roadmap projects.
2. Identify the things that need to be true for the bet to pay off (risks). List them.
   1. In my example, the bet would pay off if customers were willing to take alternative options and the revenue gained from those sales was more than the cost of development, maintenance, training and adoption.
3. See if you can de-risk the bet prior to making it. This is known as a Riskiest Assumption Test (RAT).
   1. If customers will take the alternative over the phone or email, then it's likely they'd take it in an app (bet = validated).
   2. If customers don't take the alternative over the phone or email, then it's even less likely they'd take it in the app (bet = invalidated).
4. Run the test (taste the food); measure.
5. Use the test results to evaluate whether there's a case for the bet to pay for the cost calculated in step 1. If so, make the bet (hire Alfredo).

Want help tasting the food and deciding whether to hire Alfredo? We're in your corner; shoot us a note.

Bon Appétit!

🟧
🟥 🟧

Garbage In, Garbage Out

You know the saying.

When you use data in your organization—for anything from reporting, analytics, automation, etc.⁠—you run the risk of building a giant trash pile. This isn’t your fault. Sometimes it just happens as the sources of data grow and as the business requirements change.

But, we’ve seen the consequences, and they can be severe:

• Employees performing duplicate data entry
• Executives arguing over the provenance of the data
• Data products that don’t deliver on the AI hype

To get past these barriers, it is important to recognize that your data ecosystem will evolve as your organization’s needs change. Then, you can focus on laying a solid foundation for your current (or next) use case.

🟨
🟧 🟨
🟥 🟧 🟨

An Example

I’ll explain our framework below, but first a small example.

One of our clients recently wanted to record expenses for events. Ultimately they were interested in seeing how close the cost of each event was to the initial budget. The existing solution had expense records with an estimated and actual cost. This makes sense data-wise, but the challenge was that entering the actual cost for an expense required finding the initial estimate record, which was difficult. Instead, we set them up with a single data entry source where each expense was either an estimate or an actual.

We like to be pragmatic about data structures. This isn’t a perfect solution and there are definitely other ways to do it. You might argue that this design is insufficient because it doesn’t allow the client to drill down and see which items caused the event to go over or under. But, it is also important to remember that linking the expenses to their estimates would be difficult for the employees to maintain and extremely uncertain with an automated solution. Likely to lead to a data garbage pile.

Instead, our goal is to set up a foundation so that clients can build and iterate. Now the client is able to easily enter expenses and calculate overall accuracy. Additionally, they are set up to automate expense entry in the future with AI by collecting data from receipts without worrying about matching specific expense records.

🟦
🟩 🟦
🟨 🟩 🟦
🟧 🟨 🟩 🟦
🟥 🟧 🟨 🟩 🟦

The Data Product Pyramid

Years ago I wrote about the data product pyramid, which is a framework for understanding how data products are built. The idea is that “higher-level” outcomes—stuff like workflow automation and decision support—require a solid base layer of robust data without major gaps, inconsistent sources, and data entry errors.

When we work with clients, we emphasize the importance of continuing to check the base and expanding incrementally. Raw data are at the base because they are the least refined. Each successive level builds off of the previous in a data hierarchy of needs. This stable foundation is even more pressing as organizations try to embed AI into their processes, as these tools rely on clean and consistent inputs to produce value.

Our approach is not one-size-fits-all. Avoiding (or fixing) a data dumpster fire involves understanding where each client is headed and what they have as their current foundation. We use the data product pyramid to guide our discovery. From there, we can work together to build a great product or tool, one layer at a time.

🟪
🟦 🟪
🟩 🟦 🟪
🟨 🟩 🟦 🟪
🟧 🟨 🟩 🟦 🟪
🟥 🟧 🟨 🟩 🟦 🟪