Hey there! We're Riza and we make it safe and easy to run untrusted code generated by LLMs or humans, in our cloud or on your infra.

There's a pattern emerging across Riza's early customers. They're using LLMs to build highly-adaptable software systems with a technique that we've started to call "just-in-time programming."

What’s Just-in-Time Programming?

Here’s the pattern: take some dynamic input at runtime, show it to an LLM, and ask for runnable code to manipulate that input. Then take the code from the LLM and run it immediately, in production. No code review. No tests. Just run it and handle the result. The LLM writes code "just-in-time" to process the input.

Example: ingesting varied CSV data

Here's a concrete example. Let's say you're building an application that helps users tabulate their gains and losses from stock trades.

The app accepts individual stock trade data submitted by users, but the app only needs the ticker symbol, the transaction date, the number of shares traded and the share price for each trade. You expect users to provide a CSV export from their brokerage platform that includes this information, but you won't know the exact CSV format until it hits your system, and you don't want to make users manipulate the CSV before submission.

Without an LLM, you'd have to build a multi-step workflow that takes the CSV and presents a field mapping interface to users, along with a format selector for the date and price fields.

With an LLM, you can dispense with all that. You can just prompt an LLM to write a script that extracts the relevant data from the user's CSV, then run that script immediately to get the data you need for further processing.

The same approach works for JSON or XML data, arbitrary log data, and even HTML or PDF data. Using an LLM, you can create bespoke code for each input at runtime rather than constrain input to meet the expectations of your code.

Once you have the code, you can cache it and run it on similar-looking future input without an inference round-trip.

No code review? Why it works

Just-in-time programming works because LLMs can write code fast. When I first started making web applications, it was common to just edit some scripts in your production server's cgi-bin directory and hit save. The next page view on your site would immediately run the new code. The pace of iteration was incredible, but you'd endure a few seconds (or even minutes) of downtime each time you forgot a semicolon. That much downtime is unacceptable for most production systems, so every software professional knows you shouldn't edit code live in production.

But when you think about it, this principle became dogma because humans are unpredictable, unreliable and most importantly slow. It takes time to realize our mistakes, and even more time to correct them.

LLMs code fast

In contrast, an LLM can write code, run it, see an error, rewrite the code and retry all in the span of a second or two. That's less time than it takes VS Code to start up on my machine.

Increasing reliability

Though LLMs are fast relative to humans, they aren't exactly known for their reliability. But when specifically tasked with writing code, they are already quite good and getting better. Still, they aren't going to give you correct code 100% of the time.

What we've learned is that a simple retry loop where you include the error message in a follow-on LLM prompt works remarkably well. And in cases where that fails, after a couple of attempts, you'll typically fall back to a user-facing error message explaining the failure and suggesting a manual reformat of the input.

Making Just-in-Time Programming easier

We've found ourselves servicing this just-in-time programming system design because LLM-generated code is properly treated as untrusted, and we provide an isolated locked-down runtime for untrusted code execution.

But the runtime is just the first piece of a larger just-in-time programming platform that we're excited to build. We've learned from our early customers that they'd also need help with:

• A framework for integrating the write code, see error, retry loop
• End-to-end testing
• Runtime observability
• Tailored LLM code output evaluation

We're just getting started on building the best-possible toolkit to put just-in-time programming into your production applications.