Our Lambda thinks it's a phone

Version 1.0 · Published 2026-08-26

We are not allowed to wait

So, we are technical folks, making an AI product. And we like rules. So, we gave ourselves a new rule this year: nothing on the request path is allowed to wait for another computer.

That is an absurd thing to say out loud in 2026. Every interesting feature ships as an API call, and the going advice is to call more of them rather than fewer. We wrote the rule anyway.

We do this to ourselves roughly once a decade. Ten years ago the rule was that we could not use servers, and the architecture that came out of it still runs underneath UNLESS today. A restriction you cannot argue your way around does design work that no meeting ever will.

But this new one broke something immediately. The casualty was the most important step we run.

The first thing that broke the rule

We cater to regulated industries in the European Union, like financial services. So, every question a customer types to our Customer Agent passes through a PII filter. Names, email addresses and the rest come out before any model sees the text. Keeping personal data away from model providers is a hard requirement at Unless, not a preference.

We ran that filter on AWS Bedrock Guardrails. It is managed, reliable and boring in a good way. It also runs somewhere else, so that requires waiting.

When invoked, the function stops. It waits 200-400ms. Then it gets on with the answer.

You can file that under “the network is slow” and forget about it. The wait sat in front of every single question, and it multiplied across a multi-turn interaction. We did not accept that and started thinking. Unfortunately, there is no clever way to await faster. Obeying our own rule meant the model had to come to us.

Testing an on-device model against a managed service

Luckily, something had shifted on the model side while we were looking elsewhere. Phones and browsers created demand for models that work with no network at all. Small, quantized so the weights are stored at lower precision, native, and built for hardware that belongs to somebody else.

That work is being done for a device with a battery, a slow chip and no patience. None of it is aimed at a cloud function. We realized that the differences are pretty small, though. An AWS Lambda function is like a device, give or take a screen. Small computer, on its own, short life, nobody nearby to call. It sounds like a great use case for an on-device model.

Which sounds neat and proves nothing. So we tested a phone-sized specialist against the managed service we were already paying for. And guess what? It caught more. It cost less. And it ships to every function as one layer.

What we installed

The model is @desert-ant-labs/redact version 3.0.0. Redact detects 17 GDPR-relevant labels and does nothing else at all.

Underneath sits a prebuilt native core running on LiteRT, Google’s runtime for on-device inference. Node calls into it through koffi FFI. The model weights do not ship inside the npm package, so we pin a version at build time.

Those files and the native core go into one Lambda layer built with CDK, 47 MB published. Any function that needs filtering attaches it. One artifact to maintain, no per-function copies.

The whole thing behaves like an installed app. It arrives at a pinned version, it sits on disk, and it starts when the function does. Nothing phones home.

We expected to pay for this

Self-imposed rules usually cost you something. We assumed a model small enough to ship in a 47 MB layer would lose to a managed service from AWS. We wanted the size of the loss before committing to it, or even use it as an additional control mechanism.

So we wrote 102 user queries, phrased the way people really type to the agent, EU-focused. Every piece of personal data in the set is synthetic. Nothing real was exposed in testing.

Then we tried to trip both detectors. Alongside the queries carrying real PII we planted hard negatives: order numbers, SKUs, dates, prices and public company names. They look like personal data, and they are exactly what an eager detector grabs - and what spoils the result. Strip an order number out of a question and the agent answers a mangled version of it. The customer gets a worse reply and nobody ever sees why.

One caveat on the scoring, because it changes what the numbers mean. We scored at query level: did each one come out classified correctly, PII or clean? Production does the harder job of masking every entity inside a question, so read these as detection numbers.

The price came out negative

Across the 102 queries the local model made zero errors. Zero. Guardrails made eight. Calling that 92% against 100% hides the interesting part, because they are two different mistakes that fail in opposite directions.

The PII that got through

Of the 102 queries, 57 contained PII. A miss here is the one that counts, because personal data then reaches the model provider - usually a third-party cloud provider in Europe. That is the exact event the filter exists to prevent, because we do not like loose ends. And neither do our customers.

Guardrails caught 53 of them and missed 4, a catch rate of 93.0%. Redact caught all 57. On our set, four leaks against none.

The clean queries that got mangled

The other 45 queries were clean, hard negatives included. A false positive costs quality rather than compliance. The agent ends up answering a redacted version of a perfectly innocent question.

Guardrails flagged 4 of the 45, a false-positive rate of 8.9%. Redact flagged none of them. It won on both kinds of mistake at once.

Result on 102 queries	AWS Bedrock Guardrails	Redact 3.0.0 (local, 17 labels)
Correct	94/102 (92.2%)	102/102 (100.0%)
True positives (PII caught)	53	57
False negatives (PII missed)	4	0
True negatives (clean, not flagged)	41	45
False positives (clean, flagged)	4	0

Before anyone gets excited, we put a specialist against a generalist. Guardrails is broad and tuned conservatively for many use cases, while Redact does one job. A specialist beating a generalist at its own job is not news, but the margin and the running cost are.

Two more details keep it honest. The AWS column is our real production configuration, not a strawman we detuned to lose. And 100% means zero errors on 102 curated synthetic queries in a single run. That is a strong result on this set, not a forecast of what happens at scale.

Don’t forget, you pay twice for a wait

Accuracy was the reason to switch. Latency and cost were the refund. The managed call took 200-400ms per question, and the local model takes 30-50ms on the warm path.

Those two numbers are not measured the same way. The first includes the round trip to Bedrock. The second is warm-path inference inside a running function, with the one-time load on a cold start left out.

Then there is the fee. Guardrails is billed at $0.10 per 1,000 text units, and a text unit is 1,000 characters. Our questions are short, so call it one unit each, or about 10 cents per thousand.

Here is the naive way to read the swap. We partially deleted a 10-cent fee and added 30-50ms of compute. It misses half of what happened. Obviously, your on-device model may not be free either, but if you host it yourself the costs are probably just a fraction. But that is not what we mean.

The call was awaited, and the function sat there blocked with nothing else to do. Lambda bills by duration. Idle waiting is billed exactly like working, so we were paying Amazon to hold the line.

So the swap cuts on both ends. The API fee is gone, and 200-400ms of paid waiting became 30-50ms of paid work. Call it 150-370ms of billed duration per question.

One honest offset belongs in your own math. The model and its runtime needed room, so the function went from 1,024 MB to 1,536 MB. Higher memory raises the per-millisecond price of the entire invocation, not only the filter step.

So run the real sum on your own function: (new memory × new duration) − (old memory × old duration). Ours lands clearly on the local side. Yours might not.

What the rule cost us

Nothing here is free, and the bill arrives on cold starts. A cold container loads the model once before the first inference, which adds about 240-330ms.

We measured 2% of invocations hitting a cold start. Every warm one after that runs at the 30-50ms number. Cold starts never became a problem for this workload.

Spiky traffic breaks that arithmetic completely. Measure init time on your own config before you trust anyone’s latency figures, ours included.

What the rule does not buy

The rule bought latency and money. It did not move the filter out of the processing chain, and we hold ourselves to that distinction.

A PII detector reads raw input by design, because that is how it finds personal data at all. It stays a listed sub-processor whether a third party operates it or we do. Running it inside our own function changes nothing about what it can see.

What the filter protects is narrower and checkable. Raw identifiers stay out of the third-party model that phrases the answer, and that model receives tokenized fields it cannot reverse. Our Privacy Vault rests on that fact, before this change and after it.

Filtering is best-effort minimization against detectable identifiers, and never anonymization. The four misses in the table are why the distinction earns its keep. We have written more about PII filtering under GDPR and about the security boundary around a Customer Agent.

Pick a rule you cannot argue with

Picking a hard rule and seeing how far we would get brought us into this new class of micro-models. Anything that could run in a mobile app is great for Lambda.

And the list of jobs a small local model can take over keeps growing. We have not tested the rest, so take this as a direction rather than a result. Summarizing an interaction, giving one a title, sorting a question into a topic. Each of those is an await today. Each is also a small, well-scoped job that a specialist could plausibly do in-process.

So go and look at the awaited calls on your own hot path. Measure the wait you are already paying for, then ask whether something built for a pocket could do that job inside your function.

Our PII filter has never seen a phone. It just needed a small computer with a short life, and that is exactly what a Lambda is.