Why a familiar coding format beat a custom agent harness
A paywall editor that finished requests still disappointed users on spacing, alignment, and overall polish. Replacing a custom JSON-patching workflow with editable JSX and a standard coding loop improved design quality, cut prompt size, and shifted user feedback.
Completion was not the same as quality
The first version could create a paywall or revise a section, so completion metrics looked acceptable. Users and internal reviewers still saw weak spacing, alignment, and nested-card choices. Trying other models, adjusting prompts and skills, and adding screenshot prompts helped somewhat, but the gap remained large. A comparison with a general coding assistant made the issue clearer: the same request produced a far more polished result there, even without paywall-specific design instructions.
The custom format became the bottleneck
Paywalls were stored as JSON component trees, and the agent manipulated that structure directly through custom tools guided by an annotated schema. That design prevented some formatting errors and showed valid shapes, but it required the model to learn an unfamiliar representation on the fly. It had to convert a request into visual intent, then into JSON operations, then evaluate a screenshot. The hypothesis was that taste was being lost during that translation, because models have far more training exposure to JSX, SwiftUI, and similar UI code than to this bespoke patching syntax.
JSX as the editing surface, compiler as the gate
The replacement kept the backend JSON contract but changed the agent-facing representation. A transpile step turns JSON into JSX that resembles the underlying component tree, the agent edits it as a file, and a strict compiler either reports blocking problems with line-level guidance or converts the JSX back to JSON. A successful compile also returns a screenshot for visual checking. In the purchase-button example, the JSX form expresses the same stack, background, padding, and text in a familiar component syntax (see protected module code0002, contrasted with the JSON form in code0001).
A coding workspace instead of a bespoke harness
The workspace was then organized like a coding project: a main paywall file, read-only snippets for reusable component patterns, lazily loaded type definitions instead of one large schema payload, translation files under a strings folder, and a context file for app background. The agent used familiar read, write, and edit operations plus compilation. This structure also makes new capabilities additive: add files or folders and load them only when needed. The supplied workspace layout is preserved as protected module code0003. With the familiar format, more than half the system prompt and almost all skills could be removed, because much of that text had been compensating for the format.
Validation before rollout
Before production, the team replayed hundreds of sampled production traces against old and new agents, used an automated judge to catch functional regressions, and asked internal reviewers to blind-vote on taste. Rollout began with users who had raised taste concerns. Production work included more compute for extra turns and compilation, a model switch based on taste performance, and hardened retries and fallbacks after observing stalled streams and empty responses from a new inference provider. The editor reached full rollout on September 2nd.
Results and trade-offs
Across roughly 46,000 requests, positive feedback increased 83% and negative feedback decreased 40% versus the prior two weeks. Simple tasks such as copy changes, adding a button, compliance checks, and translations showed little difference. The improvement appeared where judgment mattered most: large edits, creating a new paywall, and other taste-dependent work. The new agent is slower because it reads context, edits, compiles, and iterates, but the feedback data suggested quality outweighed that latency cost in this case.
Decision checklist for agent builders
Use this checklist when a custom agent feels stuck: 1) Separate completion from taste by collecting blind human comparisons. 2) Treat a growing pile of brittle prompts and skills as a signal to inspect the format first. 3) Prefer representations and tools the model already knows well. 4) Add a compiler or validator that gives precise, actionable diagnostics. 5) Prototype a large format change when you have evaluations, because the payoff can outweigh the cost. 6) Budget for latency, compute, and provider failure modes before rollout.
| Area | Earlier approach | Rebuilt approach |
|---|---|---|
| Editing representation | Direct JSON tree manipulation with custom tools | JSX file edited with familiar coding operations |
| Validation | Schema guidance plus screenshot self-check | Strict compiler with diagnostics and screenshot on success |
| Context delivery | Large schema payload at session start | Lazily loaded type files, snippets, strings, and context file |
| Prompt burden | Long system prompt and many skills compensating for format | System prompt cut by over half; most skills removed |
| Reported feedback | Baseline before rollout | Positive feedback up 83%; negative feedback down 40% across about 46,000 requests |
Comparison based on the described Paywalls AI Editor rebuild; metrics are from the reported rollout period.
Frequently asked questions
Why did the first agent complete tasks but produce poor designs?
The earlier design asked the model to learn a bespoke JSON schema and custom patching tools, then translate visual intent into tool calls and judge the result from a screenshot. That translation step lost design quality even when tasks completed.
How did the JSX approach work?
A transpile step converts stored JSON into editable JSX, the agent edits it as a file, and a strict compiler reports blocking issues or returns the JSX to the expected JSON structure. The compiler also provides a screenshot on success.
What changed after rollout?
Across about 46,000 requests, positive feedback rose 83% and negative feedback fell 40% compared with the two weeks before rollout. Gains were concentrated in large edits, new paywall creation, and taste-dependent work, with little difference on simple copy or button changes.
What should teams building their own agent check first?
Check whether prompt and skill volume is compensating for an awkward format, measure taste with blind human comparisons in addition to completion, prototype large format changes when evaluations exist, and weigh added latency against quality gains.
{
"type": "purchase_button",
"stack": {
"type": "stack",
"background": { "type": "color", "value": { "light": { "type": "hex", "value": "#ff8a00ff" } } },
"shape": { "type": "pill", "corners": null },
"padding": { "top": 12, "bottom": 12, "leading": 8, "trailing": 8 },
"components": [
{
"type": "text",
"text_lid": "PBcCWPIKbT",
"color": { "light": { "type": "hex", "value": "#ffffffff" } },
"font_weight": "semibold",
"margin": { "top": 0, "bottom": 0, "leading": 0, "trailing": 0 }
}
]
}
}
// ... one of 20+ components on this screen