Skip to content
AppDues
Log in
Contact salesSign up

Why a familiar coding format beat a custom agent harness

A paywall editor that finished requests still disappointed users on spacing, alignment, and overall polish. Replacing a custom JSON-patching workflow with editable JSX and a standard coding loop improved design quality, cut prompt size, and shifted user feedback.

Completion was not the same as quality

The first version could create a paywall or revise a section, so completion metrics looked acceptable. Users and internal reviewers still saw weak spacing, alignment, and nested-card choices. Trying other models, adjusting prompts and skills, and adding screenshot prompts helped somewhat, but the gap remained large. A comparison with a general coding assistant made the issue clearer: the same request produced a far more polished result there, even without paywall-specific design instructions.

The custom format became the bottleneck

Paywalls were stored as JSON component trees, and the agent manipulated that structure directly through custom tools guided by an annotated schema. That design prevented some formatting errors and showed valid shapes, but it required the model to learn an unfamiliar representation on the fly. It had to convert a request into visual intent, then into JSON operations, then evaluate a screenshot. The hypothesis was that taste was being lost during that translation, because models have far more training exposure to JSX, SwiftUI, and similar UI code than to this bespoke patching syntax.

JSX as the editing surface, compiler as the gate

The replacement kept the backend JSON contract but changed the agent-facing representation. A transpile step turns JSON into JSX that resembles the underlying component tree, the agent edits it as a file, and a strict compiler either reports blocking problems with line-level guidance or converts the JSX back to JSON. A successful compile also returns a screenshot for visual checking. In the purchase-button example, the JSX form expresses the same stack, background, padding, and text in a familiar component syntax (see protected module code0002, contrasted with the JSON form in code0001).

A coding workspace instead of a bespoke harness

The workspace was then organized like a coding project: a main paywall file, read-only snippets for reusable component patterns, lazily loaded type definitions instead of one large schema payload, translation files under a strings folder, and a context file for app background. The agent used familiar read, write, and edit operations plus compilation. This structure also makes new capabilities additive: add files or folders and load them only when needed. The supplied workspace layout is preserved as protected module code0003. With the familiar format, more than half the system prompt and almost all skills could be removed, because much of that text had been compensating for the format.

Validation before rollout

Before production, the team replayed hundreds of sampled production traces against old and new agents, used an automated judge to catch functional regressions, and asked internal reviewers to blind-vote on taste. Rollout began with users who had raised taste concerns. Production work included more compute for extra turns and compilation, a model switch based on taste performance, and hardened retries and fallbacks after observing stalled streams and empty responses from a new inference provider. The editor reached full rollout on September 2nd.

Results and trade-offs

Across roughly 46,000 requests, positive feedback increased 83% and negative feedback decreased 40% versus the prior two weeks. Simple tasks such as copy changes, adding a button, compliance checks, and translations showed little difference. The improvement appeared where judgment mattered most: large edits, creating a new paywall, and other taste-dependent work. The new agent is slower because it reads context, edits, compiles, and iterates, but the feedback data suggested quality outweighed that latency cost in this case.

Decision checklist for agent builders

Use this checklist when a custom agent feels stuck: 1) Separate completion from taste by collecting blind human comparisons. 2) Treat a growing pile of brittle prompts and skills as a signal to inspect the format first. 3) Prefer representations and tools the model already knows well. 4) Add a compiler or validator that gives precise, actionable diagnostics. 5) Prototype a large format change when you have evaluations, because the payoff can outweigh the cost. 6) Budget for latency, compute, and provider failure modes before rollout.

Custom harness vs coding-loop rebuild
AreaEarlier approachRebuilt approach
Editing representationDirect JSON tree manipulation with custom toolsJSX file edited with familiar coding operations
ValidationSchema guidance plus screenshot self-checkStrict compiler with diagnostics and screenshot on success
Context deliveryLarge schema payload at session startLazily loaded type files, snippets, strings, and context file
Prompt burdenLong system prompt and many skills compensating for formatSystem prompt cut by over half; most skills removed
Reported feedbackBaseline before rolloutPositive feedback up 83%; negative feedback down 40% across about 46,000 requests

Comparison based on the described Paywalls AI Editor rebuild; metrics are from the reported rollout period.

Frequently asked questions

Why did the first agent complete tasks but produce poor designs?

The earlier design asked the model to learn a bespoke JSON schema and custom patching tools, then translate visual intent into tool calls and judge the result from a screenshot. That translation step lost design quality even when tasks completed.

How did the JSX approach work?

A transpile step converts stored JSON into editable JSX, the agent edits it as a file, and a strict compiler reports blocking issues or returns the JSX to the expected JSON structure. The compiler also provides a screenshot on success.

What changed after rollout?

Across about 46,000 requests, positive feedback rose 83% and negative feedback fell 40% compared with the two weeks before rollout. Gains were concentrated in large edits, new paywall creation, and taste-dependent work, with little difference on simple copy or button changes.

What should teams building their own agent check first?

Check whether prompt and skill volume is compensating for an awkward format, measure taste with blind human comparisons in addition to completion, prototype large format changes when evaluations exist, and weigh added latency against quality gains.

{
  "type": "purchase_button",
  "stack": {
    "type": "stack",
    "background": { "type": "color", "value": { "light": { "type": "hex", "value": "#ff8a00ff" } } },
    "shape": { "type": "pill", "corners": null },
    "padding": { "top": 12, "bottom": 12, "leading": 8, "trailing": 8 },
    "components": [
      {
        "type": "text",
        "text_lid": "PBcCWPIKbT",
        "color": { "light": { "type": "hex", "value": "#ffffffff" } },
        "font_weight": "semibold",
        "margin": { "top": 0, "bottom": 0, "leading": 0, "trailing": 0 }
      }
    ]
  }
}
// ... one of 20+ components on this screen