How do you enforce the read-only guarantee across language runtimes and probe types? Is there a policy layer that rejects expressions with side effects before instrumentation, and do you expose an audit trail showing exactly what each agent probe captured?

by defalut, probes basically tell the fileName/className and lineNo/methad name to attach to (in additon to metadata like serviceId, environemnt name etc)

This is readonly and safe by default.

expressions come into play for conditonal probes

read only safety guarantees here depend on the runtime

NodeJS: handled implicitly by using `throwOnSideEffect: true` any side possible effects are prevented using this

Python and Java: As of now, we don't let conditions have method invocations at all and only allow a subset of comparator operators. no assignment allowed

usually property can invoke getter which usually should be safe to execute by design, but since we cant guarantee how it would have been written, we dont allow that as well for now.

order.total > 50 => not allowed

total > 50 => allowed

to get around this we use multiple probes, agrregated by the current context's traceId (if avaliable)

we plan to eliminate this problem by adding a custom DSL + AST parsing which can act as the policy layer to dissallow condtional probes

Audit trail is in our roadmap. As of now, you can delete the data that's collected by probes. the only problem we have with audit trail is what if you capture something sensitive and that remains in your audit trail.. so we need some immutability that registers audit trails.. but then have enough flexibility to remove the data collected.. can be done

How do you distinguish durable user preferences from transient screen context before an agent turns recorded activity into an automation? I'm especially curious whether each inferred memory keeps provenance and an expiry or confidence signal so stale behavior does not become a permanent rule.

yes we keep track of memories confidence level and timestamp,

we found that in order to find automation opportunities it's better to have agents segment, classify (for example personal vs pro), enrich (which customers/people are related to this workflow) and organize the data first, then it's much easier to find repetitive tasks

Interesting direction. When Claude Code and Codex disagree on an implementation path, do you keep their rationales separate for review or merge them into one confidence state?

Every time I've seen different agents disagree they provide their reasoning and eventually converge. If they ever weren't able to come to a resolution, the author agent could end the loop by declining to push new changes, and a person could enter the loop and unblock them.

Interesting technical direction. What signal do you use to decide when the agent should stop gathering context and start making a concrete code change?

We don't. We let agent decide it. But tools themselves are configured in a way that agent can easy understand how much and what data it needs. More importantly, it can snapshot the state before it makes the change, and do the diff after the change and see if any unintended impact was made.

Thanks for sharing this. For session-shaped benchmarks, how would you keep the evaluation fair when cache state and accumulated context differ across Claude Code and Codex runs?

Here's my current plan, the "session" will be made up of multiple SWE bench tasks stitched together.

Each "task" is the equivalent of a new user query and we also pre-program "cache expiration" (sleep for 5 mins) into the session. This ensures parity across providers (both default to 5 min TTLs).

The goal of this exercise is to tease out how Claude Code and Codex differ in managing their context and how that impacts cost and quality for the same simulated session.

Interesting approach. How do you define the boundary contracts so they stay strict enough for cheaper models without becoming too brittle when the architecture changes?

Through the use of module boundary level contracts and strict checks like dependencies/imports, because that’s where cheaper models would often stray. I think it’s important to keep the checks simple instead of complex coding checks. For example, “accounts is not allowed to import billing”.

Interesting approach. How do you handle conflicts between an older persistent memory and the current repository state—for example when APIs or architecture changed since the memory was written?

Curious how you handle trust boundaries for tool outputs here. Do you keep a signed or replayable trace so a developer can audit what the agent saw before it acted?

Interesting approach. Does keeping the model in HTML also preserve enough structure for tracked changes/comments, or do you handle those as a separate layer when converting back to DOCX?

Thank you!

My thesis is that an intermediate layer would eventually end up being equivalent to the docx format, so I've decided not to have any intermediate representation.

We convert docx to html and send it AI. When AI rewrites the HTML and it back, we diff the rewritten HTML against the docx's document.xml and make the modification. This is a simplistic explanation of it. There are a bunch of validations and processing going on.

Regarding the tracked changes/comments, we simply invent new HTML tags for those things e.g. <ins>, <del>, <commentRangeStart> and etc.

One detail I would be curious about: how do you make the agent run auditable enough that another developer can understand why it chose a specific tool or edit path?