AI Engineer NYC 2026 Main stage · 15 min

What does an AI engineer do?

Aparna DhinakaranArize AI

Arize Signal reads traces, finds recurring problems, and can propose fixes. Building that loop requires decisions about memory, temporary access, recovery, and what counts as a good fix.

Build the loop that builds the feature

1:27:26 ↗

Using an assistant to write a feature and operating agents that keep improving the product are different engineering jobs. The first gives a developer another way to produce code. The second requires a system that can decide what work needs doing, propose a change, and tell whether the change helped.

Aparna Dhinakaran starts with the gap between those ways of working. As engineers manage more agents, they need to build review systems and turn telemetry—the record of what the application did—into something an agent can investigate.

Arize’s Signal is her concrete example. It reads traces, identifies recurring issues, and—when given repository access—can propose a fix or a pull request. The interesting engineering lies in making that cycle repeat without losing context, leaking access, or mistaking every pattern for a real problem. Watch the shift at 1:29:44.

Give every run a beginning and an end

1:31:10 ↗

Signal begins by reading a configurable volume of traces on a cadence. It needs to remember issues it has already found. It also needs to recognize when something that previously looked unimportant has become a pattern worth investigating as more evidence arrives.

This changes the role of observability. The primary consumer may be an agent rather than a person scanning a dashboard. Telemetry can also become the trigger that wakes the agent. A scheduled run might inspect recent traces every few hours; an event, a metric, or a user complaint in Slack can provide another starting point.

Once triggered, the run needs a defined environment. Signal gathers trace context, relevant application-performance data, repository access, and the credentials required for the job into an isolated sandbox. A single proxy controls information moving in and out. The run itself is traced, and its outputs are saved separately.

Signal’s agent-run anatomy, showing trigger and admission steps, the ephemeral sandbox layers, and results, traces, and teardown.

The run diagram at 1:33:58 distinguishes the short-lived workspace from the outputs that survive it.

At completion, useful state remains: identified issues, the history of the agent’s work, and any learned skills saved to the repository. The sandbox, credentials, and repository access are torn down. Persisting the right information means the next run does not start from nothing, while the temporary execution environment does not become an indefinitely privileged machine.

Failure needs the same separation. If a sandbox or harness fails partway through, state outside the sandbox allows a replacement to resume from the failed step. The system’s continuity comes from that saved progress, rather than from assuming one process will stay alive forever.

Define good behavior before automating more of it

1:36:24 ↗

Dhinakaran describes Signal as independent of any single model or coding harness. That flexibility makes it possible to compare quality and cost: a stronger model may produce better proposed fixes, but the team wants to know whether a less expensive configuration can preserve the quality that matters.

Evaluations answer that question only if they measure the right thing. A basic regression test can compare outputs for a known set of inputs. More difficult judgments—whether an issue is real, whether a proposed change addresses it, whether the agent’s diagnosis is useful—need alignment with human judgment.

Much of the engineering work is learning to make those judgments repeatable. Reviewing traces alongside the agent exposes recurring errors; those errors become test cases. Each useful case saves the next reviewer from having to rediscover what a bad diagnosis looks like.

An evaluator also needs evaluation. Arize builds datasets with human judgments, checks the evaluator against those judgments, and works to align them before shipping a new judge. Trust in the evaluator should come from its agreement on meaningful cases, rather than from the evaluator’s confident wording.

The same evaluation can play different roles before and after deployment. Offline tests assess a proposed change. Online evaluations inspect behavior in production, where new cases and patterns appear.

Trace the agents that build the agents

1:39:57 ↗

The final loop extends observability to the development machinery itself. Tracing the deployed agent is useful, but it leaves out the coding agents that created or changed it. Their tool calls, edits, and proposed fixes also belong in the record of how the product evolved.

Production traces and online evaluation results then become the next input to Signal. The system collects data, evaluates behavior, runs agents over that evidence, identifies gaps, and proposes improvements. Each cycle creates the evidence for the next.

Business data is messier than the constrained cases an agent may handle well in a lab. Dhinakaran’s answer is to keep that mess inside the learning loop: trace the failure, turn it into a useful evaluation, and give the next agent run enough evidence to propose a better fix. That continuing process becomes part of the product the AI engineer builds.

Full transcript

Timestamps follow the original livestream. Select a time to watch that moment. Transcripts are machine-generated with reviewed name and terminology corrections; occasional recognition errors may remain.