Agents for correct, transparent, and reproducible analysis

Simon Couch and Sara Altman

posit::conf(2026)

how is traffic trending for our site?

Analyzing site traffic

⌕ Searching workspaceweb analytics ✓

◫ Querying warehouseanalytics.pageviews ✓

∿ Calculating daily visitslast 90 days ✓

Daily visits are down 64% over the last 90 days.

Daily site visits−64%

This morning, your VP pulled up a coding agent and asked a question.

[yada yada this is what happen]

Now, as data people, we might have asked ourselves “How are daily visits being measured?” “What event triggers an entry in this column?” “What other possible explanations could account for this change in trend other than a genuine reversal in the traffic going to this site?” If you had dug further, you would have learned that this kind of pageview event no longer triggers on newer versions of Chrome. As users gradually update Chrome, their page visits are no longer registered in this table. Regardless, this isn’t even the typical table that the data science team uses to measure site traffic anyway.

Nevertheless, you’re not there. Your VP brings this chart to leadership. They decide to spend millions to reverse the trend.

could you find the site traffic chart we made a month ago?

Looking for prior analysis

⌕ Searching conversationssite traffic chart ✓

◫ Checking generated filestemporary workspace ✓

I couldn’t find that conversation or any saved analysis files. Do you want me to recreate that chart for the last 90 days?

yeah

Analyzing site traffic

⌕ Searching workspaceweb analytics ✓

◫ Querying warehouseanalytics.sessions ✓

∿ Calculating daily visitslast 90 days ✓

Daily visits increased 22% over the last 90 days.

Daily site visits+22%

A month later, leadership will ass to see the chart again. The VP will open up the same agent, but it can’t find the conversation. (It was deleted after 30 days.) All of the analysis had happened in a tempfile that is long gone.

The agent offers to make it again. This time, when the VP asks for the same plot, they see that site visits had been steadily trending upward the whole time.

This is something that we’ve seen quite a bit internally at Posit, and also with many of the organizations we work with. It’s becoming normal to vibe-analyze data.

The idea for this hypothetical came from this hilariously-titled episode of The Test Set podcast.

In the battle between convenience and correctness, convenience is winning.

Joe Cheng (CTO @ Posit)

Posit’s CTO recently summarized this trend like so:

It is true that AI has largely been a threat to correctness. A broader population of people can now convincingly do data analysis.

It just so happened that Joe said this a month ago, but he could have said it 6/7 years ago (ayyy) and it would still have been true.

In 2019, Conviviality (a beverage company) discovered they had been mistakenly reporting healthy financials for years due to a “spreadsheet arithmetic error.”

£5.2m of that £14m came thanks to “a material error in the financial forecasts”. The error, it later admitted, was “a spreadsheet arithmetic error”.

Or Joe could have said it 20 years ago, when it came to light that a whole subfield of protein modeling had been distracted by a bug in in-house, closed-source software.

From 2001 to 2006, Geoffrey Chang published a series of heavy-hitting papers that reshaped his field, while other labs struggled to reproduce their findings. The software that the Chang lab used to carry out the analysis contained a bug that flipped the sign of a computation. The bug came to light when another lab solved a similar protein structure and saw that it produced a near-mirror image of Chang’s. At this point, the Chang lab checked their code, found the bug, and retracted 5 papers.

I bring these examples up to say that there have always been threats to correctness in data analysis. This is why Posit was founded; correctness in high-stakes data analysis has always been our thing. But our stance is not “you need to forgo convenience in favor of correctess.”

I was doing some research for this talk when I came across this old weblog…

Our goal is to develop a powerful tool that supports trustworthy, high quality analysis.

At the same time, we want RStudio to be as straightforward and intuitive as possible.

JJ Allaire (Founder @ Posit), 2011

He’s saying “it can be correct and convenient.”

What if our stance was the same with AI? In other words, can we build data analysis agents that are both correct and convenient to use?

Correct and convenient

this pull between correctness and convenience has weighed on all of our minds in this new age of llms

and it became especially relevant last year in 2025 for us and for many of you I imagine 2025 was the year AI got very very convenient and we needed to think seriously about correctness

and we were so worried

August 2025

so worried that when we released our new sota coding agent in 2025, we did so under what was essentially a warning label

this is an image from a blog post joe and I wrote to accompany the release of databot, which was an exploratory data analysis agent that lived in positron

and this post was actually one of the first things I worked when I joined the AI team, and it gave me the opportunity to do one of the things that I think I do best: be a pessimist

because we were all so excited about databot, felt like you were flying through data, but also so worried about people trusting its answers when it was wrong

that we wrote an entire blog post explaining the ways in which it might fail

In my 30-year career writing software professionally, Databot is both the most exciting software I’ve worked on, and also the
most dangerous.

Joe Cheng, Posit CTO

and if you were here at conf last year, you might remember Joe’s keynote where said something a bit like this

and given that I think at this point there might be a question on your mind, which I think is fair to ask and people ask me and I think we all have asked ourselves in the past few years

Why are we doing this?!

if AI is this agent of chaos that we are so worried, why involve ourselves at all

and there are many answers. I’m going to talk about one.

before this, I worked in data science education. I taught people data science… for a lot of people, including myself, curiosity is what gets them into this

Maybe it was curiosity about a scientific field that led you to data analysis (doing a PhD), or curiosity about the tools themselves, how computers work, or statistics.

But without the right tools, data can be impenetrable. As an undergraduate with knowledge of statistics and some R as an undergraduate, I felt like I was looking at an ocean from above.You know there is an incredible amount there, but you cannot see it.

learning the tidyverse, and data science…these tools actually gave me a way to investigate my data

The right tools let us see below the surface.

Photo: Lars von Ritter Zahony / Ocean Image Bank

the right tools let us see below the surface The tools enable the curiosity.

AI is a continuation of our work to find new ways to explore data.

what motivates me and my coworkers we are curious about how LLMs work, about how to build around LLMs to make them useful, and how to turn them into tools that open up new worlds of analysis for all of you

I also want to spend a few minutes talking about what’s changed since a year ago

The models got better

Model: “There’s a strong positive relationship between carat and price!”

Can models interpret plots that contradict their expectations?

around a year ago we did an experiment we wanted to know how well models interpret plots we showed them plots like this one, which plots diamonds and asked them to interpret them and they said stuff like this: and that sounds good until you release: that is not …

this is what we wanted to know: would models correctly interpret plots that contradicted their expectations?

November 2025

We tried a range of interventions to improve these scores, but none moved them much.

September 2026

until we waited 6 months

these are results from now

Correct analysis still takes more than a good model.

but this doesn’t mean that data science is solved good analysis requires more than just a good model and a dataset

after we did that experiment, we wanted to know something else: can models spot data quality issues in visualizations?

we showed them plots that look like this take a look at this plot. what do you notice?

one thing that flags my attention is these points through the middle of the cloud that fall exactly on a straight line. this is something i might want to dig into further and make sure nothing strange was happening in the underlying data.

we wanted to see if models would be able to pick up on this kind of subtle visual artifact

even frontier models don’t do very well so they’ve gotten better, but it’s not like they’ve solved data science and don’t make mistakes they still fail at this task and others that are likely to come up in data science workflows

but there’s another problem

Data analysis depends on context that is not in the data.

but there is a second problem: data analysis depends on context that is not in the data

all data is constructed, and it is often meaningless without its context

that context might live in documentation files, in data dictionaries, in semantic views, or in the mind of your coworker who’s been at the organization for 20 years and knows absolutely everything but has never written anything down

it’s not that this information is impenetrable to an llm…but…

so the models got better, but they still aren’t perfect they are going to make mistakes so if we are going to use AI for data analysis, the question becomes…

How do we make AI-involved analysis correct?

Agent

Human

does stuff

verifies, approves, etc.

• Here’s a common thought about AI
  • It’s untrustworthy, but useful.
  • To make it trustworthy, we need people to verify the output from or supervise agents.

it is an intuitively appealing division of labor. the agent does what it’s good at, the human provides the judgment to keep it from going off the rails.

*the slap a human on it approach

this sounds so nice! it’s really tempting to just do this everything and call it a day

the human is there to prevent mistakes and all will be good.

I like to call this simplistic division of labor the “slap a human on it” approach. we’re not really thinking about what the human will do or how exactly, but we are glad they are there and just hope for the best

and I’m being a bit facetious but …

“Databot, and LLMs in general, aren’t at a point where you can abdicate responsibility or suspend skepticism when working with data. … These skills help you catch errors, interpret results in context, and avoid being misled by confident but incorrect claims.”

this isn’t too far from what we wrote about databot a year ago

we wrote a whole article about the dangers of Databot and why users needed to remain skeptical and apply their expertise

All of that is still true. I stand by it. It’s inspirational! I would love to avoid being misled by confident but incorrect claims.

but notice that this puts the responsibility for correctness on the user: be there, remain vigilant, and apply your expertise

but we didn’t say HOW

Adding a human does not guarantee a better result.

• Human–AI combinations do not reliably outperform the better of the two.
• Approval fatigue.
• Trust, timing, cognitive load, social cues, and information design all shape decisions.

this does not mean every human-AI team is worse. it means that adding a human does not automatically produce a better system

people get tired of approving things

approval fatigue can be a big issue think about your own coding agent. when it asks you to approve a command, how often do you carefully read it, and how often do you just hit accept?

trust, timing, cognitive load, social cues, the interface, and the amount of information all affect how someone decides whether code or an action is correct or safe

full citations, in case anyone asks: - Vaccaro, M., Almaatouq, A., & Malone, T. W. (2024). “When combinations of humans and AI are useful: A systematic review and meta-analysis.” Nature Human Behaviour, 8, 2293–2303. - Crootof, R., Kaminski, M. E., & Price, W. N., II. (2023). “Humans in the Loop.” Vanderbilt Law Review, 76, 429. - Hughes, J. (2026, March 25). “How we built Claude Code auto mode: A safer way to skip permissions.” Anthropic. - Phillippi, C. (2026, August 7). “Auto mode is now the default in Claude Code for Pro, Max, and Team plans.” Claude by Anthropic.

People are not fixed safety components.

because people are not fixed safety components we can’t just slot them in whenever we are worried an agent system might be wrong

that doesn’t mean they shouldn’t be there! but we need to go further

How do we design the entire system to produce correct work?

the question is how we design the interaction and the entire system so that human expertise can actually support correctness

Build trust and correctness into the agents themselves

Consider how the agent will interact with the user and support their expertise

Help the agent be correct

Make it less bad when the agent is wrong

these end up mapping onto two principles that we’re going to get coming back to help the agent…

but we know it’s going to make mistakes, so

make it less bad…

Our flagship agent is Posit Assistant. Posit Assistant is a coding and data science agent. It’s available in RStudio, Positron, and the terminal. When it’s running inside of your IDE, it works inside of your active R and Python session. You can use whatever model from whatever provider you want with it.

In thinking through how to demo out Posit Assistant, I thought about how Sara described her first moments of really getting the hang of data science tools as like seeing below the surface. I think many of us might have had a moment like this at some point, when we began to understand how powerful data science and statistics are for answering questions about the world.

The first time I felt this way, I had been asked to use R to understand what affects flight delays. I had to load in some data and then write dplyr and ggplot to see if I could make any decisions that would give me better chances of my flight leaving on time.

In demoing Posit Assistant, I thought I might revisit that moment with you.

I’m in Positron, with Posit Assistant pulled up on the left hand side and a live R session. I’ll start off by asking the agent about R packages that could help me grab data on flights.

…

It can do normal coding agent stuff

It can read documentation (in the same way that you would, too)

Also, note that it preferred to keep data in the file system

• Cleaning mode: we felt that most coding agents had a superficial relationship with data quality
• It pays attention to missing data, outliers, etc
• Cleaning script is persisted

You can see plots

The agent shows restraint in interpretation; is concerned about sample sizes before interpreting this anomoly in the plot

Slash commands (has a bunch of specialized skills) + can write quarto reports

Now, this is the end of my demo of Posit Assistant. I do have one more thing I want to call out, though.

The whole conversation was 5 cents. Part of our mission is providing the best tools to folks regardless of their economic means, and we remain committed to that. Totally free usage is not possible—AI inference is quite expensive—but Posit Assistant is token efficient, and we’ve worked hard to serve incredibly cheap and capable open-weights models with strong privacy guarantees for people that use our Posit AI Pass subscription. Per token, this model, GLM 5.3 Flash, is less than a thirtieth of the price of Claude Opus 5, and is served under Zero Data Retention.

Help the agent be correct by encouraging sound statistics and reproducibility practices

Make it less bad when the agent is wrong by making good use of your expertise and sharing an R/Python environment

Let’s revisit those lessons learned on agent-building from Sara. She said we ought to Help the agent be correct, and make it less bad when the agent is wrong.

Much of helping the model not be wrong is by asking nicely :) We tell it to approach data analysis with an openness to uncertainty, we tell it work in persistent files, we tell it to take data cleanliness seriously.

Various elements of Posit Assistant help you and the agent have a richer collaboration. Data cleaning mode loops you in on important decisions that affect interpretation. We show you plots because our bluffbench evaluations show we can’t trust the models to interpret models well. We place you and the agent in the same computational environment so that, when things go wrong, you can quickly get on the same page and see eye-to-eye.

That said, what about people that don’t have data expertise and don’t know how to write R or Python code?

posit assistant assumes you are a data scientist who wants to work in an ide

But what if the user is not a data scientist?

but what if the user is not a data scientist? how do we help them get correct answers too?

how is traffic trending for our site?

Daily visits are down 64% over the last 90 days.

Daily site visits−64%

let’s go back to that vibe-analyzing VP easy to say… but they are curious too! they had a question and reached for the tool that was accessible to them.

Unfortunately, the answer they got was wrong.

This isn’t the table that the data scientists usually use to measure site traffic…they have code they often use to analyze site traffic that gives them the correct results, and they vet and maintain this

Trusted code

This vetted code used by the data scientists is what we’ll called trusted code they rely on it and trust it to give them the right answers

They also might have trusted code in

• Shiny apps
• R and Python packages
• Quarto documents

Instead of the model writing code afresh every time it’s answering a question, what if it could discover and use this trusted code?

This is one of the core ideas behind a new open source package that we’re excited to share with you all today. It’s called commons.

commons helps data scientists build trustworthy data agents for their collaborators.

It’s an open source R and Python package built on ellmer / chatlas and shinychat. You can use it with whatever model from whatever provider you want.

and it’s a bird because…

let’s see how the VP’s analysis would work with a commons agent

The first thing the agent does it search for a trusted calculation. Here it finds and runs the data scientist’s site traffic calculation

the answer is marked as verified to signify that it used vetted code

However, its unlikely that this trusted code will cover all questions a user might have. For these situations, the agent can write custom r, python, or sql.

First, the agent will look through additional context in the form of data dictionaries, markdown files, and, if you have them, existing context layers from Databricks and Snowflake.

Then, it will write whatever code it needs to to find an answer.

Finally, when it presents an answer, it can provides a citation to that context. That citation will be deterministically checked against trusted context that has appeared so far in the conversation—if it matches, this citation tag will be shown.

There’s one more possible path, which is that the model cannot find trusted context to justify its approach. In this case, we mark the answer with this little yellow tag, which lets the user know that this answer should be considered less trusted.

This is how we’ve chosen to balance convenience and correctness. The tool has to feel convenient enough that the user actually wants to reach for it. Basically, it needs to feel like a normal chat. In the situation where we suspect the correctness is similar to a normal coding agent, we signal to the user that they ought to be more skeptical of this answer.

Search trusted calculations

Find one↓ Don’t find one↓

Run trusted calculation

↓

Search context

↓

Write SQL/R/Python

Cite↓ Don’t cite↓

Commons records which tools produced the answer. Trust is based on that execution path, not on the model’s confidence in its own answer. trust is deterministic

The citation itself is checked against trusted context. A custom answer is not promoted merely because the model says it used a trusted source.

Build

You construct the agent by hooking it up to trusted code.

Deploy

Your colleagues use the agent deployed on Connect.

Improve

You review the agent’s behavior, adding trusted code to cover common use cases.

So far, we’ve only focused on the experience of the VP. I talked about how this was a tool for data scientists, so let’s take a step and see what it looks like from yall’s perspective.

Remember when Sara showed you the simple loop? Working with commons is more like three loops.

1. build: encode trusted calculations and context After installing the package, you point the commons agent at all of your trusted code. Shiny apps, R and Python libraries, Quarto docs, or existing semantic layers in Snowflake and Databricks Notably, you are not defining system prompts or tool definitions. commons ships with its own harness.
2. deploy: agent answers through the trusted or custom path Apps deploy to Connect like any Shiny app The agent runs R and Python code in a sandbox so that it can be safely deployed
3. improve: review behavior and update the layers With the newest Connect release, once you and your admin opt-in, conversation histories will be stored with a native OpenTelemetry integration. After the app has been deployed, you can read in conversation histories and find common use cases where the agent has to improvise an approach. Based on those findings, you can revisit step 1, providing more trusted code to the agent.

Help the agent be correct with trusted calculations and context

Make it less bad when the agent is wrong by signaling trust and introducing feedback loops

Let’s revisit those guidelines one more time.

pos.it/commons

thank you and momentous feelings

Not just BI

we showed you commons in the example of the vp but commons is not just for bi it’s for anyone

commons creator	commons user	Example question
Biostatistician	MD	“Compare treatment groups.”
UX researcher	Product manager	“Summarize experiment results and explain which segments changed.”
Policy analyst	Program lead	“Simulate the budget impact of this policy change.”

its for any data scientist or data analyst who works with non-data people who still want trustworthy answers from their data this could be … including where correctness really matters and also cases where you might run really complex code

Clinical trials agent

It’s (still) very bad to be wrong

this was actually a prototype title for like three versions of this talk ago ago… it’s now incredibly easy to ask an agent a question about your data and get a wrong answer that looks completely plausible

the problem is not only that the answer is wrong. we may also be unable to tell how it was calculated or reproduce it

and if these answers are fast and convenient enough, we may start to tolerate that loss of correctness and trust as the price of using AI

but Posit has always built tools to support correct analysis. we do not think that standard should change just because the interface has changed

and it really still is bad to be wrong! correct answers, and correct answers whose methodology we can see and reproduce, are still important

AI has expanded who can get answers from data.

That changes the infrastructure we need for correct, transparent, and reproducible analysis.

and this is the part we find really motivating

AI has expanded who can ask questions about data and get answers directly. that changes the infrastructure we need around those answers

with Commons, a data team establishes which code and calculations it trusts and provides the organizational context the agent needs

the team can then give that agent to someone who might otherwise do a vibe analysis. that person keeps the convenience of asking questions in natural language, but the answers are grounded in the data team’s trusted work and can be traced and reproduced

and that is where all of you data people come in. this infrastructure has to be built by people who understand the data, the calculations, and the organization

Commons is the package that helps you build it and make it available to others

Looking forward

databot 2025 we were pushing the frontier

Canvas

Learn more at Joe Cheng’s talk: Lanier Ballroom A&F, Tuesday at 3:20

• We have been experimenting with something called canvas
• Canvas is a next generation agent experience
• Assumes that you won’t look at the code usually
• frees up real estate / degrees of freedom to other expansive and new agent collaboration interfaces
• Feels aggressive in the way that Databot felt to us a year ago
• If you want to learn more, see Joe’s talk

Thank you!

Slides pos.it/keynote Commons pos.it/commons AI Newsletter pos.it/ai-newsletter

we will be in the lounge