Anatomy of a conversation

LLMs for Data Analysis in R

R/Medicine 2026

Anatomy of a Conversation

first, let’s talk about how talking to llms works

you’ve probably talked with chatgpt or claude or gemini and seen a conversation that looks something like this:

You ask a question and ChatGPT responds. You can keep talking to ChatGPT and it keeps responding, and it feels kind of like you’re having a conversation

we can do this same thing in r

We can do this from R!

we can do this same back and forth from R with a package called ellmer

library(ellmer)

chat <- chat("anthropic")

chat$chat("Tell me a quick fact about sheep.")

lets take a look at the code at a high level and then we’ll go into the details

in ellmer, you set up a chat – a connection to an LLM

and then once you have that connection, you can send requests to the model

and it will send you a response back

library(ellmer)

chat <- chat("anthropic")

chat$chat("Tell me a quick fact about sheep.")
#> Sheep have rectangular pupils that give them 
#> a nearly 360-degree field of vision, 
#> allowing them to see predators approaching 
#> from almost any direction without turning 
#> their heads.

How does this work?

Most LLMs are accessible through HTTP APIs

it works because most llms are accessible through http apis

this is pretty much just a normal api. if you’ve ever queried an api to get data, this is fundmentally the same thing, and we’re going to use these apis to build tools from r that use llms

lets take a look at the code

First step - load the ellmer package, just like any other R package.

library(ellmer)

Now we create a chat object. This sets up a connection to (here) anthropic’s API. The chat object is what we’ll use to send messages and get responses.

library(ellmer)

chat <- chat("anthropic")

Now we can use the chat object to send a message using the chat method

and pass in our message as a string.

library(ellmer)

chat <- chat("anthropic")

chat$chat("Tell me a quick fact about sheep.")

And we get a response back!

i have fact checked this fact

library(ellmer)

chat <- chat("anthropic")

chat$chat("Tell me a quick fact about sheep.")
#> Sheep have rectangular pupils that give them 
#> a nearly 360-degree field of vision, 
#> allowing them to see predators approaching 
#> from almost any direction without turning 
#> their heads.

One of the nice things about ellmer is that you can easily inspect the chat object to see the full conversation history.

If we just call ‘chat’, we can see all the messages in the conversation so far click see basics of convo, tokens used and cost (tokens are units of llms–will talk about later)

You can see the user message (what we sent) marked with user click and the assistant response click

click

chat

<Chat Anthropic/claude-sonnet-4-5 turns=2 input=15 output=37 cost=$0.00>
── user ─────────────────────────────────────────────────────────────────────────────────
Tell me a quick fact about sheep.
── assistant [input=15 output=37 cost=$0.00] ────────────────────────────────────────────
Sheep have rectangular pupils that give them a nearly 360-degree field of vision, allowing them to see predators approaching from almost any direction without turning their heads.

Messages have roles.

and these messages are marked with user and assistant because each message in the conversation is assigned a role.

let look into this a bit more because role are important

When you send a message to an LLM, you’re not just sending text - you’re sending text with a specific ROLE attached to it.

There are three main roles that messages can have, and understanding these roles is key to working with LLMs programmatically.

[As the diagrams appear] - USER messages - what the person using your app types - And ASSISTANT messages - what the LLM responds with

This might seem like a small detail, but these roles are fundamental to how you control and guide LLM behavior.

there’s also the SYSTEM PROMPT - This is your instruction manual for the LLM. It’s where YOU as the developer tell the model how to behave, what personality to have, what constraints to follow. The user typically never sees this. think about how you would never really come across the chatgpt system prompt. The system prompt is particularly powerful because it persists across the entire conversation and shapes how the assistant responds to every user message.

When you send a message to an LLM, you’re not just sending text - you’re sending text with a specific ROLE attached to it.

There are three main roles that messages can have, and understanding these roles is key to working with LLMs programmatically.

[As the diagrams appear] - USER messages - what the person using your app types - And ASSISTANT messages - what the LLM responds with

This might seem like a small detail, but these roles are fundamental to how you control and guide LLM behavior.

there’s also the SYSTEM PROMPT - This is your instruction manual for the LLM. It’s where YOU as the developer tell the model how to behave, what personality to have, what constraints to follow. The user typically never sees this. think about how you would never really come across the chatgpt system prompt. The system prompt is particularly powerful because it persists across the entire conversation and shapes how the assistant responds to every user message.

When you send a message to an LLM, you’re not just sending text - you’re sending text with a specific ROLE attached to it.

There are three main roles that messages can have, and understanding these roles is key to working with LLMs programmatically.

[As the diagrams appear] - USER messages - what the person using your app types - And ASSISTANT messages - what the LLM responds with

This might seem like a small detail, but these roles are fundamental to how you control and guide LLM behavior.

there’s also the SYSTEM PROMPT - This is your instruction manual for the LLM. It’s where YOU as the developer tell the model how to behave, what personality to have, what constraints to follow. The user typically never sees this. think about how you would never really come across the chatgpt system prompt. The system prompt is particularly powerful because it persists across the entire conversation and shapes how the assistant responds to every user message.

Message roles

to recap

Role	Description
system_prompt	Instructions from the developer (i.e., you) to set the behavior of the assistant
user	Messages from the person interacting with the assistant
assistant	The AI model’s responses to the user

the chat() functions includes a system_prompt argument where you can set a system prompt Let’s try adding one

This is a silly example - we’re telling the model to always answer in haikus. But it demonstrates how system prompts work and can control model behavior

library(ellmer)

chat <- chat(
  "anthropic",
  system_prompt = "Always answer in haikus."
)

chat$chat("What is chirality?")

It actually answered in a haiku.

The system prompt shapes EVERY response the model gives. this is really powerful because you can exert quite a bit of control over how the model behaves

library(ellmer)

chat <- chat(
  "anthropic",
  system_prompt = "Always answer in haikus."
)

chat$chat("What is chirality?")

Molecules can twist,
Left hand, right hand—mirror forms,
Same but different.

Now when we inspect the chat object, we can see all three roles - system, user, and assistant. The system prompt is there shaping the conversation.

chat

<Chat Anthropic/claude-sonnet-4-5 turns=3 input=17 output=23 cost=$0.00>
── system ───────────────────────────────────────────────────────────────────────────────
Always answer in haikus.
── user ─────────────────────────────────────────────────────────────────────────────────
what is chirality
── assistant [input=17 output=23 cost=$0.00] ────────────────────────────────────────────
Molecules can twist,
Left hand, right hand—mirror forms,
Same but different.

Your Turn 02_conversation

1. Set up a chat with a system prompt instructing the model to answer briefly.

2. Ask: What ellmer function tells me what Anthropic models are available?

3. Ask: What about OpenAI models?

4. Create a new chat with no system prompt and ask the second question again.

5. How do the answers to 3 and 4 differ? Think about both the content and the style.

05:00

Two takeaways: (1) the system prompt persists across chat() calls — the first chat answers briefly throughout, (2) a new chat object has no memory and no system prompt — it doesn’t know you were asking about ellmer, and it gives a longer answer.

Demo: clearbot

Demo showing what’s really happening behind the scenes - the full conversation history being sent with each request. This reveals that LLMs are stateless.

👩‍💻 _demos/01_clearbot/app.py

Is this actually a conversation?

You just saw in clearbot what’s really happening when you have a conversation with an LLM.

Those conversations look something like this - you ask, it responds, back and forth. But as clearbot showed, something interesting is happening behind the scenes.

It IS a conversation, but it works differently than you might think.

LLMs are stateless

• The LLM doesn’t remember anything between requests

• You have to send the entire conversation history with every message

• The LLM reconstructs the “conversation” from what you send

The LLM has amnesia - it forgets everything after each response. Each API request is completely independent.

Just like if you were talking to someone with amnesia, you have to recap the entire conversation every time you want to continue. That’s what clearbot showed you - the full history being sent with each request.

This isn’t a limitation of transformers specifically - it’s how the HTTP API is designed. Though it aligns with how transformers work (each forward pass is independent).

How LLMs work (briefly)

Now that you understand how conversations work with LLMs - the statelessness, sending full history - let’s talk briefly about how LLMs themselves actually work. How do they understand what you’re asking? And how do they construct their responses?

How do LLMs understand?

This is how LLMs understand the conversation. They’re trained on massive amounts of text data - essentially everything written on the internet, in books, etc. By learning patterns from all this text, they develop the ability to understand context, meaning, and how language works.

If you read everything
ever written…

• Books and stories

• Websites and articles

• Poems and jokes

• Questions and answers

…then you could…

• Answer questions
• Write stories
• Tell jokes
• Explain things
• Translate into any language

How do LLMs respond?

Now let’s talk about how LLMs construct their responses. They don’t generate whole sentences at once - they generate responses token by token, choosing the most likely next piece based on everything they’ve seen so far.

LLMs think in tokens

LLMs don’t think in words - they think in tokens. This is the fundamental unit they use to construct responses, one token at a time.

• Fundamental units of information for LLMs
• Words, parts of words, or individual characters
  • “hello” → 1 token
  • “unconventional” → 3 tokens: un|con|ventional
• Important for:
  • Model input/output limits
  • API pricing is usually by token
• Not just words, but images can be tokenized too

Demo:
token-possibilities

This demo shows how LLMs construct responses in real-time. You’ll see the model generating tokens one at a time, and you can see the probabilities for what token might come next. This is how all LLM responses are built - token by token, choosing the most likely next piece.

The breakthrough insight is the attention mechanism in the Transformer architecture. Instead of processing text sequentially (word by word), GPT models can look at all words in a sequence simultaneously and understand how they relate to each other.

The attention mechanism allows the model to:

• Weigh the importance of every word in relation to every other word
• Capture long-range dependencies that traditional models missed
• Process sequences in parallel rather than sequentially

👩‍💻 _demos/02_token-possibilities/app.R

How to think about LLMs

approach LLMs with curiosity and experimentation rather than preconceived notions about limitations, while building understanding from the ground up through practical experience.

Think Empirically, Not Theoretically

• It’s okay to (mostly) treat LLMs as black boxes.

• Just try it! When wondering if an LLM can do something,
  experiment rather than theorize

• You might think they could not possibly do things
  that they clearly can do today

• And you might think surely they can do something
  that it turns out they’re terrible at

• Understanding the technical details can lead to bad intuition about capabilities
• You might think “they could not possibly do things that they clearly can do today”
• Empirical testing reveals actual capabilities vs. theoretical limitations

Don’t expect smooth, predictable performance across tasks. LLMs can surprise you - excelling at some complex tasks while struggling with seemingly simple ones.

LLMs are jagged

Embrace the Experimental Process

will talk about ways to formally run experiments in the next section

• Explore!
  Focus on learning and engaging with the technology, not outcomes

• Failure is valuable!
  those are some of the most interesting conversations that we have

• It doesn’t have to be a success.
  Attempts that don’t work still provide insights

• Use the best models available for experimentation

• Don’t constrain yourself to “practical” applications initially

• Think of it as forming your own independent conclusions about usefulness

What if I want to keep chatting back-and-forth?

Transition to showing how to build interactive chat interfaces with shinychat.

ellmer can do that, too!

	Console	Browser
	live_console(chat)	live_browser(chat)

ellmer gives you two ways to have an interactive back-and-forth conversation: live_console() stays in the terminal, live_browser() opens a chat UI in your browser. Both use the same chat object we’ve been working with.

Demo:
live

This demo shows live_console() and live_browser() - easy ways to chat with an LLM interactively from R.

👩‍💻 _demos/03_live/03_live.R