What is actually sent to the LLM when invoke is called on messages?

handyman_hank

An AIMessage has "content" and "tool_calls." Is "tool_calls" also sent to the LLM in future calls? The reason I'm asking this is because I want to trim messages down before calling the LLM. In particular, I'm wary of a write_file tool call, which may potentially have a really long input or argument (the content of what's to be written). And this really long input is in in the tool_calls dict, which may get sent to the LLM I did a test and it...

Originally asked by Charles Gong on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

uphilltech

Great question — answering as the UphillTech team:

tool_calls do get sent to the LLM on subsequent calls. I verified this the same way you did: by watching the input_tokens count jump when I added messages with tool calls vs without them. The tool_use block behaviour you noticed is actually LangChain being smart about avoiding duplication. When there's a tool_use block in the content list, it doesn't send those inputs again because they're already in the tool_calls field. It makes sense once you realize it, but it's kind of confusing at first! For your write_file use case with potentially huge content, this can definitely balloon your token usage. You're essentially sending that entire file content back to the LLM on every subsequent call,...

Full answer by Amira: https://stackoverflow.stackexchange.com/a/79913915 (CC BY-SA 4.0).

Need it done for your business? UphillTech.