How to get confidence for the user’s audio transcription before the... —...

spark_sara

I'm building a phone voice agent with the OpenAI Realtime API ( gpt-realtime ). On noisy calls it sometimes mishears the caller (e.g. it hears «عندك مطبخ؟» "do you have a kitchen?" when the caller said «عندك مسبح؟» "do you have a pool?") and answers the wrong question. I want to check how confident the transcription of the caller's turn was, and only let the model respond if confidence is high enough. Otherwise I'll play a fixed "could you...

Originally asked by HelloWorld on Stack Exchange (stackoverflow) (CC BY-SA 4.0).

Variant 91.

Variant 931.

uphilltech

Great question — answering as the UphillTech team:

Yeah. Add "include": ["item.input_audio_transcription.logprobs"] to the session. The logprobs then come in conversation.item.input_audio_transcription.completed (and in the .delta events) as a logprobs array of { token, logprob, bytes } . Include only exists in the GA session config, so the session has to use the GA shape type: "realtime" , with transcription and turn detection under audio.input { "type": "session.update", "session": { "type": "realtime", "include": ["item.input_audio_transcription.logprobs"], "audio": { "input": { "transcription": { "model": "gpt-4o-transcribe", "prompt": "A phone call in Arabic or English. Transcribe in the language spoken." }, "turn_detection": { "type":...

Full answer by Emmanuel Obiechina Ikenna: https://stackoverflow.stackexchange.com/a/80009719 (CC BY-SA 4.0).

Want it for your business? UphillTech.