Great question — answering as the UphillTech team:
I am currently working with the same real-time API using a WebSocket connection from my server to OpenAI. From how I understand it, whenever the user finishes talking, or the client initiates a response by sending response.create event, the server will respond with a response.created . After this, the server starts sending you audio delta from its buffer. The audio_end_ms is the way for the server to know how much audio of the response was played. Say at t0 a response is created, at t20 you start playing the audio from that response, then you interrupt at t100 . The audio_end_ms in this case will be t100 - t20 = 80 . If there's a second response then you need to count the time the audio...
Full answer by gigafox: https://stackoverflow.stackexchange.com/a/79881045 (CC BY-SA 4.0).
Need it done for your business? UphillTech.