AI and emotion in voice: measuring expression, clarifying needs

A voice assistant that slows down when asked, explains a difficult step again and leaves room for a reply can feel attentive. That experience does not prove that the machine understands someone's inner life. For a voice product, a useful starting point is giving people control over the conversation.

🎙️ 1. What is being measured?

Hume distinguishes the expression perceived by listeners from the speaker's actual feelings. Its documentation also explains that expression scores are independent and do not add up to one. A score should therefore not become a claim that someone is “80% sad.” Hume's explanation of scores.

Consider a caller saying, “Yes, I understand.” They might have solved the problem, or they might simply want to end the call. A confirmation question is more useful than presenting an emotional label as a fact.

🧭 2. Separate evidence from decisions

Divide the product design into input, hypothesis and action. The input might be the caller's latest request. The hypothesis might be that a shorter explanation would help. The action is to offer a summary or a step-by-step version.

Avoid a field such as customer_is_angry when all you have is an uncertain inference. Store a confirmed preference instead, such as preferred_response_length: short. Clear field names help a team distinguish what it knows from what it assumes.

🎛️ 3. Start with a small task

Imagine an assistant that helps change a delivery address. Write three fictional scenarios: a caller is in a hurry, a caller needs one step explained, and a caller wants a human agent. Let testers state their needs directly. Do not ask the system to infer personality or mental health from their voices.

The prototype needs controls for pace, repetition and switching to text. Establish whether these controls improve task completion before adding expression analysis.

📝 A brief for the product team

Design a conversation for changing a delivery address.
Use fictional data, without recording real customers.
Ask about one thing per step.
Offer: short answers / step by step / human support.
Ask for confirmation when the user's needs are unclear.
Do not label emotions or diagnose from voice.
Return three scenarios and completion criteria for each.

✅ 4. Measure useful outcomes

Count successful address changes, requests for repetition and irrelevant questions. At the end, ask testers whether the guidance suited them and invite an explanation.

Compare two versions with the same information: one fixed version and one with user-controlled pacing. Do not treat the model's own emotion scores as the sole evidence of its usefulness. Task outcomes and direct feedback provide more inspectable measures.

🔒 5. Make analysis a choice

If an experiment genuinely requires audio analysis, explain what is transmitted, why it is needed and how to disable it. Retain only what is necessary for the agreed evaluation, and avoid sending recordings to unexpected services.

In this design, disabling analysis should still allow the basic task to be completed. This also tests whether the extra feature adds value or simply adds complexity.

💡 6. Build an assistant that asks

A small improvement is enough to begin: after a long explanation, ask, “Would you prefer a summary or one step at a time?” That is a specific behavior you can test.

Voice AI should make conversations clearer. Respect shows up in following requests, accepting corrections and adapting to preferences the person has actually expressed.