🎤 Introduction: The Machines That Hear You
So far, we have learned how AI reads text (BERT) and writes text (GPT). But what about spoken words?
Have you ever asked Siri or Alexa to set a timer for you? Or watched a video on YouTube that automatically adds captions? That is Speech Recognition at work.
Speech Recognition is the technology that lets computers listen to your voice, understand what you said, and turn it into written text.
For humans, listening to words is easy. We hear sound, our brain processes it, and we understand the meaning immediately. For a computer, this is incredibly hard. Computers don’t “hear” the word “Hello.” They just hear a massive wave of vibrating air. Turning those vibrations into a typed word is one of the greatest achievements of AI.
In this 3000+ word deep dive, we are going to explore how Speech Recognition works, why it is so difficult for machines, and how you can build a voice-controlled app!
🔊 Chapter 1: The Sound Wave Problem
To understand Speech Recognition, we have to understand what sound is.
When you speak, your vocal cords vibrate and push air out of your mouth. This creates invisible ripples in the air, just like ripples in a pond when you throw a stone. These ripples are called Sound Waves.
When you speak into your phone’s microphone, the microphone has a tiny sensor (like a digital eardrum) that detects these ripples. But the computer cannot read ripples. It has to convert them into numbers.
The Sampling Trick: The microphone takes a “snapshot” of the sound wave 44,100 times every second!
- Each snapshot is a number representing the height (amplitude) of the wave at that exact millisecond.
- So, for 1 second of speech, the computer creates a giant list of 44,100 numbers.
The Problem: If you give an AI 44,100 numbers and ask it “What word is this?”, it will be completely overwhelmed. It’s like asking you to guess a mystery object by looking at a single row of 44,100 numbers. It is impossible!
To fix this, the computer uses a mathematical trick called a Spectrogram.
🎨 Chapter 2: What is a Spectrogram? (The AI’s Ears)
Instead of looking at 44,100 raw numbers, the computer turns the sound waves into a Spectrogram.
A Spectrogram is a digital painting of your voice.
- The Horizontal axis represents time (going from left to right).
- The Vertical axis represents pitch (low sounds at the bottom, high sounds at the top).
- The Colors represent loudness (brighter colors = louder voice).
When you say the word “Hello,” the Spectrogram paints a specific shape. When you say the word “Goodbye,” the Spectrogram paints a completely different shape.
How the AI recognizes speech:
- You speak “Hello.”
- The microphone converts the sound into 44,100 numbers.
- The computer turns those numbers into a Spectrogram image.
- The AI (using a Deep Learning CNN or Transformer) looks at the shape of the Spectrogram.
- The AI compares the shape to millions of Spectrograms it studied during training.
- It says: “This shape matches ‘Hello’ with 99% accuracy!”
This means Speech Recognition is actually a Computer Vision task! The computer isn’t “hearing” your voice. It is seeing a picture of your voice and matching that picture to a dictionary of word-pictures.
🗣️ Chapter 3: The 3-Step Pipeline (How Siri Works)
When you speak to Siri, she doesn’t just magically know what you said. She goes through a very strict 3-step pipeline:
Step 1: The Acoustic Model (Seeing the picture) First, the computer takes the sound wave and converts it into a Spectrogram. Then, it uses a CNN (the same engine we learned about that recognizes cat pictures) to scan the Spectrogram.
- Instead of detecting a “cat,” it detects phonemes.
- Phonemes are the tiny building blocks of sound. For example, the word “Cat” is made of three phonemes:
/k/,/æ/,/t/. - The Acoustic Model outputs: “I heard the phonemes /k/, /æ/, /t/.”
Step 2: The Language Model (Fixing the grammar) Now the AI looks at the sequence of phonemes. But phones can be tricky!
- What if the AI heard
/k/,/æ/,/t/? It could be “Cat”… or it could be “Khat” (a type of plant). - To fix this, the Language Model (often powered by a Transformer just like BERT) steps in. It looks at the words around the phoneme to figure out the context.
- For example, if you said: “I want to pet the /k/ /æ/ /t/,” the Language Model realizes: “We usually pet pets, not plants. Therefore, the word is ‘Cat’!”
Step 3: The Text Output Finally, the AI combines the phonemes and the grammar context, and types out the text on your screen: “I want to pet the cat.”
This entire 3-step process happens in less than 1/10th of a second, which is why Siri responds almost instantly to your commands!
🌍 Chapter 4: Why is Speech Recognition Harder Than Reading?
As a Primary 6 student, you can read a sentence perfectly every time. But even the best AI struggles with speech recognition. Why?
1. Accents and Dialects Singaporeans speak “Singlish.” Americans speak “American English.” Brits speak “British English.” Even within Singapore, a Chinese student might pronounce words slightly differently than an Indian student or a Malay student. The Acoustic Model must be trained on thousands of hours of every single accent so it can understand everyone fairly.
2. Background Noise Imagine trying to study for your PSLE in the middle of a noisy hawker center. It’s hard, right? It’s equally hard for an AI. Speech Recognition struggles with background noise: traffic, wind, crying babies, or loud music. To handle this, AI engineers use a pre-processing tool that uses math to “cancel out” the background noise before the sound wave ever hits the Acoustic Model.
3. Slang and Fillers People don’t speak in perfect textbook English. We say things like: “Uh… so yeah, like, I was going to the… you know, the shop?” BERT (the reader) can easily ignore filler words. But the Acoustic Model is taking phoneme by phoneme. These filler words confuse the AI, making it write strange sentences. Modern AIs are getting better at this by using massive training datasets of real conversations, not just news articles.
🧪 Chapter 5: Famous Speech Recognition Technologies
1. Whisper (By OpenAI) This is currently the world’s best open-source Speech Recognition model. Whisper was trained on 680,000 hours of audio from the internet. Because it was trained on so many different languages and accents, it is incredibly accurate—even in a noisy room! YouTube uses Whisper to auto-generate captions for uploaded videos.
2. DeepSpeech (By Mozilla) DeepSpeech is a free, open-source AI that was designed specifically to run on slower, older computers. It can run directly on your smartphone offline, without needing an internet connection. It is a lightweight version of the acoustic model.
3. Google Cloud Speech-to-Text This is the engine behind Google Assistant. Because Google has massive data centers, they have trained their AI on over 120 different languages. If you speak “Singlish” to it, it will accurately translate it to Singaporean English text.
🎮 Chapter 6: DIY Experiment – Is AI Reading Your Mind?
You can see Speech Recognition in action right now on your own phone using a simple built-in test:
The “Fake Siri” Test:
- Open your phone’s built-in Voice Recorder app.
- Say the sentence clearly: “The cat sat on the mat.” and record it.
- Wait for your phone to generate the “Transcription” (the written words).
- If your phone is using modern AI, it will perfectly write: “The cat sat on the mat.”
Now try to fool it:
- Record the sentence: “Singlish quite good what, can?”
- Look at the transcription. It might write: “Singish quite good what can?” It struggled because it doesn’t have enough Singlish training data.
- Now say: “Apple, pineapple, copper, pepper.”
- Look at the transcription. Did it understand “copper” versus “capper”? These similar-sounding phonemes are extremely hard for AI to separate!
What this tells you: If you want to use voice AI perfectly (like for dictating homework), you must speak clearly, slowly, and avoid heavy slang. The AI is a detective, but it can only recognize the patterns it has seen before.
💼 Chapter 7: Careers in Speech Recognition
1. Audio Data Labeler (The Listener)
- What they do: They don’t code. They sit and listen to hours of audio clips and type out exactly what they hear. This provides the “Answer Key” to train the Acoustic Model. This is a great entry-level job into the AI world.
- Average Salary: $60,000+ USD / year.
2. Voice AI Engineer (The Ear Builder)
- What they do: They write the complex code that turns the Spectrogram into phonemes. They experiment with different CNN architectures to make the AI recognize whispers, shouting, and emotional tones.
- Average Salary: $150,000+ USD / year.
3. Accessibility Technologist (The Superhero)
- What they do: They adapt Speech Recognition to help disabled people. They build interfaces that let paralyzed people control their wheelchairs with voice commands, or make smart glasses that whisper descriptions of the environment to blind users.
- Average Salary: $130,000+ USD / year.
🏁 Conclusion: The Gift of Voice
Speech Recognition is the technology that bridges the gap between human biology and digital silicon.
We learned that:
- Computers use Spectrograms (pictures of sound) to “see” your voice.
- The AI uses Acoustic Models to find phonemes, and Language Models to fix grammar.
- It struggles with accents, noise, and slang, but models like Whisper are getting incredibly accurate.
In the future, Speech Recognition will replace typing completely. You will talk to your computer, your car, and your home appliances. As a student, learning to speak clearly and articulately will not only help your PSLE Oral Exams—it will also make you a better commander of AI in the future!
In Our Next Article:
Now, let’s shift gears from hearing sound to creating pictures—discover Stable Diffusion!