NLP: Teaching Machines to Understand Human Language

NLP · beginner

Featured image for NLP: Teaching Machines to Understand Human Language

🗣️ Introduction: The Language Bridge

Throughout this entire course, we have traveled through the vast universe of AI. We’ve seen AI drive cars, paint pictures, trade stocks, and even become a personal tutor.

But the backbone of almost every application we studied—from ChatGPT writing essays to AI catching phishing emails—is one specific technology: NLP.

NLP stands for Natural Language Processing.

Natural Language is just a fancy term for “Human Language”—the English, Chinese, and Malay we use to talk to each other. “Processing” means the computer is analyzing, understanding, and generating that language.

NLP is the field of AI that builds a bridge between human words and computer code. Before NLP, computers could only read numbers. With NLP, computers can read your words, understand your intent, and reply in a way that sounds perfectly human.

In this 3000+ word deep dive, we will go back to the absolute foundations of NLP. We will learn how machines break down sentences, how they understand emotions, and why this is the most important field of AI for the future of human-computer interaction.

🔤 Chapter 1: Tokenization – Breaking Down the Sentence

How does a computer read a sentence? It doesn’t see the word “Apple” as a picture of a fruit. It sees a string of characters: A, p, p, l, e.

The first step of NLP is to break this string into manageable pieces called Tokens.

What is a Token?

A token is a unit of text. It is usually a word, but it can also be a sub-word (like un and believe in “unbelieveable”) or even a character.

Why do we need Tokenization?

Imagine you are building an AI translator. You feed it the sentence: “The cat sat on the mat.”

  1. The Tokenizer reads the sentence.
  2. It splits it into tokens: ["The", "cat", "sat", "on", "the", "mat"].
  3. It converts these tokens into numbers (IDs) based on a giant dictionary. The = 1, cat = 235, sat = 648, and so on.
  4. The AI can now do math on these numbers to understand the sentence.

The Sub-word Trick

What if the AI sees a word it has never seen before, like supercalifragilisticexpialidocious?

If it only stores whole words, it will get an error. But modern Tokenizers break it down into sub-words: ["super", "cali", "fragilistic", "expiali", "docious"].

Even if it has never seen the full word, it recognizes the pieces. This is how modern LLMs (like GPT) can understand almost any word you throw at them.

🧩 Chapter 2: Word Embeddings – The Math of Meaning

Once the AI has the tokens (numbers), it needs to understand that Apple and Orange are similar, but Apple and Computer are not.

This is done using Word Embeddings.

The Vector Space Analogy

Imagine a giant 3D room. You place every word in the English language somewhere in this room.

  • You place “Apple” near “Orange” because they are both fruits.
  • You place “Apple” near “MacBook” because Apple makes computers.
  • You place “Cat” near “Dog” because they are both pets.

How does the computer place them?

An algorithm called Word2Vec (Word to Vector) scans millions of sentences. It notices that “Apple” and “Orange” often appear in the same sentences: “I ate an apple,” “I ate an orange.”

It calculates the math distance between words. If two words show up in the same context, they are moved closer together in the 3D room.

The Magic of Math

If you take the vector for King, subtract Man, and add Woman, you get Queen!

  • The math literally connects the concepts. This proves that the AI has learned the meaning of words, not just the spelling. This semantic understanding is what makes AI capable of translating languages and answering complex questions.

🎭 Chapter 3: Part-of-Speech Tagging – The Grammar Detective

To understand a sentence, the AI must understand grammar.

The sentence “I fish in the river” has a totally different grammar than “I bought a fish”.

The AI Grammar Checker (POS Tagging)

POS (Part-of-Speech) Tagging is an NLP technique that labels every word in a sentence with its grammatical role:

  • N = Noun
  • V = Verb
  • Adj = Adjective
  • Prep = Preposition

How it works

The AI looks at the sentence: “TheAdj cuteAdj catN satV onPrep theDet matN.”

  • This tells the AI that the subject is the “cute cat” and the action is “sat.”

Why it matters

When you ask Siri, “What is the weather in Singapore?” the POS tagger helps Siri understand that “weather” is the object you are asking about, and “Singapore” is the location. Without POS tagging, Siri would just see a random jumble of words and fail to answer.

📊 Chapter 4: Named Entity Recognition – Extracting the Facts

When an AI reads a news article, it doesn’t just read it for fun. It needs to extract the important facts. This is called NER (Named Entity Recognition).

What is a Named Entity?

A Named Entity is a specific real-world object:

  • People (e.g., “Mr. Lee”, “Elon Musk”)
  • Places (e.g., “Singapore”, “New York”)
  • Organizations (e.g., “Apple Inc.”, “Google”)
  • Dates and Times (e.g., “January 1st”, “2024”)

How NER works

The AI scans the sentence: “Elon MuskPER announced that TeslaORG will open a new factory in** AustinLOC on** July 1stDATE.”

  • The AI tags Elon Musk as a Person (PER).
  • It tags Tesla as an Organization (ORG).
  • It tags Austin as a Location (LOC).
  • It tags July 1st as a Date (DATE).

Why it matters

Google News uses NER to group articles. If you search for “Tesla,” the NER algorithm scans millions of articles, finds every instance of the Organization “Tesla,” and compiles them into a single news feed for you. It automatically categorizes the chaos of the internet!

🧠 Chapter 5: Sentiment Analysis – The Emotion Detector

How does a company know if you love or hate their product? They use Sentiment Analysis.

The Process

  1. A customer writes a review: “The service was incredibly slow, and the staff were rude.”
  2. The NLP AI scans the sentence. It looks for keywords: slow (negative), rude (negative).
  3. The AI calculates a “Sentiment Score” between -1.0 (very negative) and +1.0 (very positive).
  4. The AI flags this review as Negative (-0.9).
  5. The company’s Customer Service bot instantly offers an apology and a refund.

Advanced Sentiment (Sarcasm Detection)

Sarcasm is the hardest thing for an AI to understand. If a user writes: “Oh wow, the service was AMAZING. I waited 3 hours for a sandwich.”

  • A normal NLP model sees “AMAZING” and might think it’s positive.
  • But advanced Transformers (like BERT) look at the context. They read the second half of the sentence: “I waited 3 hours for a sandwich.”
  • The AI realizes: “This sentence is highly positive in the first half, but highly negative in the second half. This is sarcasm!”
  • It correctly classifies the review as negative.

🌐 Chapter 6: Text Summarization – The Ultimate Shortcut

Imagine reading a 500-page novel for a book report. That takes days. What if an AI could read the 500 pages and condense it into 5 bullet points in 2 seconds?

Extractive vs. Abstractive Summarization

  • Extractive: The AI highlights the most important sentences in the book and copies them exactly. (It’s like highlighting a textbook with a yellow marker).
  • Abstractive: The AI reads the whole book, understands the plot, and writes a completely new summary in its own words. (This is much harder, and requires LLMs like GPT).

How it benefits you

AI summarization tools are used in schools to help students study for exams. You can take a 50-page textbook chapter, scan it into an AI, and ask it to: “Summarize this into 5 key points.” It gives you the exact information you need to study, without the fluff.

🏗️ Chapter 7: The Evolution of NLP (From Rules to Deep Learning)

NLP didn’t start with Transformers. It has evolved over 70 years:

1. Rule-Based NLP (1960s – 1980s)

Scientists wrote strict grammatical rules by hand. They said: “If you see ‘the’ before a word, that word is a Noun.”

Result: It worked for basic English, but failed with slang, typos, and complex sentences.

2. Statistical NLP (1990s – 2010s)

AI started looking at the frequency of words. It calculated: “The word ‘apple’ is often next to ‘fruit’. So ‘apple’ is a fruit.”

Result: It was better than rule-based, but still couldn’t understand long paragraphs.

3. Deep Learning / Transformers (2017 – Today)

The invention of the Transformer and Attention completely revolutionized NLP. We learned about this in the Transformers article. The AI now looks at every word in a sentence simultaneously, understanding long-range context perfectly.

Result: GPT, BERT, and Gemini. We have reached the peak of language understanding.

💼 Chapter 8: Careers in NLP

1. NLP Engineer (The Language Coder)

  • What they do: They build the pipelines for tokenization, NER, and Sentiment Analysis. They connect Hugging Face’s BERT models to customer service chatbots. They tune the parameters to make the AI understand specific jargon (e.g., legal terms, medical terms).
  • Average Salary: $150,000+ USD / year.

2. Computational Linguist (The Grammar Genius)

  • What they do: They are half-linguist, half-programmer. They design the algorithms for POS Tagging and parsing complex grammar rules. They are the ones who teach the AI the difference between “The dog bit the man” and “The man bit the dog.”
  • Average Salary: $140,000+ USD / year.

3. Voice User Interface (VUI) Designer

  • What they do: They design the conversational flow for Siri and Alexa. They don’t write code; they write script templates that the AI follows. They ask: “If a user says ‘Turn on the lights,’ what should the AI respond with?”
  • Average Salary: $130,000+ USD / year.

🧪 Chapter 9: Experiment – Tokenization in Python

You can easily see how an NLP model breaks down words using Python’s nltk (Natural Language Toolkit) library.

The “Sentence Splitter” Test

  1. Install the library: pip install nltk
  2. Create a Python file nlp_demo.py.
  3. Paste the following code:
import nltk
nltk.download('punkt')

from nltk.tokenize import word_tokenize, sent_tokenize

text = "Hello, Mr. Lee! How are you today? The PSLE English exam is coming up."

# 1. Sentence Tokenization (Break into sentences)
sentences = sent_tokenize(text)
print("Sentences:", sentences)

# 2. Word Tokenization (Break into words/tokens)
words = word_tokenize(text)
print("Words/Tokens:", words)

🏁 Conclusion: The Language Revolution

NLP has transformed how we interact with computers. From chatbots to translation, it’s changing everything.

We’ve Learned

  • Tokenization breaks text into pieces for AI to process

  • Word Embeddings capture meaning through math

  • POS Tagging helps AI understand grammar

  • Named Entity Recognition extracts facts from text

  • Sentiment Analysis detects emotions in writing

  • Text Summarization condenses information

  • NLP has evolved from rules to deep learning

What This Means for You

Understanding NLP helps you:

  • Use AI assistants more effectively

  • Understand how ChatGPT works

  • Communicate better with AI

In Our Next Article:

Now that you understand NLP, it’s time to explore BERT: Google’s Search Engine Genius** in depth! If Google Search were a student, BERT would be the teacher’s pet who reads the question from both ends – so when you type “I’m hungry,” it knows you want pizza, not a math test!

🌳 Back to AI Family Tree