Python Examples for AI/ML & Data Science

Regular expressions (RegEx or Regex) are an incredibly powerful and versatile tool for pattern matching and manipulation within text. In the fields of Natural Language Processing (NLP), Artificial Intelligence (AI), Machine Learning (ML), and Data Science, mastering RegEx in Python is a fundamental skill for efficient text preprocessing, data cleaning, and information extraction. This comprehensive guide provides clear Python code examples covering essential RegEx functionalities using the built-in re module, specifically tailored for students and aspiring professionals.

1. Importing the re Module

The very first step to leverage the power of regular expressions in Python is to import the re module.

import re

print("re module imported successfully!")

NLP Relevance: This is the foundational step for any text processing task involving regular expressions in Python, enabling all subsequent operations.

2. Literal Matching in RegEx

The simplest application of RegEx is to find exact matches of a literal string within a larger text.

import re

text = "The quick brown fox jumps over the lazy dog."
pattern = "fox"

# re.search() scans the entire string to find the first occurrence
match = re.search(pattern, text)

if match:
    print(f"Literal match: '{pattern}' found at index {match.start()} to {match.end()} (exclusive).")
else:
    print(f"Literal match: '{pattern}' not found.")

Output:

'fox' found at index 16 to 19 (exclusive).

NLP Relevance: Ideal for basic keyword spotting, checking for the presence of specific terms, or validating exact phrases in text data.

3. re.search() vs. re.match() in Python RegEx

Understanding the distinction between re.search() and re.match() is crucial for effective RegEx usage. They differ in where they begin their search for a pattern.

re.search(): Scan Anywhere in the String

re.search() scans the entire input string from left to right, returning the first location where the regular expression pattern produces a match.

import re

text = "My email is user@example.com, please contact me."
pattern_search = r"example\.com" # '\.' matches a literal dot, essential for domain names

# re.search() will successfully find the pattern anywhere in the string
match_search = re.search(pattern_search, text)

if match_search:
    print(f"re.search(): Found '{match_search.group()}' at index {match_search.start()}.")
else:
    print("re.search(): No match found.")

Output:

re.search(): Found 'example.com' at index 14.

NLP Relevance: Indispensable for extracting specific entities like email addresses, phone numbers, or dates that can appear at any position within a document or text block.

re.match(): Match Only at the Beginning of the String

re.match() strictly checks for a match only at the very beginning (index 0) of the string.

import re

text1 = "Hello world, this is a test."
text2 = "world, hello this is a test."

pattern_match = r"Hello"

# This will match because 'Hello' is at the beginning of text1
match_match1 = re.match(pattern_match, text1)
print(f"re.match() on '{text1}': {match_match1.group() if match_match1 else 'No Match'}")

# This will NOT match because 'Hello' is not at the beginning of text2
match_match2 = re.match(pattern_match, text2)
print(f"re.match() on '{text2}': {match_match2.group() if match_match2 else 'No Match'}")

Output:

re.match() on 'Hello world, this is a test.': Hello
re.match() on 'world, hello this is a test.': No Match

NLP Relevance: Useful for validating string formats that must start with a specific sequence, such as ensuring a file name or a record ID adheres to a predefined prefix.

4. RegEx Quantifiers: Defining Repetitions

Quantifiers are special characters that specify how many occurrences of the preceding character or group should be matched. They are fundamental for flexible pattern definition.

*: Zero or More Occurrences

Matches zero or more repetitions of the preceding element.

import re

text = "abbc, ac, abbbbc"
pattern = r"ab*c" # Matches 'a' followed by zero or more 'b's, then 'c'

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern 'ab*c' matches: ['abbc', 'ac', 'abbbbc']

NLP Relevance: Handling variations in word spellings (e.g., theatre vs theater) or optional characters in a pattern.

+: One or More Occurrences

Matches one or more repetitions of the preceding element.

import re

text = "abbc, ac, abbbbc"
pattern = r"ab+c" # Matches 'a' followed by one or more 'b's, then 'c'

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern 'ab+c' matches: ['abbc', 'abbbbc']

NLP Relevance: Ensuring a character or sequence appears at least once, commonly used with character classes like \d+ (one or more digits) for numbers.

?: Zero or One Occurrence (Optional)

Matches zero or one repetition of the preceding element, effectively making it optional.

import re

text = "The color is red, while the colour is blue."
pattern = r"colou?r" # Matches 'colo' followed by an optional 'u', then 'r'

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern 'colou?r' matches: ['color', 'colour']

NLP Relevance: Excellent for handling common spelling variations (e.g., American vs. British English, plural vs. singular forms).

{n}: Exactly n Times

Matches exactly n occurrences of the preceding element.

import re

text = "aaa, aa, aaaa, a"
pattern = r"a{3}" # Matches exactly three 'a's

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern 'a{3}' matches: ['aaa', 'aaa']

NLP Relevance: Matching fixed-length codes, IDs, or specific numerical sequences (e.g., a three-digit area code).

{n,m}: Between n and m Times

Matches between n and m (inclusive) occurrences of the preceding element.

import re

text = "aa, aaa, aaaa, aaaaa"
pattern = r"a{2,4}" # Matches between two and four 'a's

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern 'a{2,4}' matches: ['aa', 'aaa', 'aaaa', 'aaaa']

NLP Relevance: Provides flexible matching for varying lengths of tokens or sequences, useful in parsing structured data with variable-length fields.

5. Essential RegEx Metacharacters for Text Processing

Metacharacters are special symbols that have a predefined meaning in regular expressions, allowing for powerful and concise pattern definitions.

.: Any Character (Except Newline)

Matches any single character, except for a newline character (\n).

import re

text = "cat, cot, cut, c@t"
pattern = r"c.t" # Matches 'c' followed by any character, then 't'

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern 'c.t' matches: ['cat', 'cot', 'cut', 'c@t']

NLP Relevance: Useful for wildcard matching, fuzzy searches, or when a single character can vary within a pattern.

\w: Word Character (Alphanumeric + Underscore)

Matches any alphanumeric character (letters a-z, A-Z, digits 0-9) or the underscore (_).

import re

text = "Hello_World123! How are you?"
pattern = r"\w+" # Matches one or more word characters

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern '\w+' matches: ['Hello_World123', 'How', 'are', 'you']

NLP Relevance: A cornerstone for basic tokenization (splitting text into words) and extracting meaningful word-like units from a sentence.

\d: Digit (0-9)

Matches any decimal digit from 0 to 9.

import re

text = "My phone number is 123-456-7890. My PIN is 4321."
pattern = r"\d{3}-\d{3}-\d{4}" # Matches a common phone number format

match = re.search(pattern, text)
print(f"Pattern '{pattern}' matches: {match.group() if match else 'No Match'}")

Output:

Pattern '\d{3}-\d{3}-\d{4}' matches: 123-456-7890

NLP Relevance: Essential for extracting numerical data such as phone numbers, zip codes, dates, or prices from text.

\s: Whitespace Character

Matches any whitespace character, including spaces, tabs (\t), newlines (\n), carriage returns (\r), form feeds (\f), and vertical tabs (\v).

import re

text = "This   text has\tmultiple\nspaces."
pattern = r"\s+" # Matches one or more whitespace characters

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern '\s+' matches: ['   ', '\t', '\n']

NLP Relevance: Crucial for text cleaning and normalization, such as collapsing multiple spaces into single spaces or removing leading/trailing whitespace.

^: Start of String/Line Anchor

Matches the beginning of the string. When the re.MULTILINE flag is used, it also matches the beginning of each line.

import re

text = "First line\nSecond line\nThird line"
pattern_start = r"^Second"

# Without re.MULTILINE, it won't match "Second" as it's not at the string's start
match1 = re.search(pattern_start, text)
print(f"Match for '^Second' (no MULTILINE): {match1.group() if match1 else 'No Match'}")

# With re.MULTILINE, it will match "Second" at the start of its line
match2 = re.search(pattern_start, text, re.MULTILINE)
print(f"Match for '^Second' (with MULTILINE): {match2.group() if match2 else 'No Match'}")

Output:

Match for '^Second' (no MULTILINE): No Match
Match for '^Second' (with MULTILINE): Second

NLP Relevance: Useful for identifying specific patterns that must appear at the beginning of a document, a paragraph, or a line, often used in log parsing or structured text analysis.

$: End of String/Line Anchor

Matches the end of the string. With the re.MULTILINE flag, it also matches the end of each line.

import re

text = "First line.\nSecond line.\nThird line."
pattern_end = r"line\.$" # Matches 'line.' at the end of the string/line

# Without re.MULTILINE, it matches only the end of the entire string
match1 = re.search(pattern_end, text)
print(f"Match for 'line.$' (no MULTILINE): {match1.group() if match1 else 'No Match'}")

# With re.MULTILINE, it matches "line." at the end of each line
matches2 = re.findall(pattern_end, text, re.MULTILINE)
print(f"Matches for 'line.$' (with MULTILINE): {matches2}")

Output:

Match for 'line.$' (no MULTILINE): line.
Matches for 'line.$' (with MULTILINE): ['line.', 'line.', 'line.']

NLP Relevance: Identifying patterns that must appear at the end of a document or a line, useful for validating sentence endings or specific data formats.

6. Character Sets ([]) in RegEx

Character sets, defined by square brackets [], allow you to match any one character from a specified set.

Basic Character Set

Matches any single character that is explicitly listed inside the brackets.

import re

text = "gray, grey, gr@y"
pattern = r"gr[ae]y" # Matches 'gr' followed by either 'a' or 'e', then 'y'

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern 'gr[ae]y' matches: ['gray', 'grey']

NLP Relevance: Excellent for handling common variations in spelling (e.g., colour/color, analyse/analyze) or matching specific sets of characters.

Character Ranges

Specify a range of characters using a hyphen within the brackets (e.g., a-z for all lowercase letters, 0-9 for all digits).

import re

text = "abc123xyz789"
pattern_letters = r"[a-z]+" # Matches one or more lowercase letters
pattern_digits = r"[0-9]+"  # Matches one or more digits

matches_letters = re.findall(pattern_letters, text)
matches_digits = re.findall(pattern_digits, text)

print(f"Lowercase letters: {matches_letters}")
print(f"Digits: {matches_digits}")

Output:

Lowercase letters: ['abc', 'xyz']
Digits: ['123', '789']

NLP Relevance: Extracting specific types of characters, like all words, all numbers, or all uppercase letters, crucial for text normalization and feature engineering.

Negated Character Set ([^...])

Matches any character not present within the brackets. The ^ immediately after the opening bracket negates the set.

import re

text = "This text has 123 numbers and !@# symbols."
pattern = r"[^a-zA-Z0-9\s]+" # Matches one or more characters that are NOT letters, digits, or whitespace

matches = re.findall(pattern, text)
print(f"Non-alphanumeric, non-whitespace characters: {matches}")

Output:

Non-alphanumeric, non-whitespace characters: ['!', '@', '#', '.']

NLP Relevance: Widely used for cleaning text by efficiently removing punctuation, special characters, or any unwanted symbols that don’t contribute to semantic meaning.

7. Alternation (|) in RegEx

The | (OR) operator allows you to match either the expression before it or the expression after it.

import re

text = "I have a cat and a dog. Also a bird."
pattern = r"cat|dog|bird" # Matches 'cat' OR 'dog' OR 'bird'

matches = re.findall(pattern, text)
print(f"Animals found: {matches}")

Output:

Animals found: ['cat', 'dog', 'bird']

NLP Relevance: Ideal for identifying multiple keywords, synonyms, or variations of a term within a text, enhancing the flexibility of your search patterns.

8. Grouping (()) and Capturing in RegEx

Parentheses () are used to group parts of a regular expression together. This serves two primary purposes: applying quantifiers to an entire group and capturing specific portions of the matched text.

Basic Grouping

Allows applying quantifiers to a sequence of characters.

import re

text = "ababab, ab, aabb"
pattern = r"(ab)+" # Matches the sequence 'ab' repeated one or more times

matches = re.findall(pattern, text)
print(f"Pattern '{pattern}' matches: {matches}")

Output:

Pattern '(ab)+' matches: ['ab', 'ab', 'ab']

NLP Relevance: Useful for matching repeated sequences or complex patterns that are treated as a single unit.

Capturing Groups (match.group(n))

When you use parentheses, the matched content within each group is “captured” and can be accessed using match.group(n). n refers to the group number, starting from 1 for the first opening parenthesis. match.group(0) or match.group() returns the entire match.

import re

text = "Name: John Doe, Email: john.doe@example.com"
pattern = r"Name: (\w+ \w+), Email: (\w+\.\w+@\w+\.\w+)" # Group 1: Name, Group 2: Email

match = re.search(pattern, text)

if match:
    print(f"Full Match: {match.group(0)}")
    print(f"Captured Name (Group 1): {match.group(1)}")
    print(f"Captured Email (Group 2): {match.group(2)}")
else:
    print("No match found.")

Output:

Full Match: Name: John Doe, Email: john.doe@example.com
Captured Name (Group 1): John Doe
Captured Email (Group 2): john.doe@example.com

NLP Relevance: Crucial for information extraction, allowing you to pull out structured data (like names, email addresses, dates, or specific data fields) from unstructured text.

9. re.findall(): Extracting All Matches

The re.findall() function returns a list of all non-overlapping matches of the pattern found in the string.

import re

text = "apple banana apple cherry orange apple"
pattern = r"apple"

all_apples = re.findall(pattern, text)
print(f"All occurrences of '{pattern}': {all_apples}")

Output:

All occurrences of 'apple': ['apple', 'apple', 'apple']

NLP Relevance: Ideal for tasks like counting word frequencies, extracting all instances of a particular token, or collecting all relevant keywords.

re.findall() with Capturing Groups

If the pattern contains capturing groups, re.findall() returns a list of tuples. Each tuple contains the captured groups for a single match.

import re

text = "Phone: 123-456-7890, Mobile: 987-654-3210"
pattern = r"(\d{3})-(\d{3})-(\d{4})" # Three capturing groups for phone number parts

phone_numbers_parts = re.findall(pattern, text)
print(f"Phone number parts: {phone_numbers_parts}")

Output:

Phone number parts: [('123', '456', '7890'), ('987', '654', '3210')]

NLP Relevance: Perfect for extracting structured data where different components of the match need to be separated (e.g., area code, prefix, and line number from phone numbers).

10. re.finditer(): Iterating Over Match Objects

The re.finditer() function returns an iterator that yields match object for each non-overlapping match found. This is particularly memory-efficient for very large texts or when dealing with a high number of matches, as it generates matches one by one.

import re

text = "Diwali is a festival of lights, Holi is a festival of colors!"
pattern = r"festival"

print(f"Searching for '{pattern}' in the text:")
for match in re.finditer(pattern, text):
    print(f"  Match found: '{match.group()}' at START index: {match.start()}, END index: {match.end()}")

Output:

Searching for 'festival' in the text:
  Match found: 'festival' at START index: 12, END index: 20
  Match found: 'festival' at START index: 40, END index: 48

NLP Relevance: When you need not only the matched text but also its precise position (start and end indices) within the original string. This is crucial for tasks like named entity recognition (NER), text annotation, or highlighting matches.

11. re.sub(): Substitution and Replacement with RegEx

The re.sub() function is used to substitute (replace) occurrences of a pattern in a string with a specified replacement string.

Simple Replacement

import re

text = "The cat sat on the mat."
pattern = r"cat"
replacement = "dog"

new_text = re.sub(pattern, replacement, text)
print(f"Original: '{text}'")
print(f"Modified: '{new_text}'")

Output:

Original: 'The cat sat on the mat.'
Modified: 'The dog sat on the mat.'

NLP Relevance: A core function for text normalization, censoring sensitive words, replacing slang with formal terms, or correcting common misspellings.

Replacing Multiple Occurrences

By default, re.sub() replaces all non-overlapping occurrences of the pattern.

import re

text = "I love apples, and she loves apples too. Apples are great!"
pattern = r"apples"
replacement = "fruits"

new_text = re.sub(pattern, replacement, text)
print(f"Original: '{text}'")
print(f"Modified: '{new_text}'")

Output:

Original: 'I love apples, and she loves apples too. Apples are great!'
Modified: 'I love fruits, and she loves fruits too. Apples are great!'

Note: The last “Apples” was not replaced because the pattern apples is case-sensitive. To address this, use the re.IGNORECASE flag (see below).

Limiting Replacements with count Parameter

The optional count parameter in re.sub() allows you to limit the maximum number of pattern occurrences to replace.

import re

text = "one two one three one"
pattern = r"one"
replacement = "X"

# Replace only the first 2 occurrences
new_text = re.sub(pattern, replacement, text, count=2)
print(f"Original: '{text}'")
print(f"Modified (first 2 replaced): '{new_text}'")

Output:

Original: 'one two one three one'
Modified (first 2 replaced): 'X two X three one'

NLP Relevance: Useful for targeted text modifications, for instance, replacing only the first few mentions of a term or correcting a specific number of errors.

12. Important RegEx Flags for NLP

Flags modify the behavior of regular expression matching, providing more control and flexibility.

re.IGNORECASE (or re.I): Case-Insensitive Matching

Makes the pattern match regardless of the case of the characters.

import re

text = "Apple, apple, APPLE."
pattern = r"apple"

# Without IGNORECASE, only lowercase 'apple' matches
matches_case_sensitive = re.findall(pattern, text)
print(f"Case-sensitive matches: {matches_case_sensitive}")

# With IGNORECASE, all case variations of 'apple' match
matches_case_insensitive = re.findall(pattern, text, re.IGNORECASE)
print(f"Case-insensitive matches: {matches_case_insensitive}")

Output:

Case-sensitive matches: ['apple']
Case-insensitive matches: ['Apple', 'apple', 'APPLE']

NLP Relevance: Absolutely essential for robust text processing where case variations should not affect matching (e.g., searching for “Python” should also find “python” or “PYTHON”).

re.DOTALL (or re.S): Dot Matches All Including Newlines

Allows the . metacharacter to match newline characters (\n) as well. By default, . does not match newlines.

import re

text = "Line 1\nLine 2\nLine 3"
pattern = r"Line.*Line" # Matches 'Line', then any characters, then 'Line'

# Without DOTALL, .* stops at newline, so it only matches 'Line 1'
match_no_dotall = re.search(pattern, text)
print(f"No DOTALL: {match_no_dotall.group() if match_no_dotall else 'No Match'}")

# With DOTALL, .* crosses newlines, matching from the first 'Line' to the last 'Line'
match_dotall = re.search(pattern, text, re.DOTALL)
print(f"With DOTALL: {match_dotall.group() if match_dotall else 'No Match'}")

Output:

No DOTALL: Line 1
With DOTALL: Line 1
Line 2
Line 3

NLP Relevance: Crucial for extracting multi-line blocks of text, paragraphs, or entire sections where content spans across newline characters.

13. The Importance of Raw Strings (r'') in RegEx

It is highly recommended to use raw strings for regular expression patterns in Python. A raw string is prefixed with r (e.g., r"pattern"). This tells Python to treat backslashes \ as literal characters rather than escape sequences.

import re

# Problem without raw string: '\n' is interpreted as a newline character by Python
# pattern_problem = "\n"
# text = "Hello\nWorld"
# match = re.search(pattern_problem, text)
# print(f"Match for newline (problematic, might not work as expected): {match.group() if match else 'No Match'}")

# Correct way with raw string: '\n' is treated as a literal backslash followed by 'n'
# To match a literal backslash, you'd use r"\\"
# To match a newline character, you'd still use r"\n"
text = "This is a backslash: \\ and a newline: \n"
pattern_backslash = r"\\" # Matches a literal backslash

match_backslash = re.search(pattern_backslash, text)
print(f"Match for literal backslash: {match_backslash.group() if match_backslash else 'No Match'}")

pattern_newline = r"\n" # Matches a newline character

match_newline = re.search(pattern_newline, text)
print(f"Match for newline character: {match_newline.group() if match_newline else 'No Match'}")

Output:

Match for literal backslash: \
Match for newline character: 

NLP Relevance: Prevents common and hard-to-debug errors when dealing with RegEx patterns that contain backslashes (e.g., \d, \w, \s), ensuring your patterns behave as intended. Always use raw strings for your RegEx patterns!

14. RegEx in NLP Lexical Processing: A Deep Dive

Lexical processing is the foundational stage in Natural Language Processing (NLP) where raw, unstructured text is transformed into a more manageable and meaningful format for computational analysis. This critical step involves breaking down continuous text into smaller, discrete units, often called “tokens.” Regular expressions are an indispensable tool in this phase due to their unparalleled ability to define and identify complex patterns within strings.

What is Lexical Processing in NLP?

Lexical processing encompasses several key tasks aimed at preparing text data:

• Tokenization: The process of dividing a text into individual words, punctuation marks, numbers, or other significant units (tokens). This is often the very first step in any NLP pipeline.
• Lowercasing: Converting all text to a consistent case (typically lowercase) to ensure that words like “Apple,” “apple,” and “APPLE” are treated as the same token.
• Punctuation Removal: Eliminating punctuation marks (commas, periods, exclamation points, etc.) that might not contribute to the semantic meaning for a given task.
• Stop Word Removal: Filtering out common words (e.g., “the,” “is,” “and,” “a”) that occur frequently but often carry little unique information or semantic value.
• Handling Special Characters and Symbols: Cleaning text by removing or normalizing emojis, hashtags, URLs, or other non-standard text elements.
• Stemming/Lemmatization: Reducing words to their base or root form (e.g., “running,” “runs,” “ran” all reduce to “run”). While often handled by specialized NLP libraries, RegEx can sometimes assist in simpler rule-based approaches.

The Indispensable Role of RegEx in Lexical Processing

Regular expressions are the workhorse of lexical processing because they offer a highly flexible, powerful, and efficient mechanism to:

• Define Token Boundaries: Precisely specify what constitutes a “word” or a “token” by defining patterns for word characters, spaces, and punctuation.
• Clean and Normalize Text: Easily remove unwanted characters (like stray symbols, extra spaces, or HTML tags) or replace inconsistent formatting.
• Identify Specific Lexical Units: Extract specific types of tokens, such as email addresses, phone numbers, hashtags, URLs, or numerical values, which often follow predictable patterns.
• Handle Edge Cases: Manage complex scenarios like contractions (don't), hyphenated words (well-being), or emoticons that require specific pattern matching rules.

Practical Examples of RegEx in Lexical Processing

Let’s explore common lexical processing tasks and how RegEx provides elegant solutions.

Example 1: Advanced Tokenization

Beyond simple \w+, RegEx can handle more nuanced tokenization, including keeping punctuation as separate tokens.

import re

text = "Hello, world! How are you doing today? #NLP #Python. I'm learning."
# Pattern to match:
# \w+ : one or more word characters (for words)
# | : OR
# [.,!?#'] : specific punctuation marks we want to keep as tokens
tokens_advanced = re.findall(r"\w+|[.,!?#']", text)
print(f"Original text: '{text}'")
print(f"Tokens (advanced): {tokens_advanced}")

Output:

Original text: 'Hello, world! How are you doing today? #NLP #Python. I'm learning.'
Tokens (advanced): ['Hello', ',', 'world', '!', 'How', 'are', 'you', 'doing', 'today', '?', '#', 'NLP', '#', 'Python', '.', 'I', "'", 'm', 'learning']

Explanation: This pattern \w+|[.,!?#'] first tries to match a sequence of word characters. If that doesn’t match, it then tries to match any single character from the specified punctuation set. This allows for more granular tokenization, separating words from their adjacent punctuation.

Example 2: Removing URLs from Text

URLs often contain various characters and can be a source of noise in text data.

import re

text = "Check out my blog at https://www.example.com/blog or visit http://old.site.net/page.html for more info."
# Pattern to match common URL structures
# (http[s]?://)? : optional 'http' or 'https'
# (?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+ : common URL characters
# This is a simplified URL regex, real-world URLs can be more complex.
url_pattern = re.compile(r"http[s]?://(?:[a-zA-Z]|[0-9]|[$-_@.&+]|[!*\(\),]|(?:%[0-9a-fA-F][0-9a-fA-F]))+")
cleaned_text_no_urls = url_pattern.sub(r"", text)
print(f"Original text: '{text}'")
print(f"Text after URL removal: '{cleaned_text_no_urls.strip()}'")

Output:

Original text: 'Check out my blog at https://www.example.com/blog or visit http://old.site.net/page.html for more info.'
Text after URL removal: 'Check out my blog at  or visit  for more info.'

Explanation: A more complex RegEx pattern is used to identify common URL structures. re.sub() then replaces these matched URLs with an empty string, effectively removing them. The .strip() is used to clean up any extra whitespace left behind.

Example 3: Normalizing Multiple Spaces

Multiple spaces can appear due to text formatting or previous cleaning steps.

import re

text = "This   text    has     too   many    spaces."
# Pattern to match one or more whitespace characters
# Replace with a single space
normalized_text = re.sub(r'\s+', ' ', text)
print(f"Original text: '{text}'")
print(f"Text with normalized spaces: '{normalized_text.strip()}'")

Output:

Original text: 'This   text    has     too   many    spaces.'
Text with normalized spaces: 'This text has too many spaces.'

Explanation: \s+ matches one or more whitespace characters. Replacing them with a single space (' ') normalizes the spacing. .strip() is used to remove leading/trailing spaces.

Example 4: Removing Emojis (Simplified)

While full emoji handling is complex, RegEx can remove simple emoji patterns.

import re

text = "Hello world! 😊 How are you? 👍 This is great! 🎉"
# Pattern to match common emoji Unicode ranges (simplified for example)
# This is a basic example and might not cover all emojis or complex sequences.
emoji_pattern = re.compile(
    "["
    "\U0001F600-\U0001F64F"  # emoticons
    "\U0001F300-\U0001F5FF"  # symbols & pictographs
    "\U0001F680-\U0001F6FF"  # transport & map symbols
    "\U0001F1E0-\U0001F1FF"  # flags (iOS)
    "\U00002702-\U000027B0"
    "\U000024C2-\U0001F251"
    "]+", flags=re.UNICODE
)
cleaned_text_no_emojis = emoji_pattern.sub(r'', text)
print(f"Original text: '{text}'")
print(f"Text after emoji removal: '{cleaned_text_no_emojis}'")

Output:

Original text: 'Hello world! 😊 How are you? 👍 This is great! 🎉'
Text after emoji removal: 'Hello world!  How are you?  This is great! '

Explanation: This example uses specific Unicode ranges to identify and remove emojis. The re.UNICODE flag is important for correct Unicode character matching.

These examples vividly demonstrate how regular expressions are not just a programming tool but a critical component in the NLP practitioner’s toolkit for cleaning, normalizing, and preparing text data for advanced AI and Machine Learning models. Mastering these techniques will significantly enhance your ability to work with real-world textual information.