nlp-engineer

NLP Engineer

Overview

Natural Language Processing (NLP) encompasses techniques for understanding, generating, and transforming human language with computers. This skill covers both classical NLP methods and modern transformer-based approaches, with practical implementations using Hugging Face, spaCy, and NLTK.

Text Preprocessing

Standard Pipeline

import re
import unicodedata

class TextPreprocessor:
    """Standard text preprocessing pipeline."""

    def __init__(self, lowercase: bool = True, remove_urls: bool = True,
                 remove_html: bool = True, remove_special_chars: bool = False):
        self.lowercase = lowercase
        self.remove_urls = remove_urls
        self.remove_html = remove_html
        # ... (condensed) ...
        return text

    def process_batch(self, texts: list[str]) -> list[str]:
        return [self.process(t) for t in texts]

Stopword Removal and Stemming

import nltk
from nltk.corpus import stopwords
from nltk.stem import WordNetLemmatizer, PorterStemmer

nltk.download('stopwords')
nltk.download('wordnet')

class LinguisticPreprocessor:
    def __init__(self, language: str = "english", use_lemma: bool = True):
        self.stop_words = set(stopwords.words(language))
        self.lemmatizer = WordNetLemmatizer() if use_lemma else None
        # ... (condensed) ...
        elif self.stemmer:
            tokens = [self.stemmer.stem(t) for t in tokens]

        return tokens

When to Preprocess

Key insight: Modern transformer models handle raw text best. Only preprocess for classical ML or information retrieval tasks.

Tokenization

Tokenizer Types

# Word-level tokenization
text = "The cat sat on the mat."
word_tokens = text.split()  # ['The', 'cat', 'sat', 'on', 'the', 'mat.']

# Subword tokenization (BPE - used by GPT models)
from transformers import AutoTokenizer
tokenizer = AutoTokenizer.from_pretrained("gpt2")
bpe_tokens = tokenizer.tokenize("unhappiness")
# ['un', 'h', 'app', 'iness']

# WordPiece tokenization (used by BERT)
# ... (condensed) ...
# SentencePiece (used by T5, LLaMA)
tokenizer = AutoTokenizer.from_pretrained("t5-base")
sp_tokens = tokenizer.tokenize("unhappiness")
# ['_un', 'happiness']

Tokenizer Comparison

Practical Token Counting

import tiktoken

def count_tokens(text: str, model: str = "gpt-4o") -> int:
    """Count tokens for a given model."""
    encoding = tiktoken.encoding_for_model(model)
    return len(encoding.encode(text))

def truncate_to_tokens(text: str, max_tokens: int, model: str = "gpt-4o") -> str:
    """Truncate text to fit within token limit."""
    encoding = tiktoken.encoding_for_model(model)
    tokens = encoding.encode(text)
    if len(tokens) <= max_tokens:
        return text
    return encoding.decode(tokens[:max_tokens])

Named Entity Recognition (NER)

spaCy NER

import spacy

nlp = spacy.load("en_core_web_trf")  # Transformer-based (best quality)
# Or: nlp = spacy.load("en_core_web_sm")  # Smaller, faster

def extract_entities(text: str) -> list[dict]:
    """Extract named entities with spaCy."""
    doc = nlp(text)
    entities = []
    for ent in doc.ents:
        entities.append({
            # ... (condensed) ...
#   {"text": "Tim Cook", "label": "PERSON", ...},
#   {"text": "Cupertino", "label": "GPE", ...},
#   {"text": "January 15, 2025", "label": "DATE", ...},
# ]

Hugging Face NER

from transformers import pipeline

ner_pipeline = pipeline(
    "ner",
    model="dslim/bert-base-NER",
    aggregation_strategy="simple",
)

results = ner_pipeline("Elon Musk founded SpaceX in Hawthorne, California.")
# [
#   {"entity_group": "PER", "word": "Elon Musk", "score": 0.99},
#   {"entity_group": "ORG", "word": "SpaceX", "score": 0.98},
#   {"entity_group": "LOC", "word": "Hawthorne, California", "score": 0.97},
# ]

Custom NER Training

import spacy
from spacy.training import Example

def train_custom_ner(
    train_data: list[tuple[str, dict]],
    model: str = "en_core_web_sm",
    n_iter: int = 30,
    output_dir: str = "./custom_ner",
):
    """Train a custom NER model with new entity types."""
    nlp = spacy.load(model)
# ... (condensed) ...
train_data = [
    ("Order #12345 shipped via FedEx", {"entities": [(6, 12, "ORDER_ID"), (25, 30, "CARRIER")]}),
    ("Tracking number TRK-9876 for UPS", {"entities": [(16, 24, "TRACKING"), (29, 32, "CARRIER")]}),
]

Sentiment Analysis

Hugging Face Sentiment

from transformers import pipeline

# General sentiment
sentiment = pipeline("sentiment-analysis", model="cardiffnlp/twitter-roberta-base-sentiment-latest")
result = sentiment("This product exceeded my expectations!")
# [{"label": "positive", "score": 0.97}]

# Fine-grained sentiment (1-5 stars)
fine_grained = pipeline("sentiment-analysis", model="nlptown/bert-base-multilingual-uncased-sentiment")
result = fine_grained("The food was okay but nothing special.")
# [{"label": "3 stars", "score": 0.45}]

Aspect-Based Sentiment

def aspect_sentiment(text: str, aspects: list[str], client) -> dict:
    """Analyze sentiment for specific aspects of a review."""
    prompt = f"""Analyze the sentiment for each aspect in this review.
For each aspect, output: positive, negative, neutral, or not_mentioned.

Review: "{text}"

Aspects to analyze: {json.dumps(aspects)}

Output JSON: {{"aspect": "sentiment"}}"""

    # ... (condensed) ...
    "Great camera quality but battery life is terrible. Screen is decent.",
    ["camera", "battery", "screen", "price"]
)
# {"camera": "positive", "battery": "negative", "screen": "neutral", "price": "not_mentioned"}

Text Classification

Zero-Shot Classification

from transformers import pipeline

classifier = pipeline("zero-shot-classification", model="facebook/bart-large-mnli")

result = classifier(
    "The company announced record quarterly earnings today.",
    candidate_labels=["business", "sports", "technology", "politics"],
)
# {"labels": ["business", "politics", "technology", "sports"],
#  "scores": [0.92, 0.04, 0.03, 0.01]}

Training a Custom Classifier

from transformers import (
    AutoModelForSequenceClassification,
    AutoTokenizer,
    TrainingArguments,
    Trainer,
)
from datasets import Dataset

def train_classifier(
    train_data: list[dict],
    model_name: str = "distilbert-base-uncased",
    # ... (condensed) ...

    trainer.train()
    trainer.save_model(output_dir)
    return trainer

Summarization

Extractive vs Abstractive

Abstractive Summarization

from transformers import pipeline

summarizer = pipeline("summarization", model="facebook/bart-large-cnn")

long_text = """[Long article text here...]"""

summary = summarizer(
    long_text,
    max_length=150,
    min_length=50,
    do_sample=False,
)
print(summary[0]["summary_text"])

Hierarchical Summarization for Long Documents

def hierarchical_summarize(
    text: str,
    chunk_size: int = 3000,
    model_name: str = "facebook/bart-large-cnn",
) -> str:
    """Summarize long documents by chunking, summarizing, then re-summarizing."""
    summarizer = pipeline("summarization", model=model_name)

    # Split into chunks
    chunks = [text[i:i+chunk_size] for i in range(0, len(text), chunk_size)]

    # ... (condensed) ...
    if len(combined.split()) > 500:
        final = summarizer(combined, max_length=200, min_length=50)[0]["summary_text"]
        return final
    return combined

Topic Modeling

BERTopic (Modern Approach)

from bertopic import BERTopic

def discover_topics(documents: list[str], n_topics: int = "auto") -> dict:
    """Discover topics in a collection of documents using BERTopic."""
    topic_model = BERTopic(
        language="english",
        nr_topics=n_topics if n_topics != "auto" else None,
        verbose=True,
    )

    topics, probs = topic_model.fit_transform(documents)
# ... (condensed) ...
        "topics": topic_docs,
        "assignments": topics,
        "probabilities": probs,
    }

LDA (Classical Approach)

from sklearn.decomposition import LatentDirichletAllocation
from sklearn.feature_extraction.text import CountVectorizer

def lda_topics(documents: list[str], n_topics: int = 10) -> dict:
    """Classical LDA topic modeling."""
    vectorizer = CountVectorizer(max_df=0.95, min_df=2, max_features=5000, stop_words="english")
    doc_term_matrix = vectorizer.fit_transform(documents)

    lda = LatentDirichletAllocation(
        n_components=n_topics,
        random_state=42,
        # ... (condensed) ...
        top_words = [feature_names[i] for i in topic.argsort()[:-11:-1]]
        topics[topic_idx] = top_words

    return topics

Language Detection

from langdetect import detect, detect_langs

def detect_language(text: str) -> dict:
    """Detect the language of input text."""
    language = detect(text)
    all_langs = detect_langs(text)  # With probabilities

    return {
        "primary": language,
        "all": [{"lang": str(l).split(":")[0], "prob": l.prob} for l in all_langs],
    }

# For higher accuracy with short text:
from transformers import pipeline
lang_detector = pipeline("text-classification", model="papluca/xlm-roberta-base-language-detection")
result = lang_detector("Bonjour le monde")
# [{"label": "fr", "score": 0.99}]

Transformer Architecture Overview

Core Components

Input Text
    |
[Tokenization]
    |
[Token Embeddings + Position Embeddings]
    |
+-- Transformer Block (x N layers) --+
|                                      |
|  [Multi-Head Self-Attention]         |
|  [Add & Layer Norm]                  |
|  [Feed-Forward Network]             |
# ... (condensed) ...
    |
[Task-Specific Head]
    |
Output

Architecture Families

Hugging Face Pipeline Quick Reference

from transformers import pipeline

# Sentiment Analysis
sentiment = pipeline("sentiment-analysis")
sentiment("I love this movie!")

# Named Entity Recognition
ner = pipeline("ner", aggregation_strategy="simple")
ner("John works at Google in New York.")

# Question Answering
# ... (condensed) ...

# Feature Extraction (embeddings)
extractor = pipeline("feature-extraction")
embeddings = extractor("Hello world")

Checklist

• Choose preprocessing level based on the task (minimal for transformers)
• Select the right tokenizer for your model family
• Use spaCy or Hugging Face for NER based on accuracy vs speed needs
• Implement sentiment analysis with appropriate granularity
• Build text classification with zero-shot first, fine-tune if needed
• Apply hierarchical summarization for long documents
• Use BERTopic for modern topic modeling, LDA for interpretability
• Add language detection for multilingual pipelines
• Choose the right transformer architecture for your task type
• Profile inference latency and optimize (distillation, quantization)
• Version models and track performance metrics over time

When to Use

Use this skill when:

• Designing or implementing nlp engineer solutions
• Reviewing or improving existing nlp engineer approaches
• Making architectural or implementation decisions about nlp engineer
• Learning nlp engineer patterns and best practices
• Troubleshooting nlp engineer-related issues

Do NOT use this skill when:

• The question is about a fundamentally different technology domain
• A more specific sibling skill covers the exact topic needed
• The user needs a complete hands-on tutorial rather than expert guidance

Output Format

# Nlp Engineer Analysis

## Context Assessment
[Situation summary and constraints]

## Recommended Approach
[Primary recommendation with rationale]

## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]

## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]

## Next Steps
- [Immediate action item]
- [Follow-up action item]

Example

Input: "Help me implement nlp engineer for a medium-scale production application"

Output: A structured analysis covering current state assessment, recommended nlp engineer approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.

Edge Cases

• Legacy system integration: When nlp engineer must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
• Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
• Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
• Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities