embedding-engineer

Embedding Engineer

Overview

Vector embeddings are dense numerical representations of data (text, images, audio) that capture semantic meaning in a continuous vector space. This skill covers the full lifecycle of working with embeddings: model selection, generation, similarity computation, fine-tuning, visualization, and production index optimization.

Embedding Fundamentals

What Embeddings Capture

Embeddings map discrete objects (words, sentences, images) into a continuous vector space where geometric relationships encode semantic relationships. Objects with similar meanings cluster together; dissimilar objects are far apart.

from openai import OpenAI

client = OpenAI()

def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
    """Generate an embedding vector for a text string."""
    response = client.embeddings.create(
        model=model,
        input=text,
    )
    return response.data[0].embedding
# ... (condensed) ...
emb2 = get_embedding("A feline rested on the rug")
emb3 = get_embedding("Stock prices rose sharply today")

# emb1 and emb2 will be much closer than emb1 and emb3

Embedding Model Comparison

Text Embedding Models

Selection Decision Tree

What is your budget constraint?
  FREE -> Continue to open-source options
  PAID -> Continue to API options

--- Open-Source Path ---
Need max quality regardless of compute?
  YES -> e5-mistral-7b-instruct (GPU required)
  NO  -> Continue

Need fast inference (CPU)?
  YES -> all-MiniLM-L6-v2 (384-dim, very fast)
  # ... (condensed) ...

Need multilingual?
  YES -> Cohere embed-v3
  NO  -> text-embedding-3-small (best cost/quality ratio)

Matryoshka (Dimensional Reduction at Generation)

OpenAI text-embedding-3 models support shortening embeddings without retraining:

# Full dimensions
full = client.embeddings.create(
    model="text-embedding-3-large",
    input="Hello world",
    dimensions=3072,  # Full size
)

# Reduced dimensions (still useful!)
reduced = client.embeddings.create(
    model="text-embedding-3-large",
    input="Hello world",
    dimensions=256,  # 12x smaller, ~5% quality loss
)

Similarity Metrics

Cosine Similarity

Measures the angle between two vectors. Invariant to vector magnitude. Most common for text embeddings.

import numpy as np

def cosine_similarity(a: np.ndarray, b: np.ndarray) -> float:
    """Range: [-1, 1]. 1 = identical direction, 0 = orthogonal, -1 = opposite."""
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

Euclidean Distance

Measures straight-line distance. Sensitive to magnitude. Best when magnitude matters.

def euclidean_distance(a: np.ndarray, b: np.ndarray) -> float:
    """Range: [0, inf). 0 = identical. Lower = more similar."""
    return np.linalg.norm(a - b)

Dot Product

Measures both direction and magnitude similarity. Fastest to compute.

def dot_product(a: np.ndarray, b: np.ndarray) -> float:
    """Range: (-inf, inf). Higher = more similar (when vectors are normalized)."""
    return np.dot(a, b)

When to Use Which Metric

Key insight: If vectors are L2-normalized, cosine similarity and dot product produce the same ranking. Most embedding models output normalized vectors, so dot product is preferred for speed.

Batch Embedding Pipeline

import numpy as np
from typing import Generator

class EmbeddingPipeline:
    """Production embedding pipeline with batching and caching."""

    def __init__(self, model: str = "text-embedding-3-small", batch_size: int = 100):
        self.client = OpenAI()
        self.model = model
        self.batch_size = batch_size
        self.cache = {}
# ... (condensed) ...

    def _batch(self, items, size) -> Generator:
        for i in range(0, len(items), size):
            yield items[i:i + size]

Dimensionality Reduction

PCA

from sklearn.decomposition import PCA

def reduce_with_pca(embeddings: np.ndarray, target_dims: int = 128) -> np.ndarray:
    """Reduce embedding dimensions using PCA."""
    pca = PCA(n_components=target_dims)
    reduced = pca.fit_transform(embeddings)

    # Report variance retained
    explained_variance = sum(pca.explained_variance_ratio_)
    print(f"Retained {explained_variance:.1%} of variance with {target_dims} dims")

    return reduced

UMAP (for visualization and moderate reduction)

import umap

def reduce_with_umap(
    embeddings: np.ndarray,
    target_dims: int = 2,
    n_neighbors: int = 15,
    min_dist: float = 0.1,
) -> np.ndarray:
    """Reduce dimensions with UMAP (preserves local structure)."""
    reducer = umap.UMAP(
        n_components=target_dims,
        n_neighbors=n_neighbors,
        min_dist=min_dist,
        metric="cosine",
        random_state=42,
    )
    return reducer.fit_transform(embeddings)

Choosing Reduction Method

Purpose: Visualization (2D/3D)?
  YES -> UMAP or t-SNE
  NO  -> Continue

Purpose: Storage/speed optimization?
  YES -> Continue

Reduction factor < 4x (e.g., 1536 -> 512)?
  YES -> PCA (linear, fast, preserves global structure)
  NO  -> Continue

Reduction factor > 10x?
  YES -> Consider Matryoshka embeddings or re-embedding with smaller model
  NO  -> PCA with quality check on downstream task

Embedding Fine-Tuning

Sentence Transformers Fine-Tuning

from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

def fine_tune_embeddings(
    base_model: str = "all-MiniLM-L6-v2",
    train_pairs: list[tuple[str, str, float]] = None,
    output_path: str = "fine-tuned-model",
    epochs: int = 3,
    batch_size: int = 16,
):
    """Fine-tune embedding model on domain-specific similarity pairs."""
    # ... (condensed) ...
        show_progress_bar=True,
    )

    return model

Contrastive Fine-Tuning (Triplet Loss)

from sentence_transformers import losses

def fine_tune_contrastive(
    base_model: str,
    triplets: list[tuple[str, str, str]],
    output_path: str,
):
    """Fine-tune using anchor, positive, negative triplets."""
    model = SentenceTransformer(base_model)

    # Each triplet: (anchor, positive, negative)
    # ... (condensed) ...
        output_path=output_path,
    )

    return model

Generating Training Data

def generate_training_pairs(documents: list[dict]) -> list[tuple[str, str, float]]:
    """Generate similarity pairs from domain documents."""
    pairs = []

    # Positive pairs: chunks from same document
    for doc in documents:
        chunks = chunk_document(doc)
        for i in range(len(chunks)):
            for j in range(i + 1, min(i + 3, len(chunks))):
                pairs.append((chunks[i], chunks[j], 0.8))

    # ... (condensed) ...
                    0.2,
                ))

    return pairs

Multi-Modal Embeddings

CLIP for Text-Image Embeddings

from transformers import CLIPModel, CLIPProcessor
import torch

class MultiModalEmbedder:
    """Generate embeddings for both text and images in shared space."""

    def __init__(self, model_name: str = "openai/clip-vit-large-patch14"):
        self.model = CLIPModel.from_pretrained(model_name)
        self.processor = CLIPProcessor.from_pretrained(model_name)

    def embed_text(self, texts: list[str]) -> np.ndarray:
        # ... (condensed) ...
    def text_image_similarity(self, text: str, image) -> float:
        text_emb = self.embed_text([text])
        image_emb = self.embed_image([image])
        return cosine_similarity(text_emb[0], image_emb[0])

ImageBind (Text, Image, Audio, Video)

# ImageBind supports 6 modalities in a single embedding space
from imagebind.models import imagebind_model
from imagebind import data as imagebind_data

model = imagebind_model.imagebind_huge(pretrained=True)
model.to("cuda")

# Load different modalities
inputs = {
    imagebind_data.ModalityType.TEXT: imagebind_data.load_and_transform_text(
        ["a dog barking"], "cuda"
    # ... (condensed) ...

# Compare across modalities
text_emb = embeddings[imagebind_data.ModalityType.TEXT]
audio_emb = embeddings[imagebind_data.ModalityType.AUDIO]

Embedding Visualization

Interactive Visualization with Plotly

import plotly.express as px
import pandas as pd

def visualize_embeddings(
    embeddings: np.ndarray,
    labels: list[str],
    texts: list[str] = None,
    method: str = "umap",
) -> None:
    """Create interactive 2D visualization of embeddings."""

    # ... (condensed) ...
        title=f"Embedding Space ({method.upper()})",
        width=900, height=700,
    )
    fig.show()

Cluster Analysis

from sklearn.cluster import KMeans, DBSCAN

def analyze_embedding_clusters(
    embeddings: np.ndarray,
    method: str = "kmeans",
    n_clusters: int = 10,
) -> dict:
    """Analyze natural clusters in embedding space."""

    if method == "kmeans":
        clusterer = KMeans(n_clusters=n_clusters, random_state=42, n_init=10)
        # ... (condensed) ...
        "n_clusters": len(set(labels)) - (1 if -1 in labels else 0),
        "silhouette_score": sil_score,
        "centroids": centroids,
    }

Index Optimization

HNSW Parameters

# HNSW (Hierarchical Navigable Small World) is the most common ANN index

# Key parameters:
# M: Number of bi-directional links per element (higher = better quality, more memory)
# ef_construction: Size of dynamic candidate list during build (higher = better quality, slower build)
# ef_search: Size of dynamic candidate list during search (higher = better quality, slower search)

HNSW_CONFIGS = {
    "speed_priority": {
        "M": 8,
        "ef_construction": 100,
        # ... (condensed) ...
        "ef_search": 200,
        # ~99.5% recall, slower
    },
}

Memory Estimation

def estimate_index_memory(
    n_vectors: int,
    dimensions: int,
    M: int = 16,
    bytes_per_dim: int = 4,
) -> dict:
    """Estimate memory requirements for HNSW index."""

    # Raw vector storage
    vector_bytes = n_vectors * dimensions * bytes_per_dim

    # ... (condensed) ...

# Example: 10M vectors at 1536 dimensions
mem = estimate_index_memory(10_000_000, 1536)
# ~68 GB total

Quantization for Memory Reduction

# Product Quantization (PQ) can reduce memory 4-16x
# with minimal quality loss

# Scalar quantization: float32 -> int8 (4x reduction)
def scalar_quantize(vectors: np.ndarray) -> tuple[np.ndarray, dict]:
    """Quantize float32 vectors to int8."""
    mins = vectors.min(axis=0)
    maxs = vectors.max(axis=0)
    ranges = maxs - mins
    ranges[ranges == 0] = 1

    # ... (condensed) ...

def scalar_dequantize(quantized: np.ndarray, params: dict) -> np.ndarray:
    """Restore float32 from int8 quantized vectors."""
    return quantized.astype(np.float32) / 255 * params["ranges"] + params["mins"]

Benchmarking Embeddings

def benchmark_embedding_quality(
    model_fn,
    test_pairs: list[tuple[str, str, float]],
) -> dict:
    """Benchmark embedding model on similarity task."""
    from scipy.stats import spearmanr

    predicted_scores = []
    true_scores = []

    for text_a, text_b, true_score in test_pairs:
        # ... (condensed) ...
        "p_value": p_value,
        "mean_predicted_similarity": np.mean(predicted_scores),
        "std_predicted_similarity": np.std(predicted_scores),
    }

Checklist

• Select embedding model based on quality, cost, latency, and context length needs
• Choose the right similarity metric (cosine for most text tasks)
• Implement batched embedding generation with caching
• Apply dimensionality reduction if storage/speed is a concern
• Fine-tune embeddings on domain data if generic models underperform
• Visualize embedding space to verify semantic clustering
• Configure HNSW index parameters (M, ef_construction, ef_search) appropriately
• Estimate and plan for memory requirements at scale
• Consider quantization for large-scale deployments
• Benchmark embedding quality on domain-specific similarity tasks
• Monitor embedding drift as source data evolves

When to Use

Use this skill when:

• Designing or implementing embedding engineer solutions
• Reviewing or improving existing embedding engineer approaches
• Making architectural or implementation decisions about embedding engineer
• Learning embedding engineer patterns and best practices
• Troubleshooting embedding engineer-related issues

Do NOT use this skill when:

• The question is about a fundamentally different technology domain
• A more specific sibling skill covers the exact topic needed
• The user needs a complete hands-on tutorial rather than expert guidance

Output Format

# Embedding Engineer Analysis

## Context Assessment
[Situation summary and constraints]

## Recommended Approach
[Primary recommendation with rationale]

## Implementation Steps
1. [Step with specific details]
2. [Step with specific details]
3. [Step with specific details]

## Trade-offs and Considerations
- [Key trade-off 1]
- [Key trade-off 2]

## Next Steps
- [Immediate action item]
- [Follow-up action item]

Example

Input: "Help me implement embedding engineer for a medium-scale production application"

Output: A structured analysis covering current state assessment, recommended embedding engineer approach with specific patterns, implementation roadmap with milestones, and risk mitigation strategies tailored to the application scale and constraints.

Edge Cases

• Legacy system integration: When embedding engineer must coexist with legacy approaches, provide a gradual migration path rather than a complete rewrite
• Scale mismatch: When the solution complexity exceeds the project scale, recommend a simpler approach and note when to revisit
• Team skill gaps: When the team lacks experience with the recommended approach, include learning resources and simpler alternatives
• Conflicting requirements: When constraints conflict (e.g., performance vs. maintainability), explicitly state the trade-off and recommend based on stated priorities