dabstep-dataset-metadata-business-rules

Dabstep: Dataset Metadata & Business Rules

Dataset Overview

Seven files describe a payment processing system:

Step 1: Always Read Documentation First

Before any analysis, read both:

1. manual.md — authoritative definitions of all fee factors and their cost direction
2. payments-readme.md — column-by-column schema for payments.csv

These files contain the ground truth for interpreting field semantics. Do not rely on raw data averages alone for factor directionality questions.

Key Domain Knowledge (from manual.md)

Fee Formula

fee = fixed_amount + rate * transaction_value / 10000

null values in fee rules mean "applies to all values of that field."

Fee Factor Directionality (authoritative — from manual.md)

Boolean Factors Summary

• is_credit: Credit cards cost more → cheaper when False
• intracountry: Domestic (True) is cheaper; international (False) is more expensive → cheaper when True

Volume Tiers (lowest to highest)

<100k → 100k-1m → 1m-5m → >5m

Fraud Level Tiers (lowest to highest)

<7.2% → 7.2%-7.7% → 7.7%-8.3% → >8.3%

payments.csv Key Columns

• has_fraudulent_dispute: Boolean — fraud indicator from issuing bank
• is_credit: Boolean — credit vs debit card
• intracountry: Derived from issuing_country == acquirer_country
• aci: Authorization Characteristics Indicator (A–G)

ACI Codes

Question-Type Strategies

1. Metadata / Column Identification

"What column indicates X?" / "What does field Y mean?"

Check payments-readme.md first (columns), then manual.md (definitions).

Example: fraud column → has_fraudulent_dispute (from payments-readme.md: "Indicator of fraudulent dispute from issuing bank (Boolean)")

2. Boolean Fee Factor Questions

"Which boolean factor is cheaper when True/False?"

Rely on manual.md explicit statements, not raw data averages. Raw averages are confounded by other correlated variables (card scheme, MCC, etc.) and can be misleading.

Boolean factors in fees.json: is_credit, intracountry

• Cheaper when True: intracountry (domestic transactions are cheaper)
• Cheaper when False: is_credit (non-credit/debit is cheaper)

3. General Factor Directionality

"Which factors lead to cheaper fees when decreased/increased?"

Map each factor using manual.md definitions:

• Decreasing → cheaper: monthly_fraud_level, is_credit (True→False)
• Increasing → cheaper: monthly_volume, intracountry (False→True)
• Decreasing → more expensive: capture_delay (faster = more expensive), monthly_volume, intracountry

4. Volume/Threshold Analysis

"At what volume do fees stop decreasing?" / "Highest volume where fees are not cheaper?"

Analyze actual fee data from fees.json:

import json, pandas as pd
with open('fees.json') as f:
    fees = json.load(f)
df = pd.DataFrame(fees)

for vol in ['<100k', '100k-1m', '1m-5m', '>5m']:
    s = df[df['monthly_volume'] == vol]
    print(vol, s['rate'].mean(), s['fixed_amount'].mean())

The actual data shows >5m has the highest average rate — fees do not become cheaper at >5m. The >5m tier is the highest volume at which fees are not cheaper.

5. "Not Applicable" Detection

"Does X fee exist?" / "What is the value of Y?"

A question should be answered "Not Applicable" when the concept genuinely does not exist in any dataset file. Verify by:

1. Searching manual.md for the concept
2. Searching all keys in fees.json
3. Checking payments.csv columns
4. Searching payments-readme.md

Example: "excessive retry fee" — manual mentions excessive retrying causes downgrades but defines no specific retry fee amount → answer is "Not Applicable"

Common Pitfalls

Don't trust raw averages for boolean factors. The raw average rate for is_credit=True can appear lower than is_credit=False due to confounding (different MCCs and card schemes have both different base rates and different credit ratios). The manual's explicit statement is authoritative.

intracountry in fees.json uses 0.0/1.0 floats, not Python booleans. 1.0 = True (domestic), 0.0 = False (international).

null in fees.json means "applies to all values" — do not treat null rules as lower-priority; they apply when no more-specific rule matches.

Read the full manual.md — it has 10+ sections. The output may be truncated; if so, use grep or targeted searches to find specific definitions.

Answer Format Rules

• Column names: exact string as in payments-readme.md (e.g., has_fraudulent_dispute)
• Multiple factors: comma-separated, alphabetical or as listed (e.g., monthly_fraud_level, is_credit)
• Volume tiers: exact strings from data (e.g., >5m, 1m-5m)
• Fraud levels: exact strings from data (e.g., <7.2%, >8.3%)
• Not found: Not Applicable