smr-empirical-illustration

SMR Empirical Illustration

Use this to make the real-data section earn its place. SMR expects a methods paper to show that the method matters substantively — that using it instead of the incumbent leads to a different, better-justified conclusion about the social world. A throwaway "we also applied it to some data" section is a reviewer flag.

The "it changes the answer" standard

The illustration's job is to demonstrate consequence:

• Run the incumbent and the new method on the same data, and show where they diverge. The payoff sentence is "the standard approach would have concluded X; our method shows Y, and Y is the defensible answer because [reason tied to the method's properties]."
• Tie the divergence to the mechanism established in smr-derivation-and-properties and the regime identified in smr-simulation-studies: the data should sit in the regime where the incumbent is known to fail.
• State the substantive stake: who would have made a wrong inference, and about what, if they had used the old method? The stake makes the method consequential, not just correct.

Choosing the dataset

• Pick data that lives in the failure regime (e.g., few clusters, non-invariance across groups, informative missingness, network dependence) so the method has something to do.
• Prefer public or depositable data — SMR's availability policy expects the data and code behind the illustration to be accessible (see smr-software-and-reproducibility). If data are restricted, plan the availability statement now.
• A familiar, recognizable dataset lets readers judge the result against intuition; an exotic one forces them to trust you on both the data and the method.

What to report

Keep it an illustration, not a substantive paper

The danger runs both ways. Too thin and it is decorative; too thick and the paper becomes a substantive study that belongs in ASR/AJS (the failure flagged in smr-topic-selection). Calibrate: the illustration should be deep enough to show the method changes the answer, and no deeper. The unit of analysis is the method's behavior on real data, not a full substantive argument with its own literature.

Checklist

• Incumbent and new method are run on the same data with results side by side.
• The divergence is tied to the method's mechanism and the simulated failure regime.
• The substantive stake (who would be wrong, about what) is stated.
• A diagnostic shows the data are actually in the regime where the method is needed.
• Uncertainty is reported for both methods.
• Data are public/depositable, or a restricted-data availability plan exists.
• The section stays an illustration, not a full substantive study.

Anti-patterns

• Decorative application: the method is run, but it would not change any conclusion.
• Regime mismatch: data where the incumbent is fine, so the new method has nothing to prove.
• Substantive creep: the illustration grows into an ASR/AJS-style paper and loses methods focus.
• One-method reporting: showing only the new method's result, hiding what the incumbent would say.
• Inaccessible data with no plan: an illustration readers can never reproduce.

Output format

[Illustration status] consequential / decorative / not ready
[Dataset + regime] <data : why it sits in the failure regime>
[Divergence] <incumbent conclusion vs. new-method conclusion>
[Substantive stake] <who would have been wrong, about what>
[Reproducibility] data/code accessible? restricted-data plan?
[Next SMR skill] smr-tables-figures