Independent educational research

Methodology and limitations

Measurement

The unit of analysis is a product observation on Shop Fair’s anonymous public online catalog on Mercato.com. The pipeline records what a first-party page published at collection time.

Identity and missingness

Adjacent weeks are matched by canonical relative path `/item/{slug}/{productId}`, not UPC. Missing products and failed weeks stay missing; the project never interpolates prices or manufactures history.

Quality gates

Minimum product count, department coverage, valid-price rate, prior overlap, and product-count-drop checks run before publication. Partial crawls cannot silently replace a healthy baseline.

Anomalies and volatility

An anomaly requires both a movement of at least 20% and an absolute median-absolute-deviation robust z-score of at least 3.5. Four- and eight-week log-price volatility appears only after complete history exists. Neither is a forecast.

Promotions

Public weekly-ad offers are a separate best-effort channel sourced inline from originalPrice payload properties. Missing regular prices, local applicability, loyalty rules, and extraction errors remain explicit.

Responsible access

The collector identifies itself, checks robots rules, runs serially with at least one second between requests, and avoids authentication, private APIs, CAPTCHAs, and restricted paths.

Method-family basis

The project follows time-series preparation and point-outlier framing from Charu C. Aggarwal, Data Mining: The Textbook, sections 14.2–14.7, PDF pages 478–507.