Part of Dataset Foundry — synthetic, documented, citable datasets across domains
Agriculture
Log in
Methodology · Crop Yield

What Dataset Foundry's Crop Yield line actually does

This page is meant to be citable directly in a methodology chapter. Everything below is a design decision, not a discovery — Dataset Foundry does not model real farm data, it encodes a documented set of assumptions about how rainfall, soil conditions, and farm-level factors should relate to whether a plot's yield falls short and, optionally, how severe that shortfall is, so the relationships can be recovered and used to teach or test statistical methods. This is the first line in the new agri family — it doesn't share a family landing page with insurance, finance, or health yet — and it carries the same runtime stage-count choice breast cancer introduced: which stage count to generate (single vs. two-stage) is itself a request-time choice, not a fixed shape. See section 1 for what that means.

1. Toggleable one- or two-stage model

Every record starts from the same first stage: one farm plot's single growing-season observation, not a farm-level or district-level aggregate. What happens after depends on the stage_mode you choose on the generator page:

  1. Shortfall probability — whether this plot's realized yield fell below its season-expected threshold is table-rated, the same technique life's/health-claims'/property's/susu's/breast cancer's own core-anchor-plus-loadings tables use. A crop-type base shortfall rate is the anchor; rainfall deviation from the agroecological zone's seasonal norm and soil-moisture deficit at the crop's critical growth stage each apply a banded multiplicative loading on top, with improved-seed-variety adoption, irrigation access, plot size, region, and agroecological zone as further loadings. This logic is identical in both stage modes and is computed before stage_mode is even consulted — single-stage's entire output, and stage one of two-stage.
  2. Yield-loss severity (two-stage only, computed only where a shortfall occurred) — a percentage yield-loss measure modeled as a continuous latent score (an intercept plus a weighted sum of rainfall-deficit magnitude, a pest/disease-outbreak flag, delayed planting, and agroecological zone, plus noise) run through a sigmoid transform and scaled to a 0-100% figure — the first line since motor to reuse the platform's shared sigmoid() helper, chosen here over breast cancer's/susu's score-bucketing or log-normal-ratio techniques because a percentage yield loss has a natural, symmetric ceiling (100% — the whole crop) a sigmoid curve represents directly.

Single-stage reports shortfall occurrence only, no severity fields at all. Two-stage is the default — same "default the richer shape, allow opting down" reasoning the include-names toggle uses on every other line — and additionally reports pest_disease_outbreak, delayed_planting, and yield_loss_pct (non-shortfall rows carry false/0.0 for these three, since they're boolean/numeric rather than breast cancer's categorical stage-side fields — susu's loss_given_default_ghs convention applies here instead of breast cancer's empty-string one). Whichever mode you pick is recorded in the dataset's manifest and citation — two downloads of this line in different modes are, citation-wise, two different datasets, since a single-stage download simply has no severity columns at all.

Unlike every other line except breast cancer, this one carries no name field at all — a plot-season observation has no natural person-name analogue the way a policyholder or susu-group member does, so there's no include-names toggle to speak of here. Unlike every other line including breast cancer, this one also doesn't yet accept a country parameter at all — rainfall reliability, growing-season timing, and staple-crop mix genuinely differ by country (a real basis for country-scoping this line eventually), but a second country's own agroecological-zone and crop tables don't exist yet to select between, so there's nothing for a country parameter to change today. The country selector on the generator page shows Ghana only, honestly, rather than a control that would silently do nothing if a second option were offered.

2. The four hypotheses

Two hypotheses drive shortfall probability and hold in both stage modes, since stage-one logic doesn't change based on the toggle; the other two are yield-loss-severity hypotheses that only exist, and are only verified, in two-stage mode, scoped to shortfall records — property's/susu's/breast cancer's own conditional-on-occurred technique for a second-stage hypothesis. Tracked separately below rather than in one undifferentiated table, since which mode a factor actually applies in is the thing a reader needs disambiguated.

FactorStageDirectionRationale encoded
Rainfall deviation from zone seasonal norm Shortfall probability both modes Negative (deeper deficit → higher probability) The single most consistently documented driver of rainfed-crop yield variability across the agronomy literature — FAO's AquaCrop water-productivity framework treats water availability as the dominant yield-limiting factor for rainfed systems. The more mechanistic of the two core shortfall hypotheses, and why it carries the widest loading spread in the shortfall coefficient table.
Soil-moisture deficit at critical growth stage Shortfall probability both modes Positive Water stress specifically during flowering/grain-fill is repeatedly identified in crop-water-stress literature as the single most yield-critical window for staple cereals — a real, separate driver from total-season rainfall deviation (a season can be near-normal on aggregate rainfall and still deliver a damaging deficit during the few critical weeks that matter most), the second core shortfall hypothesis, drawn independent of rainfall deviation for tractability in this first version.
Rainfall-deficit magnitude Yield-loss severity two-stage only Positive Carries through mechanically from stage one: a plot that missed its shortfall threshold by a wide margin didn't just "have a shortfall," it had a worse one — the more mechanistic of the two stage-severity hypotheses. Reuses the same rainfall_deviation_pct draw already used as a shortfall-probability loading — one field, two roles, susu's loan-purpose/property's property-type/breast cancer's tumor-size reuse pattern.
Pest/disease outbreak Yield-loss severity two-stage only Positive Drought-stressed crops are well documented to be more susceptible to pest and disease pressure — West Africa's Fall Armyworm spread across maize since 2016 is the concrete regional instance of this compounding pattern repeatedly cited in the literature on the outbreak's impact. Modeled as an independent flat-rate draw rather than conditioned on the rainfall-deficit draw itself, the same "real, separately documented driver, correlation not modeled in this first version" simplification the soil-moisture-deficit hypothesis above uses.

All four hypotheses are verified at n=5,000 for shortfall-side quartile checks (motor's/health-claims'/property's/breast cancer's default) and n=40,000 for the full directional-verification pass reported alongside this line's own backend build — checked empirically per those lines' own precedent of not assuming a given n holds without checking. Both shortfall-side directions held comfortably across the range tested (rainfall-deviation quartile shortfall rates falling from roughly 31% to 9%, soil-moisture-deficit quartile rates rising from roughly 14% to 24%); both severity-side directions held too (mean yield loss falling from roughly 68% to 42% across the same rainfall-deviation quartiles; pest/disease-outbreak rows averaging a materially higher loss than rows without), with the shortfall-only subset comfortably in the hundreds to low thousands of rows per seed at these sample sizes — nothing here needed momo's larger n=20,000.

3. Coefficients

Shortfall probability is modeled as p = clip(base_rate(crop_type) · rainfall_deviation_loading · soil_moisture_deficit_loading · improved_seed_variety_loading · irrigation_access_loading · plot_size_loading · region_loading · zone_loading, min_probability, max_probability), where base_rate(crop_type) is the crop-type base-shortfall-rate table lookup, rainfall_deviation_loading/soil_moisture_deficit_loading are banded lookups on their respective continuous values, and every other term is a named multiplicative loading — irrigation access's loading applies only where irrigation access is true (a gate, not a small effect, the same technique breast cancer's postmenopausal-obesity gate and momo's cash-out gate use for their own conditional loadings). A shortfall is then drawn as occurred or not from that probability. When the mode is two-stage, yield-loss percentage (for shortfall plots only) is modeled as a continuous latent score run through a sigmoid transform, score = intercept + rainfall_deficit_coef · max(0, -rainfall_deviation_pct)/100 + pest_disease_outbreak_loading + delayed_planting_loading + zone_loading + ε, then yield_loss_pct = clip(100 · sigmoid(score), min_pct, max_pct). These coefficients are illustrative, anchored against general agronomy and crop-water-stress literature (FAO's AquaCrop framework; CIMMYT/IITA drought-tolerant-maize adoption studies; the documented regional impact of Fall Armyworm on West African maize since 2016), not derived from any real farm-monitoring dataset's own data. Exact magnitudes aren't published on this page — see the current defaults, and change them, in the Advanced: customize the model panel on the Crop Yield generator if your thesis calls for weaker or stronger effects, and note in your write-up which values you changed from the defaults.

4. Determinism

Generation runs server-side using a seeded generator (numpy's default_rng, PCG64) rather than an unseeded source of randomness — same mechanism as every other line on the platform. The same seed, record count, stage mode, generator version, and any advanced-config changes (all stamped in every citation and manifest — see the Cite section on the generator page) will always reproduce byte-identical output. Report the seed, record count, and stage mode alongside your dataset — that combination, together with the generator version, is sufficient for anyone to regenerate exactly what you analyzed.

5. What's decorative vs. load-bearing

Crop type, improved-seed-variety adoption, irrigation access, plot size, region, and agroecological zone exist to make records look like real farm-plot observations and are internally consistent with each other, but they are not the object of the four hypotheses above. Rainfall deviation is the one field that plays a load-bearing role on both sides at once — a shortfall-probability loading and, unchanged, one of the two core severity hypotheses — rather than being load-bearing on one side and decorative on the other, the same dual role tumor size plays in breast cancer's own coefficient tables. This line has no name field of any kind and no country parameter yet (section 1's own note), so unlike every other line there is neither a name-inclusion toggle nor a meaningful country selector to gate.

6. Known limitations

  • This is synthetic data. It should never be presented as, or mistaken for, real farm-monitoring or agronomic survey data in a publication.
  • Shortfall probability is table-rated with banded loadings and multiplicative factors, not a from-scratch regression — no interaction effects between rainfall deviation, soil-moisture deficit, crop type, seed variety, irrigation access, and plot size are modeled beyond simple multiplication, unless you add them yourself downstream.
  • Rainfall deviation and soil-moisture deficit are drawn independently of each other — a real, documented simplification, not an assertion that the two are uncorrelated in practice.
  • Pest/disease outbreak and delayed planting are drawn independent of the rainfall-deficit draw as flat-rate booleans, not conditioned on how severe that season's drought was, even though the literature this line cites documents drought stress as a real driver of pest/disease susceptibility.
  • Pest/disease outbreak, delayed planting, and yield-loss percentage are only present in two-stage mode and only meaningfully populated for shortfall records — a single-stage download or a non-shortfall record simply has no severity information, by design, not by omission.
  • This line doesn't yet accept a country parameter — every record is implicitly Ghana-scoped today (see section 1). A second country's own zone/crop reference data is future work, not represented here at all yet.
  • If your thesis needs a different balance between the four hypotheses, adjust the relevant loadings in the Advanced: customize the model panel on the Crop Yield generator and say so in your write-up.

7. Citing this tool

Something like: "Synthetic data generated using Dataset Foundry (seed = 42, n = 5,000, stage mode = two-stage), a deterministic Crop Yield agronomic data generator built for this thesis in the absence of accessible real-world farm-monitoring data." Include the seed, record count, and stage mode so the exact dataset is reproducible from the tool alone — or use the auto-generated citation block on the generator page, which also includes the generator version, stage mode, and any advanced-config changes.