The short version
- The paper reports 87.9% accuracy for its regrowth model and estimates 215 million hectares () where forest could regrow naturally.
- Both evaluators rated the paper 50/100 overall and had low confidence in the 215 Mha figure. Their biggest worries were about that figure: it mixes natural and human factors, and it’s far above the regrowth actually observed. Their joint report also argued that the 87.9% overstates how well the model can predict the future, because of : some predictors were measured during the regrowth period itself. (This was one of their three main concerns, and one we’d specifically asked them to look at.)
- Fouilloux’s Colombia replication supports the forecasting worry: a model trained on 2000–2012 scores 71.7% on 2012–2024, against 77.8% on held-out points from its own period (same labels). But it doesn’t support leakage as the cause: that forward model didn’t use the suspect predictors at all, and dropping them from the original setup barely changes accuracy.
- So the evaluators’ concern that the accuracy figure overstates forecasting performance looks right, but their proposed explanation for it looks weaker, at least in Colombia. Nothing here tests the 215 Mha global total directly.
Why this is useful for the session: it’s a real chain from claim, to expert assessment, to a suggested check, to an independent result. It also involves that a needs to keep apart.
1. The paper and its two claims
In short: the paper makes two different claims, and the evidence on this page bears on each of them differently.
- The accuracy claim: a model correctly classifies “regrowth” vs “no regrowth” at 87.9% of validation points.
- The area claim: 215 Mha of deforested tropical land has potential for natural regeneration. The paper links that to 23.4 of above-ground carbon over 30 years.
Both evaluators picked the area claim as the paper’s main claim. Fouilloux’s replication tests the accuracy claim and the methods behind it. It doesn’t test the global area total, which would need a different kind of check.
Start with The Unjournal’s evaluation summary, which includes the two evaluators’ joint synthesis. Evaluation 1 and Evaluation 2 have the full reasoning. The replication results come from Fouilloux’s public write-up. I haven’t re-run anything for this page.
2. What does “87.9% accuracy” actually mean?
In short: it’s the share of all test points the model got right, counting both “regrowth” and “no regrowth” answers. It does not mean “when it predicts regrowth, it’s right 87.9% of the time.”
The authors trained the model on mapped natural regrowth from 2000–2016. They then checked its classifications at 4.87 million separate points, against . The test set was : half regrowth, half not. 87.9% of the classifications matched.
“When it predicted regrowth, how often was it right?” is a different question. That’s , which remote-sensing people call . With a 50/50 test set, accuracy and precision come out similar. In a real landscape, where regrowth is much rarer, they can be far apart. The example below shows how.
Try it: how often is a “regrowth” call right when regrowth is rare?
Thought experiment: you use the model to pick sites in some region. The slider sets how common real regrowth is there. The model’s skill stays fixed (); only the landscape changes. Watch precision.
| Actually regrows | Actually doesn’t | |
|---|---|---|
| Model says regrowth | 427 | 40 |
| Model says no regrowth | 74 | 460 |
88.7%Accuracy: right across all sites
91.4%Precision: right among “regrowth” calls
At 2%, accuracy is still about 92% (most sites are easy “no regrowth” calls), but precision is about 18%: roughly four in five “regrowth” calls would be wrong.
Does the paper report precision anywhere? more detail
Yes, in its Supplementary Information (Table A1.2). Of the test points the model called “regrowth,” 92% were regrowth in the reference data. That’s the paper’s precision (user’s accuracy). Overall accuracy in the same table is 87.8%.
But about 54% of that table’s reference points are regrowth (537,956 of 1 million). In the actual landscape, observed regrowth is more like 1–2%: 31.6 Mha over 2000–2016, against about 2,190 Mha of mapped area. So 92% is precision where regrowth is common, like the 50% setting in the example above, and it would be much lower at realistic shares. The table doesn’t say exactly how its reference points were drawn, so this is my reading.
The same table uses these rates for an area-adjusted estimate of 512 Mha of regeneration potential, more than twice the mapped 215 Mha. The evaluators’ synthesis flags the mismatch: 92% of 215 Mha already implies about 198 Mha of real regrowth, far more than the regrowth actually observed.
Why 87.9% may overstate how well the map works in practice more detail
The 87.9% test points weren’t used to fit the model, so this isn’t just “predicting in sample.” But they were drawn from the same places and the same years as the training points. That makes the test easier than the job the map is meant to do, in two ways.
- Place. The test points were scattered at random among the training points, so a typical test point has training points nearby, with similar soil, climate, forest cover and land-use history. The model can score well partly by recognising the neighbourhood. In real use it has to predict for areas far from anything it was trained on. The paper’s own check (Extended Data Fig. 4) finds lower accuracy away from training points, down to 81.4% at 2–2.5 km, though the pattern is muddied because the mix of regrowth and non-regrowth also changes with distance. In Fouilloux’s Colombia data, accuracy stays around 88–92% out to about 7 km and falls to 79% beyond 8 km (a small sample).
- Time. The test asks whether the model can tell where regrowth did happen in 2000–2016, using conditions measured over those same years. The map is meant to say where regrowth will happen, from conditions measured now. If what drives regrowth changes (prices, policy, land abandonment), or if some inputs partly reflect regrowth already under way (the leakage concern), the past-period score overstates forecasting accuracy.
Two tougher tests get at this. A holds out whole regions. A trains on an earlier period and predicts a later one. Fouilloux ran both (section 4).
3. What the evaluators said
In short: both rated the paper 50/100 overall, both had low confidence in the 215 Mha figure, and together they argued that leakage (predictors measured during the regrowth period itself) makes the accuracy number look better than real forecasting performance.
| Evaluation | () | Belief in the 215 Mha claim | Suggested check |
|---|---|---|---|
| Evaluation 1 | 50 (40–60) | Low. Probably a lower bound on biophysical potential, because of human factors, but “certainly well below” 678 Mha. | Make it a genuine forecast: use only conditions observed before the regrowth period to predict what regrows afterwards, then apply that to today’s conditions. |
| Evaluation 2 | 50 (40–60) Originally 30–80; revised after the two evaluators discussed their reports. | Low. A rough correction for input-data errors gives a credible range of about 80–180 Mha, i.e. probably lower than 215. | Compare several machine-learning methods for some regions. |
The evaluators then wrote a . Its three main concerns:
- Leakage from predictors measured at the same time as the outcome (, burned area, road density, and soil characteristics). They concluded that “the reported model accuracy overstates true predictive performance.”
- Not really “biophysical potential.” Regrowth happened where both nature and people allowed it, so the model learns a mix of the two, even without socioeconomic predictors ().
- The total looks too big. 215 Mha is far above the 31.6 Mha of regrowth observed in 2000–2016, and the paper reports no to reconcile the two.
They also raise , opportunity costs, and , which matter for any funding decision.
We present three things: research ratings, judgments about the claims, and measured replication results. They answer different questions, and none converts into another.
What each one is, and where its uncertainty comes from
- “50/100, 90% CI 40–60” is an overall rating of the paper, as a percentile against comparable research. The interval is the evaluator’s uncertainty about that rating. It isn’t a 50% chance the paper is right.
- The judgments about 215 Mha concern one claim, in each evaluator’s own terms. Evaluation 1 says low confidence: probably too low, but well below 678 Mha. Evaluation 2 gives a rough credible range of 80–180 Mha. They point in opposite directions partly because they target slightly different things: Evaluation 1 is thinking about biophysical potential with human constraints removed; Evaluation 2 about natural regrowth potential once errors in the input map are corrected.
- “71.7% forward accuracy” (section 4) is a measured result from the replication. Its uncertainty comes from the test design, the labels and the sample, not from anyone’s judgment.
How The Unjournal asks for claim assessments more detail
Alongside the structured ratings, we ask evaluators to identify the paper’s main claims and say how much they believe them. This is an extra ask, and we let evaluators express their uncertainty however they find natural. We don’t require a probability .
Here both said “low.” Evaluation 1 added rough bounds (probably above 215 Mha, well below 678 Mha); Evaluation 2 gave a rough credible range of about 80–180 Mha. Neither gave a probability, and neither gave a separate belief about the 87.9% accuracy claim.
4. What Fouilloux’s replication found
In short: the model reproduces well on held-out random points in Colombia (not used to fit it, but from the same areas and years) and does worse on new regions and later years. The suspected leakage predictors don’t seem to be what makes the accuracy high, or what makes the forecast worse.
All results below are from Fouilloux’s public write-up and its notebooks (robustness checks, MapBiomas experiments), checked on 9 and 12 October. Her own overall verdict is “partially supported.”
| Test | What it suggests | |
|---|---|---|
| Held-out random points, Colombia (not used for fitting; same areas and years) | 88.7% (paper: 87.9%) | Reproduces the headline number locally. By class: 85.3% of regrowth points and 92.0% of non-regrowth points correct. |
| Held-out regions (), Colombia | 85.4–86.9% | A little lower when test regions are kept apart from training regions. A range across block designs, not a confidence interval. |
| Held-out regions across the Neotropics () | 81.2–83.4% | Lower again: 81–83% when whole blocks of roughly 100–400 km are held out, against 89% for random points. A model trained only outside Colombia scores 77% in Colombia. |
| Same period, but independent labels instead of the paper’s () | 77.8% | Much lower than 88.7%, just from changing which map of “real” regrowth you score against. |
| : train on 2000–2012, predict 2012–2024, MapBiomas labels () | 71.7% () | Worse at predicting a later period: down from 77.8% with the same labels. Compare it with 77.8%, not with 87.9%; most of that bigger gap is the label change. |
| Forward test: predicted vs observed regrowth area, Colombia 2012–2024 | 7.4 vs 1.0 Mha | Without adjusting for how rare regrowth is, the model overpredicts the area about sevenfold. Adjusting for the share of regrowth in the training period gives 0.98 Mha, close to what happened. This bears on the method behind the area estimate, though I haven’t checked how the paper handled this for its 215 Mha. |
| Land cover from 1992 or 1999 instead of 2000 | 88.65% / 88.68% (2000: 88.67%) | Essentially no change from the 2000 baseline, on the same training and test points. Not literally identical; they differ in the second decimal place. |
| Refit without productivity, burned area and road density (same Colombia sample and labels; scored on same-period held-out points) | 89.3% (89.7% with them) | Barely changes. If these inputs were leaking the outcome, dropping them should cut same-period accuracy noticeably (why). |
The leakage question in more detail: what was tested, and how much does it show? more detail
The worry. Some predictors, such as vegetation productivity, were measured over the same years as the regrowth. If regrowth itself raises productivity, the model partly “sees the answer.” That would inflate test accuracy, and the information wouldn’t be available for a real forecast.
What would leakage look like? Call the model with the suspect predictors A1, and the model without them A2. If those predictors leak the outcome, A1 should beat A2 on held-out points from the training period (the leak is there too), but do worse than A2 when forecasting a later period (where it isn’t). And if nothing else changes over time, A2 should do about as well at forecasting the later period as it does on held-out points from the period it was trained on.
Why a same-period hold-out doesn’t catch this. The held-out points carry the same contaminated predictors as the training points, so leakage raises hold-out accuracy rather than lowering it. A hold-out guards against overfitting to particular points, not against inputs that partly encode the answer. That’s why comparing A1 with A2 within the period is informative.
What Fouilloux tested.
- A1 vs A2 within the period: 89.7% vs 89.3%. The suspect predictors add almost nothing, so they can’t be doing much to inflate the same-period accuracy. (The model’s own variable selection ranked them 10th, 11th and 17th of 17.)
- Back-dating land cover to 1992 or 1999: essentially no change (88.65% and 88.68%, against 88.67% with 2000).
- Forecasting with an A2-type model: the forward-test model used only inputs dated at the start of each period (2000 for training, 2012 for the forecast: forest cover, distance to forest, land cover), plus soil, climate and biome. It had no productivity, burned area or roads. It still drops from 77.8% to 71.7%. So the forecasting gap exists without the suspect predictors, and something else drives it, such as change over time.
What wasn’t run: A1 vs A2 both scored on the later period, which is the most direct version of the test above.
A separate risk: analysis choices (which variables, which settings) tuned, deliberately or not, to how well the model does on the hold-out data. That can also make same-period accuracy look better than real performance, and only a test on outcomes the modellers never saw, with clean inputs, guards against it. The forward test comes closest: its 2012–2024 outcomes were mostly not available to the authors, and it uses no suspect predictors. But it’s Fouilloux’s refit rather than the authors’ model, with different labels, and soil data stay in.
How much this shows. I think it’s fairly good evidence against this specific explanation, in Colombia. But:
- Other predictors could carry the same information, so dropping three doesn’t rule out leakage through others. And if three suspect inputs made it into the published model, others may have too; neither the paper nor this page has audited the timing of every input.
- Soil characteristics stay in every model, including the forward one. The evaluators’ joint synthesis lists them alongside productivity, burned area and roads as “recorded over the same time period as the outcome.”
- “Barely changes” is a judgment by eye. Nobody has tested whether a drop from 89.7% to 89.3% is distinguishable from noise, or how large a drop the test could have detected (its ). That’s harder than it looks: the test set has about 123,000 points, but neighbouring 30-metre pixels share soil, climate and land-use history, so they behave like far fewer independent observations, and naive error bars would be too narrow. For a sense of scale, in the blocked tests accuracy varies between held-out regions with a standard deviation of 2–4 points.
- It doesn’t speak to the other concerns: socioeconomic confounding, the size of the total, or the missing hindcast.
The net update. The evaluators’ concern that the accuracy figure overstates real forecasting performance looks right in direction: with the same labels, 77.8% within the period falls to 71.7% going forward, and accuracy also falls in new regions. Some drop is expected whenever a model moves to a new time or place. This one removes about a fifth of the model’s skill above chance (from 27.8 to 21.7 points above 50%). That’s a real but moderate loss; the bigger practical problem is the area forecast, which without adjusting for how rare regrowth is overstates regrowth about sevenfold. Their proposed cause, leakage from predictors measured during the regrowth period, looks less likely, at least in Colombia. The forecasting gap probably reflects changes over time or patterns that don’t transfer, though that hasn’t been tested directly.
Separately, the 88.7% vs 77.8% gap shows how much the answer depends on which regrowth map you treat as the truth.
5. What would a research graph need to record?
In short: about ten kinds of record, kept separate but linked. That way an overall score can’t quietly become a claim probability, and a Colombia result isn’t read as a global one.
What we mean by a research graph. Machine-readable records of research objects (papers, claims, studies, evaluations, replication results) with typed links between them, such as “this check tests that concern” or “this result weakens that explanation.” The point is that people and software can trace and weigh evidence without rereading every paper. In the VOWELS session, groups play a funder and write an algorithm that searches such a graph for evidence to act on. Here, the question is which records and links would let that algorithm see that the 87.9% claim was questioned, tested, and partly upheld. Examples: Science Live (including Fouilloux’s signed records for this replication) and the Open Research Knowledge Graph.
| Record | In this case | Should link to |
|---|---|---|
| Research claim 1: accuracy | 87.9% of 4.87 million held-out points (half regrowth) classified correctly; tropics-wide; regrowth over 2000–2016. | The paper and its authors, plus , place, period and test design |
| Research claim 2: area | 215 Mha of deforested tropical land with potential for natural regeneration (linked to 23.4 Gt C over 30 years). | The paper and its authors, plus how the area was calculated, place and period |
| Methodological claim (general) | Inputs that are affected by the outcome, or measured while it happens, make test accuracy overstate real forecasting performance (“leakage”). Relatedly, test points close to training points overstate accuracy in new places. | The methods research behind it, e.g. Kaufman et al. 2012 and Kapoor & Narayanan 2023 on leakage; Roberts et al. 2017 and Ploton et al. 2020 on spatial validation |
| Evaluation 1: low confidence, probably a lower bound but well below 678 Mha. Evaluation 2: rough credible range of 80–180 Mha. Nothing on the accuracy claim. Ideally both claims would get quantified beliefs (). | The specific claim and the evaluation | |
| Ratings | Overall 50/100 from each evaluator, with 90% intervals (Evaluation 2’s revised from 30–80 to 40–60), plus ratings on seven specific criteria (). The lowest are for “claims, strength, characterization of evidence” and for “methods”: 20 from Evaluation 1 and 5 from Evaluation 2, on each. | The paper, the criterion and scale, and the dated version |
| The 87.9% overstates how well the model would predict future regrowth from today’s conditions, because some predictors were measured during the regrowth period (). | The claim it questions, the evaluators’ reasoning, and the general methodological claim it applies | |
| Suggested check () | Refit using only predictors observed before the outcome period, and test on later outcomes (Evaluation 1). Compare several machine-learning methods (Evaluation 2). | The concern it would resolve, and who suggested it |
| Replication tests (one record each) |
| The code, data and scope, the specific check it carries out, and the concern it tests |
| Does the general critique apply here? | In principle yes: three or four predictors overlap the outcome period. In Colombia, the tests suggest leakage isn’t what drives the high same-period accuracy, though forecasting is still worse. | The methodological claim, the concern, and the tests that bear on it |
| Scoped interpretation | Forecasting concern supported; the specific leakage explanation weakened, in Colombia. | The test results, and what’s still uncertain |
| Decision question | Would funding natural regeneration here change outcomes, at what cost, and would it last? | The funder’s goal and the evidence still missing |
Why record the general methodological claim separately? Evaluators often apply known critiques, like leakage or spatial autocorrelation, to a specific paper. If the critique is its own record, with the methods research behind it, each application becomes a link plus a verdict: applies here, tested, mattered a little or a lot. Across many cases, that would show which critiques matter in practice, where (which fields, data and designs), and which further methods research would be most useful. It would also help improve methods systematically, rather than one paper at a time.
Evaluation 2’s revised interval is a small example of something a graph should handle: the evaluation summary keeps the original rating in a footnote, so the change is visible and dated. Later updates after the replication would need the same treatment, without overwriting the originals.
The JSON version optional
The JSON example holds the same records: claims (paper statements and any claim-specific assessment), paper_assessments (overall ratings and intervals), concerns (reasoning and suggested checks), reported_replication_tests (results with their place, period and design), and decision_context (the funding question).
The field names are my suggestion, not a tested import. Mapping them to Science Live’s existing fields would be a sensible next step.
Fouilloux’s existing signed records, and a possible small pilot optional
Fouilloux’s registry lists a published chain of (claim, study, outcome, citation), including a signed citation and a signed outcome.
A first pilot could link one evaluator-suggested check to one replication result, and add a dated note to the Unjournal evaluation package pointing to it. The original evaluations and ratings stay as they are.
The registry dates the chain to 4 October. The numbers on this page come from the live write-up as of 9 October; I haven’t checked that each one appears in the signed records.
Next: using this on Tuesday ↓
6. Using this on Tuesday
In short: pick a , then decide how each kind of evidence above should move that funder’s decision, and what’s missing.
- Pick a decision. For example, a restoration funder deciding where to support natural regeneration. Is the goal carbon, biodiversity, livelihoods, or something else?
- Weigh the evidence. How much should the overall ratings, the claim assessments, and the replication results each count? They answer different questions.
- Update. After the forward test, what should the funder believe about using the map for future targeting? After the leakage checks, which concern should be downgraded?
- Name the gaps. What would the funder still need? For example: whether paying for regeneration changes outcomes compared with what would happen anyway, how long the regrowth lasts, costs and local feasibility, and whether the 215 Mha total holds up.
Three questions to leave with
- Can someone trace a back to the evaluator’s reasoning, and forward to the test of that ?
- Can the graph say “this concern held up, this explanation was weakened, this is still open,” rather than forcing one verdict on the paper?
- What would need to be added before this becomes usable evidence for an actual funding decision?
A one-slide version: the two claims, one evaluator concern, and what the replication did to it (see the short version).
Workshop participants: questions, objections and additions are welcome right on this page. Select any text and choose “Annotate” to leave a Hypothes.is comment (free account). I’ll read and respond after the session.
Sources, comments and updates
- Williams et al. (2024), Nature · author correction
- Unjournal evaluation summary and joint synthesis · Evaluation 1 · Evaluation 2
- Anne Fouilloux’s replication write-up · her published-record registry
- IOSP assessment (VOWELS) session
Comments: this page has Hypothes.is turned on. Select any text and choose “Annotate,” or open the sidebar with the arrow at the top right. You’ll need a free Hypothes.is account. Public annotations are visible to anyone who opens the page.
I put this page together with AI help (for drafting, organizing and the page itself) and checked the numbers against the linked sources on 9 October. The evaluations and replication results belong to their authors. No new analysis was run.
Update history
12 October 2026, v2026-10-12.12: record table reworked: a general methodological claim as its own record, with methods references, and a link for whether it applies here; concern and suggested check as separate records; each replication test listed separately; all criterion ratings in a tooltip; more definitions. JSON updated to match.
12 October 2026, v2026-10-12.11: record table lists the two research claims separately and explains what a claim record holds.
12 October 2026, v2026-10-12.10: defines “research graph” in section 5; puts the forecasting drop in proportion; explains the missing power test; quotes the synthesis on soil; clearer wording in the leakage drop-down.
12 October 2026, v2026-10-12.9: explains why a same-period hold-out doesn’t catch leakage, and the separate risk of analysis choices tuned to the hold-out; clearer description of the drop-predictor refit.
12 October 2026, v2026-10-12.8: explains how the forward test relates to the paper; gives the exact land-cover-year results instead of “no change.”
12 October 2026, v2026-10-12.7: section 4 says the Colombia test points were held out but from the same areas and years; explains the Neotropics blocked and transfer tests, how the MapBiomas labels compare with the paper’s, and what “Result” measures.
12 October 2026, v2026-10-12.6: correction: Evaluation 2 did quantify its belief about the 215 Mha claim (rough credible range 80–180 Mha), and Evaluation 1 gave rough bounds (below 678 Mha); fixed in the table, the claim-assessment notes, the record table and the JSON. The “three things” box is shorter, with detail in a drop-down.
12 October 2026, v2026-10-12.5: clearer accuracy/precision example: says what the slider represents, moves the detail into a tooltip, adds a 2% preset.
12 October 2026, v2026-10-12.4: checked the paper’s Table A1.2 (precision and the 512 Mha area-adjusted estimate); rewrote the explanation of why 87.9% may overstate real-world performance; exact evaluator wording in tooltips; clearer suggested check; more links.
12 October 2026, v2026-10-12.3: new title; links to the paper in the header, short version and section 1; an invitation for workshop participants to comment via Hypothes.is (section 6).
12 October 2026, v2026-10-12.2: adds the forward test’s predicted vs observed regrowth area; removes the offer of an in-person introduction.
12 October 2026, v2026-10-12.1: responses to comments. Compares the forward test with the right baseline (77.8% within-period with the same MapBiomas labels, not 87.9%); explains what the leakage checks did and didn’t compare, including that the forward model had no suspect predictors; makes clear leakage was one of three main concerns, not the evaluators’ biggest worry; more links; uses Fouilloux’s surname; “about this page” moved to a tooltip; tooltips can now hold links, and 13 of them link to a reference (mostly Wikipedia).
9 October 2026, v2026-10-09.2: plainer wording, summaries and signposts for each section, more definitions, Hypothes.is comments. Corrected Evaluation 2’s interval to the current 40–60 (originally 30–80), here and in the JSON. Added the joint synthesis concerns, the Neotropics-wide blocked results, and the per-class hit rates; the accuracy/precision example now uses those rates. Flagged that soil isn’t in the drop test.
9 October 2026, v2026-10-09.1: first hosted version.