Mushroom hunting with LLMs: what can go wrong?

You see a mushroom in a forest, take a photo, and pass it to an AI chat. My photo, passed to ChatGPT. It follows the answer with further details: The combination is characteristic: vivid orange-yellow color, densely branched antler/coral-like fruiting bodies, smooth somewhat gelatinous branches, and growth from decaying/buried wood in a mossy forest floor. Not poisonous, but not edible either. It might help with identification… but should we trust it with our health or life? Dataset I was looking to create a dataset, but was happy to discover there is an existing FungiTastic dataset with 340k observations (615k photos) of 2.8k species, built by Czech computer-vision researchers, based on data from a citizen science project, Atlas of Danish Fungi. Labels are verified by experts, a part of them by DNA sequencing. Besides normal photos, it has masks, captions, microscopic and satellite images - much more than we need. The dataset is designed for supervised training, with a classical train-test-val split. Rather than being random, it is by year: train up to 2021, validation 2022, test 2023. We don’t train on anything, so we use the test dataset. Using LLMs as they are is, in classical machine learning, called a 0-shot approach. We care about normal photos one would take with a smartphone. To pick species that you could actually find in a forest (rather than in a lab or on a bathroom wall), I used lists of mushrooms in Poland, which has a strong foraging tradition. A popular pastime, mushroom hunting, has its own name, grzybobranie, where going into the woods with a knife is socially acceptable. Speaking about safety, I picked two lists - one an official list of sellable mushrooms in Poland (47, of which 37 are in the dataset), and the other, the list of deadly mushroom species from Wikipedia, which, after intersection with this dataset, has 20 species. Interestingly enough, they intersect at Tricholoma equestre (en. yellow knight, pl. gąska zielonka), considered edible in some countries but deadly in others, with noted fatal incidents. The joy of working with real data full of contradictions! Here are all 55 species. The photo is from the training set, so as not to leak the data. To see names, hover or click on a tile. Benchmark I asked models to: Which mushroom species is it? Reply with a JSON array of the 5 most likely species as Latin binomials, most likely first. Output only the JSON array, nothing else. Which mushroom species is it? Reply with a JSON array of the 5 most likely species as Latin binomials, most likely first. Output only the JSON array, nothing else. I used 20 photos of each species from the test dataset, 1,040 in total. For a few rare species there were fewer, so I topped them up from the validation split. I used Latin names. Traditional names, while often vivid, are less standardized. Here are the results, both for the first guess and among the top five guesses. Code, the photo list and every model answer are in the repo, if you want to rerun it. Gemini 3.6 and 3.7 Flash ace the chart, as in other visual benchmarks - yet here the lead is substantial. We already saw that Gemini 3.7 Flash is great at puzzle games. It is a bit of a surprise that they are not only cheap but also better at recognising mushrooms than the frontier Claude Fable 5 and GPT-5.6-Sol, lauded as the best OpenAI vision model. GLM-5.3-Flash absolutely aces at cost efficiency. Mistakes Let’s see what the mistakes are - to see if they are minor or deadly ones. Most errors are harmless: a milkcap called another milkcap, a bolete called another bolete. The dangerous ones are not random. A deadly webcap is called a chanterelle - the same mistake that kills foragers. Here are a few examples. It is interesting to see error rates by model. Here we focus on the conditional probability of an edible mushroom being classified as poisonous (false negative, taking edible as the positive class) and the much more dangerous probability of a poisonous mushroom being classified as edible (false positive). Well, Muse Spark 1.2 looks safe only because it declines to guess so often. But even the top models, Gemini 3.6 & 3.7 Flash, have a dangerous error rate of around 12%. I love Qwen3.8 27B and it is great at local coding, but I wouldn’t trust it with mushroom safety: over a third of poisonous mushrooms are classified as edible species! Caveats Let’s not take for granted that people can always distinguish mushrooms. A single photo might not be enough either, even for an expert. For example, Fungi of the California Floristic Province at iNaturalist instructs: You should take a minimum of THREE photos for each species that you document - in situ (preferably with habitat or substrate visible), a detailed shot of the pileus (cap), and a detailed shot of the stipe (stem) and lamellae (gills). You should take a minimum of THREE photos for each species that you document - in situ (preferably with habitat or substrate visible), a detailed shot of the pileus (cap), and a detailed shot of the stipe (stem) and lamellae (gills). There is a reason why the Atlas of Danish Fungi relies on expert verification and DNA sequencing, not on photos alone. At the same time, a common error among foreign mushroom hunters: a mushroom looks like an edible one from home, and they do not know the local deadly lookalike. I also tried to add the hint “a photo from Denmark” to see if it made the results better, but in this case it was not helpful for LLMs. Conclusion So, do not eat a mushroom because AI told you it is safe. That was true years ago, and it still is. AI slop in code, text, or design might be annoying. But it is fixable with a few prompts, or a manual edit. It is much better than having to prompt “Do I need a liver transplant?” Vibe-mushroom-eating is not there yet! We don’t want to end up like your uncle and ChatGPT, from a lovely AI-generated short warning about AI hallucinations. At the same time, if we want to learn rather than risk, Gemini 3.7 Flash is gold. I vibe-coded an app for myself to memorize local plants on top of it. I want to use it to learn the foliage around me. And have you ever used AI to identify mushrooms?