The Frustration of Fantasy Taxonomies
Something that has always annoyed me about fantasy worldbuilding is inconsistent taxonomies of creatures. Buckle up for this, because there's going to be a lot of "well, technically"s and a lot of scooting glasses up the bridges of noses while checking pocket protectors. The first time I remember a fantasy taxonomy irking me was in J.K. Rowling's magical Earth. Because Rowling never really explains what magic is, it's unclear why (for example) a hippogriff is considered magical and therefore hidden from muggles by wizard law. The hippogriff has no magic powers (except, possibly, that it is capable of flight despite being stupidly huge). So what's magic about it?
And then comes the immersion-breaking truth: the cause-effect relationship is backwards. The wizards don't consider the hippogriff to be magical for any wizardly reason. Rowling has to have them classify it as magic because if they didn't, it wouldn't be hidden, and it would ruin the suspension of disbelief she wanted to build for her readers about the series taking place on our Earth.
Her merfolk and centaurs are no more explicitly magical than humans, but they're hidden, too, supposedly at their request. (Really? Every single centaur wants to be forced to stay away from muggles in the woods? None of them want to hang out and enjoy modern technology? Not one? )
But D&D has had far more nuanced problems with classifying creatures. It, too, suffers from the problem of segregating Earth-extant and non-Earth fauna, albeit inconsistently. Was there any particular reason that a naturally-occurring stirge, ankheg or flying monkey was a "Magical Beast" in 3.5E rather than an "Animal" or "Vermin?" None of those creatures really have magical properties, and they all potentially existed as natural parts of their ecosystems. They just happened to exist in D&D worlds and not on Earth.
So why can druids only turn into Earth "Animals?" This problem persists into 5E, where the distinction is between "Beasts" and "Monstrosities." What exactly about a griffon or a bulette-- creatures no more or less magical than a giant owl or a rhinoceros-- prevents a druid from using nature magic on them?
It's terracentrism at work. The taxonomy governs actual game rules and is therefore a True Categorization which dictates definitive effects on the world (as opposed to our real-world scientific classification system which is imperfect but tries to derive classifications from observed traits). But it only really makes sense if you're thinking about what does and doesn't exist on Earth. Which is dumb. Earth should be irrelevant to Faerun or Eberron. To quote Arnie Niekamp, when I'm thinking about fantastical worlds, "I don't want to talk about Earth stuff."
As a side note, lots of prehistoric megafauna-- like the huge dragonfly meganeura and the truly horrifying avens magnificens-- would probably be too big to fly on modern Earth because the atmosphere today is less oxygen-rich and therefore less dense. They needed that extra buoyancy. But the 3.5 pterosaur was an Animal, rather than a Magical Beast, despite facing similar logistical problems to a pegasus or hippogriff.
Speaking of terracentrism, the whole taxonomy is also hugely anthrocentric. The rules in both 5E and 3.5E-- which, I remind you, are incontrovertible facts of the game world, and so ingrained as to be quite difficult to expunge through house rules-- refer to the elder races of elves and dwarves as "humanoids." Given that in most settings the elder races precede the existence of humans, and that they live longer, and are just generally more powerful and cultured, this seems a little ludicrous. Humans should be Elfoids.
Meanwhile in 3.5, kobolds-- tiny lizard people-- are also humanoids, but lizardfolk-- who are also lizard people but are actually human-sized, are "Monstrous Humanoids." Minotaurs, who are mostly just big humans with weird heads, are Monstrous Humanoids. Merfolk, who are humans who have gills and fish tails, are regular Humanoids. Satyrs, who are humans with horns and goat legs? They're "Fey."
D&D 3.5E made a much bigger deal of the distinction between Humanoids and Monstrous Humanoids than 5E does (a feather in 5E's cap). Many spells, like Charm Person, specified that they only worked on Humanoids-- so you could charm that small lizardman or an orc, but not a medium lizardman. The medium lizardman had some nondescript game-ascribed trait that made Charm Person not work on it. It was intuitive that Charm Person didn't work on a non-person... but, as you can see, the difference between humanoids and non-humanoids was not intuitive, which meant that players had to metagame and memorize individual creatures to know whether their spells would be effective or fail for reasons not apparent in the fiction.
Part of the problem with D&D creature types is that D&D attempts to make multiple real-world myths and cosmologies fit together into one giant hodge-podge, and expects them to play nice with one another.
Take the Fey creature type, for example. On the one hand, Wizards of the Coast clearly wants Fey in D&D to mean "spirits of the natural world," which is a mythologically valid definition. They put Greece's satyrs, nymphs, nereids and dryads into this category, because these are various classifications of divine/supernatural nature-beings.
But there are other components which leaked into their Fey label, and the reason is that "Fey" has other mythologically valid definitions: "nobility of the otherworld," "trickster spirit." The Celtic concept of what fairies were shifted over time and culture and fairies are associated with both ideas (this is more common than you might think-- humans tend to go from worshipping natural forces to worshipping human-like forces as they advance). So we also wind up with the Eladrin (technically "Outsiders" in 3.5), the Joystealer, and the Master of the Hunt. Meanwhile, many of the creatures actually associated in our real-world mythology with the Celtic fairies, like Fomorians, became "Giants" instead. Giants who, let's sigh, live in the fairy otherworld.
The Fey label also got the Native American Wendigo myth-- Wendigos being shapeshifting, sorcerous giant-spirits that compelled people to cannibalism. The idea was that they were the spirits of frozen wilds, but ultimately, the original myth has more in common with the late European idea of a demon-- a spirit that possesses and compels humans to sin-- than any of the other fey creatures.
The whole mess was so confusing that, when Wizards of the Coast made the 3.5E Spirit Shaman class, they had to devote an entire sidebar to defining what a spirit even was-- despite "spirit" being a more or less universal cultural concept. What's worse, that strange little matrix of types and qualifiers still left out some obvious contestants, like demons and devils in ethereal form.
Part of the problem is that classifying things into hierarchies is a very logical, modern mode of thought, and sits at odds with the awe of monster myths.
In the modern day, the labels "zombie" and "vampire" both fall immediately and neatly into the category of "undead," and have a number of traits, weaknesses, and subdivisions associated with them. We think of a "werewolf" as being in a totally different category. But the truth is, werewolves and vampires come from the same myth. Some slavic peasants thought they saw someone who'd died in the mist one night and freaked out. Later they saw a wolf acting weird and assumed it was related. To them, it was all the same phenomenon. There was no list of canonical vampire powers or weaknesses. If you can come back from the dead, why wouldn't you be able to turn into a wolf, control the weather, and be responsible for everything else that goes wrong from now on? The vrykolakas might be able to do anything.
Monsters are captivating and cool and terrifying because they connect us to the numinous unknown. But in the modern world, one of the ways we protect ourselves from the unknown is by making lists and taxonomies, so we feel like we've mastered it. Classified the threats. And that disconnect is part of where we go wrong when trying to classify types of magical creatures-- magical thinking, by nature of its connection to the unknown, resists rational understanding. Mythic thinking is even more evasive.
So if your fantasy aims to emulate myth, which D&D certainly seems to want to do at times, perhaps the answer is to stop thinking about the magical world from inside your reason box-- the part of your mind that, familiar with natural selection, demands that creatures must maintain a breeding population, must not exhaust the local ecosystem, and must be well-adapted to the world they live in.
Greek myth makes an elaborate taxonomy of its supernatural beings, tracing their divine lineage, but it also keeps track of each individual member of these races. Their children inherit traits but rarely keep to speciation. The Greeks didn't know about evolution, they thought that the gods had made the world wholecloth and sometimes grain turned into mice.
Of course, the Greeks more or less invented taxonomy. Other ancient cultures didn't categorize monsters at all, really-- their monsters were one step away from godhood, and they were unique. Gilgamesh's Humbaba wasn't any type of fey monster, he was just the divine guardian of the cedar forest. Grendel wasn't a hill giant, he was a Grendel. Where'd he come from? A Grendel's Mother. Where'd she come from? Who knows! Probably a Grendel's Grandmother, who is so scary Beowulf would rather fight a dragon.
As unique, named beings, monsters can retain some of the terror and awe they hold in myths and legends. As members of a race or species, with a thought-out ecology and biology, they can still be fantastical, but the kind of fantasy they'll evoke will be closer to science fiction about alien worlds than to myth. You'll have put them in a modern, rational context. Is there a middle road? Absolutely. Do both avenues lead to powerful storytelling? Sure. But unless you are aware of the distinction, you can't consciously use the genre you want to.