One of my favorite courses I took in graduate school was a course entitled The Psychology of Dreams. We read Freud's Interpretation of Dreams (A fascinating genius of a book and author, imo; definitely worth a look). One of the aspects of dreams we learned was the biological approach to dreaming.

Here, what you have described is often categorized by psychology as Hypnagogic hallucination. Hypnagogia refers to the state between wakefulness and sleep (Greek: agogos: lead (to); Hupnos: sleep).

The body is an amazing mechanism. It knows that even though you "think" you are "awake" (i.e., conscious), you are "really" sleeping. So during this dreaming sleep-stage, often called REM, your body actually paralyzes itself. It does this so when you think you are, for example, "fighting a huge purple monster with fangs," you don't actually swing and whack your partner who's sleeping beside you in the head (ever happen to anyone?). It's a safety mechanism.

What people think happens is, sometimes there's a "short"--could be environmental, could be physiological--but something happens to where you get "trapped" between stages, and your mind thinks you're awake, but your "body" (or, at least certain parts or systems within your body) thinks you're asleep!! So, you literally are in a sleep paralysis! This is also why sometimes we can feel that sensation of "falling," then we are abruptly awoken? That's it! Except there, the mind/ body "connection" is working properly and you "wake up--" with the paralysis or "night terror," you don't... The "jerking" feeling is literally your body "falling" asleep--i.e., into a dream-state where your body is in "reality" immobilized.

Of course, this is how we can get into "sleepwalking," and such: people doing wild things when asleep, even dangerous things.

On the subject of "lucid dreaming." Because science can, to some degree, "measure" the different stages of sleep, it is also possible to demarcate and even "influence" these various stages in different ways. For example, because we know it takes X amount of time, typically, to get into that "hypnogogic" state, or to start dreaming, a person, in theory, could craftily set an alarm to "wake them up" at a certain time, with the idea being that they may be still in that sleep-state. Or, another way to do it, is something you yourself, @artisticscreech, seem to already be honing in on--You start to "teach" yourself that there are certain "markers" within the dream that designate the fact that you are "dreaming;" ones which "only you" recognize. A good object to begin with (from what I hear) is actually your own body, something easy to see, like your hands. So you would start trying to focus on your hands in the dream, eventually learning to use it a s a marker that you are dreaming, and then you go from there. To that point, though, you touched on something else very interesting. This idea of the dream "knowing" you "know you're dreaming," and "not liking it." That is very true, I think. ( Perhaps the Nightmare or Elm Street series and Freddy Krueger knew something we didn't ;)) ...