Yes, on an abstract level it is correct as well, although you kind-of loose all of the technical magic and some concepts do not fit this intuition (e.g. you can encrypt a message with someone's public key and post it on a public bulletin board - only the recipient will be able to decrypt it).

A similar nontechnical intuitive analogy could be of a padlock. You know, the kind which locks itself without a key, but you need a key to open it. The "public key" is such a padlock (in an open state). Anyone can use it to secure a message. Only you have the key to open it, though.

Bitcoin addresses, in fact, are public keys indeed (while the "wallet" is simply the corresponding private key).