I don't know if I'd call it a lack of agency, but I definitely feel like there's not much momentum, it seems like most of the character's "turns" are equivalent to "tell me more about that" or "I carefully skirt that situation" so there's not much to play off of or actions to react to. I don't know what the actual rules look like, but I suspect that more participants and more of a high-energy pace of comments are part of the game's expectations -- if more people are clamoring to get some input in as the world then opportunities to embellish or expand are probably more easily absorbed, plus with each additional participant there's a greater chance that they'll find something inspiring to riff off of. Since some of my responses have been more in the vein of "I might as well say something, to at least keep things moving" I've found it a little frustrating.