Yes, for 90+% of use-cases I agree with you.

However, for performance oriented cases:
it is still better to not create an empty list and not to use it's auto-scale if possible (e.g. if you know for sure there are never more than X objects in it). The core reason of this is .NET Garbage Collection.
The more instances it has to collect, the longer the GC pauses are. Also the more memory areas it has to move from Gen0 to Gen1 or has to consolidate, the longer it takes.

If performance is really that critical, then it might be better to go with Span from the beginning, a good blog post about it can be found here: http://adamsitnik.com/Span/
Anyway, always do micro benchmarks for segments where you want to optimize performance in that level. And avoid premature optimization, it's often better to have a function up and running with tests, before doing them.

Also 2 things to be aware of with Lazy<T>:

• Don't use it for short-lived instances, since by default it's using Activator.CreateInstance, which is known for bad performance.
• if a factory is used and it's initial initialization throws an exception, all following calls throw the same exception without re-executing the initialization