You are absolutely right.
There are a lot of self-help talk about getting out of debt and those same concepts are applicable to building your stack. Those debt reduction plans often discuss cutting out unnecessary expenses, such as that high priced coffee, wasting gas money driving to lunch, wasting money on lunch, saving money by bringing a lunch and beverage. Don't forget that you probably could drink that coffee at work, even though it does not have a fancy name.
After you have eliminated your debt, now is time to build your wealth with those same plans and use the money on the silver/gold stack.