I agree scraping 100's of websites doesn't make sense, but the pushback I'm getting from my team relates to the lack of a clear standard and the potential to have to support every backward compatible solution we introduce from here until whenever. If more people use this, will they also continue to support the website based bp.json file? If a full standard is developed in the future which puts this data directly into tables instead of as json, will that then be a third approach (with two legacy approaches to support)? That, I think, is the primary pushback I'm hearing from my team. A desire to plan out good specifications first, before introducing temporary solutions which will then need to be supported in order to not break applications which start to rely on them.
Unless we think the bp.json approach (both on the web and stored on EOS) is the solution the EOS community should be using for quite some time? I mostly agree with you that this is a good improvement on what is being done now, but I'm doing my best to relay concerns my team has to hopefully find some common ground. Maybe it's okay that app developers just have to change things as older methods go away naturally, but I think some prefer more structured, planned approaches and assurances that when we introduce a new approach, we've thought through if this is the best solution for the problem we're solving.
If this isn't for node discovery, then the current bp.json hosted on websites were were provided via listproducers still seems to be needed. If that's the case, is this data duplication?