In contrast to all the screaming going around, which I understand is a natural reaction to the way the change rolled out, I would like to add something constructive. Keep in mind that everything is easier to catch in hindsight. That does not excuse how poorly the change rolled out, but it is more constructive to identify some improvements that could help the process.

Random initial thoughts, before I dig into what problems that were observed and the ways they were fixed, and where the process could be improve.

• Don't announce success immediately on flip ;)
• Especially for newer features, some state monitoring is a must. I feel that this could have caught issues before they became realized. On the main net.
• We do not know the full extent of tests that were done on the test net, so I would hold off judgement before we know more. Besides, there are many reasons why the test net may not accurate reflect real conditions, as much as they tried to do so.
• Steemit will likely do a full postmortem with their knowledge which should hopefully clear up many gaps in my knowledge.

Massively Negative Mana

What happened? The switch flipped, and nobody could do a thing! It was quickly revealed that nearly everyone had resource credits that went to an extreme negative.

Immediately they identified the problem in what allowed the computations to become this way, but it required a costly resync of the nodes to recompute the mana correctly.

This is one of the worst types of bugs to happen, that cause a standstill and require a resync of nodes. Note that had the negative mana issue been caught, that only would fast forward to today's state where we patiently wait for RC costs to normalize, which may have been more palatable.

As I've understood this, the RC balances of each account could have been inspected on the main net before the cutover time, and that would have immediately pointed to a problem with the mana calculation. Of course monitoring all accounts is probably not feasible, but a random sampling would have sufficed. Please correct me if I am off base here

Small Note About Detection

There has been word that the negative mana issue was present in the test net, and people are all up in arms about it, but I would like to raise the counterpoint here: the team could not determine if it was an actual issue, or if it was due to the firehose of real data streaming into the testnet, which does not accurately account into pricing. So cut them just a little bit of slack here. There are plenty of reasons to dismiss a weird state in the test net. Perhaps this should have been a warning to check the values in the main net though. Again, hindsight is 20/20.

Voting Power

There was another bug where the voting mana did not preserve the existing voting power on the cutover. I'm not entirely sure if this could have been caught, but if the voting mana was also being computed along the way it may have been auditable.

There are many posts about the effects of this. In case you do not know already, after the one time penalty, voting power will behave as usual. Sucks to lose that amount, but there's no going back.

Delegation Behavior, or Powering Up

See this reported bug by @yabapmatt. This is something I also encountered while trying to help someone boost their RC, and was disappointed that it didn't help immediately (except in the regeneration rate).

This, while not as minor, should have been a simple test case on the test net. That said, it's easy to spot now that it was observed. Expected RC behaviors though should be clearly outlined somewhere and checked off in terms of "things we could have tested on the test net".

The same thing happens when powering up. Yes, the power up costed me some mana, but it also didn't increase my existing mana either. I saw a PR to address both issues.

RC Costs Being Too High

Yes, we knew that it would take time for the market to stabilize, but it seems that the equilibrium might not be working very well. We do expect this to be usable by accounts of all sizes, and indeed there's a change that adjusts parameters to account for this here. This seems to need a resync? Though I'm not sure.

Summary

All in all, the problems are being identified and addressed. I feel that there are a few sanity checks that could have caught the really bad stalling, but it is also a bit of speculation here. My post lists a few of the issues that I was tracking, but the main take-away is this:

• Monitoring the state pre-flip for the new features would have caught the issue before it became a problem.

Of course the test net could also be better, and I'm sure the testing facilities will only improve over time. I cannot comment on how it can improve since I don't know the details of it. Other than I was able to make a test post in the test net just fine.

We will get it to a workable state. Just be patient, as the friction now I believe will pay off on the long run. Having a better system for bandwidth is a good system level change in my opinion, as those of us that experienced the bandwidth hammer know (and promptly forgot about).

Oops this isn't really a summary section at all. Ah whatever. And also keep in mind that I'm a back seat driver, so I could be full of shit too. This is just what I have the visibility to observe after all.