ARK Core: Testathon Summary for Core v2

arkecosystem (61)in #arkecosystem • 5 years ago

ARK Core Testathon has ended after a full week of testing! Here is a recap
including our participants’ findings and developments made by the team.

Our first Testathon was a great success! Multiple bugs were discovered and
subsequently fixed in the PRs. This discovery and correction process gave our
core dev team many great ideas for future improvements. Here are a few of the
most notable issues and fixes from this week-long event.

One of the first things tested was mass spamming and breaking of the network
(special thanks to Marc for helping with this part), revealing the need to
improve how the transaction pool handles incoming transactions. Initially, every
transaction was accepted into the pool increasing the time to verify new
upcoming transactions. This increased load resulted in forgers missing blocks.
After improvements, only transactions that meet a nodes criteria to enter its
pool are included. All other transactions are further broadcasted until accepted
by a node. If no nodes accept the transaction it gets dropped. Following
implementation of this fix, the network was spammed by our testers who sent
several thousand transactions at a time and the network performed brilliantly.

A GitHub issue was reported by user *wownmedia *relating to a check done on the
VendorField for its length. This check was not being performed properly for
special UniCode characters. Further investigation from our team resulted in
implementing a proper check on the VendorField and stricter validation on the
length of the field.

During the Testathon we have also found state machine inconsistencies in some
edge cases that our Core team has been working on resolving over the past few
days.

Participants

We would like to thank each and every person who helped test the new Core — be
it reporting issues or, providing solutions, helping with running devnet nodes
or just providing moral support in the channel. Thank you all (not just the one
listed for bounties)!!

Eligible submissions for the Testathon bounty are as follows:

Github User : paroxysm

refactor(core-tester-cli): use full Ark
amounts

Github User : Dated

Github User : JeremiGendron

Github User : zillionn

Github User : thomas-neuman

fix(core-webhooks): include webhook ID in update
schema

Github User :wownmedia

VendorField doesn’t accept 64 unicode
characters

All of the above go into the random drawing for the ARK Branded Ledger Nano S
that will occur on the November 15th at 4PM CET.
(http://www.randomresult.com/ticket.php?t=971229VWNR3)

What was happening before and after the Testathon?

Before the Testathon and following its completion our team continued to improve
the Core, focusing on issues discovered prior, during and after testing periods.
Here is a quick summary of the biggest recent changes:

Refactored fork handling (changes include peers grouped by height, refreshed
after a forking and checked for common blocks).
**Peer ban after fork **(when entering a fork due to a bad block from a peer,
that peer is now correctly banned. Previous bans were too short. Additionally,
all peers are refreshed and those without common blocks are now suspended).
**Transaction ping and rebroadcast **(removed the isBroadcasted flag and
instead introduces a transaction ping counter which is incremented
whenever
a transaction is received from a peer while it is in the transaction pool. While
the ping count is 0, we assume the transaction was not properly broadcasted. On
startup all transactions from the pool where the ping count is 0 will be
rebroadcasted).
Reverted not matching dynamic fee state (before this change even if the
transaction was not accepted in the pool, the sender state was applied — which
in turn resulted in the wrong state of the sender and denying of later
transaction).
Avoid that transactions below the minimum fee enter the pool (if fee doesn’t
match the recipient pool, it is not allowed to enter the pool. Transaction will
be broadcasted to other peers).
Added update counter (previously we would always check the peers when
missing blocks, this change introduces a counter to only check if we keep
missing blocks consecutively. It should take care of unwanted rollbacks and
other issues).
**Properly verify the payload length of blocks **(the payload length was checked
against undefined, which would always result in false and allow blocks to be
broadcasted that are bigger than the allowed size. You could for example forge
blocks that contain more transactions than the allowed maximum).
Added SSL support (added configuration options for SSL for your node to
serve API data via SSL connection).
**Only call checkLater if not called yet **( When the blockchain is ready it
callscheckLater to wait 60000 ms before syncing with the network again. If a
forger wakes up a relay to sync with the network it will end up issuing another
checkLater. Over time the checkLater calls stack up and ripple since each
checkLater ends up callingcheckLater).
Improved behaviour when network is missing blocks (removes unnecessary code
and improves how missing blocks are handled. Also changes that more offences
stay banned until expired when recovering from a fork).
Correctly update lastDownloadedBlock (the lastDownloadedBlock was not
correctly updated in manageUnchainedBlock and thus effectively prevented the
chain from advancing under certain circumstances).
Added Core Vote Report package (adds an optional package to generate the
popular vote report
https://explorer.ark.io/VoteReport.txt
based on in-memory data which will show real-time data).
**Added Core Elasticsearch package **(adds an optional elasticsearch integration
to provide more powerful searching capabilities for large datasets).
Made shutdown more graceful (The exit handler now emits a shutdown event
to stop the blockchain. Fixes the blockchain so it can be stopped from all
states, so it really stops processing blocks when requested, waits 1 second to
let a round properly finish, with the more consistent shutdown, a table rebuild
is only triggered when the wallets cannot be verified. A rebuild can take a few
seconds on devnet, while the verification only takes a few hundred
milliseconds).
Make sure all numerical inputs are positive (in some instances it was still
possible to pass in negative values which shouldn’t be possible).
Require Node 10 as minimum engine (with implementation of ARK’s snapshotting
system we made a minimum requirement of running NodeJS 10+ for ARK Core).
Added Hapi Rate Limit plugin (Added as
hapi-rate-limit
rate-limiting/throttling plugin.
**Stop plugin setup on container shutdown **(when the exit handler is running it
was still possible that new plugins would be registered).
**Reduce number of loops when reading/writing transaction bytes **(Writing
transaction bytes in Block.serializeFull requires three loops and two loops when
reading. Both are replaced with a single loop which achieves the same).
**allow P2P config without headers **(allows the P2P /config endpoint without
the need for the port, nethash or version endpoint).
Stateful data storage (introduces a new storage for the blockchain state. It
caches the last processed blocks (100 by default). getCommonBlocks and
getBlocks have been changed to make use of the cache and they will only
fallback to the database if necessary).
Stricter API validation (added stricter API validation to make it more clear
to the user what parameters are allowed and what they should be).
**Don’t accept forger as peer **(added an additional header which is used by
forgers so that they don’t get erroneously accepted as a peer only to end up
getting banned).
Use 1e8 instead of math.pow (Replaces Math.pow(10, 8) with 1e8 as it is
quicker to type and yields the same result, just a syntax change).
Handle unexpected transaction errors in guard (invalidates a transaction
when an unexpected errors occurs instead of returning a code 500).
**Added rate limit offence **(added a check for status 429 (rate-limit) in case
that code ever comes up with a peer).
Added ability to skip initial peer discovery (Starting nodes with
--network-start will already skip the peer discovery but it will also skip a
bunch of other things in regards to the state-machine. The --skip-discovery
flag will allow a node to be started in normal mode and skip the initial peer
discovery like --network-start does without any of the side effects).

We have also started to implement stricter coding rules for Core and added new
plugins to further enhance our Core foundation including:

Stricter JavaScript styling rules (following AirBnB
specifications).
Implementation of the Flow** **into the Core, which is a
static type checker for JavaScript that identifies problems as you code and
catches bugs that might otherwise go unnoticed (not only saving time, but
helping uncover bugs).
Setup Snyk** **to stay on top of known security issues and
vulnerabilities in npm modules.
Added support for error tracking and monitoring via
Sentry** or
Bugsnag
**so node maintainers can provide better feedback if they decide to use a proper
monitoring service for their node.

And now...

For the one thing everyone has been waiting for… an announcement to
announce an announcement. With final testing, pending improvements and fixes
that our Core team is currently doing as you read this, we can say with
confidence that next week we will announce Core v2 MainNet release date— stay
tuned!