ARK Core Upgrade To V2.4 — Introducing P2P WebSockets

ARK Core continues to evolve! For the 2.4 upgrade we focused on Peer 2 Peer
(P2P) layer switching from API to WebSockets. P2P is a communications layer
where interconnected nodes (“peers”) share resources amongst each other without
the use of a centralized administrative system. It is how nodes that run Core
interact with each other on the ARK Public Network. The switch to use WebSockets
brings better performance, higher resistance to some types of potential attacks
and real-time exchanging of information. This is only one of the changes in this
release. This post will cover the most significant changes in today’s upgrade.

In addition to Core v2.4 upgrade, there is a new Desktop Wallet update available
as well,
v2.4.1.
Please update your wallet to properly interact with the Core v2.4 update. There
will also be larger Desktop Wallet update coming soon, so please stay tuned.

**Download newest Desktop Wallet v2.4.1 that supports Core v2.4 at:
**https://wallet.ark.io

What’s New in Core v2.4?

Since the biggest change was the switch to WebSockets protocol, lets go over the
reasoning and benefits behind this first. Before the switch, P2P communication
was done via API (HTTP requests). Whenever nodes made a request (blocks,
transactions, wallets, …) a connection was opened, data was transferred, and
then after completion the connection was closed. This continuous opening and
closing creates a lot of overhead, and for ARK’s Core that relies on constant
responses or real time interactions, this isn’t the most viable long-term
approach.

WebSockets is a very flexible protocol for transferring data via P2P. Unlike
API, the socket that is connected to the server stays “open” for communication.
That means data can be “pushed” to the other node in real-time on demand,
without creating additional requests.

We chose Socketcluster**, **which is a pure
WebSocket framework solution, to help us with P2P WebSockets implementation.
Some of the features it supports:

• Scales linearly as you add more CPU cores and workers.
• Scales horizontally across multiple machines.
• Resilient on both the client and backend — Process crashes, lost connections and
  other failures are handled seamlessly.
• Supports custom codecs for compressing messages during transmission.
• Supports both pub/sub channels and direct client-server communication.
• Authorization via middleware functions (access control for emit, publish in,
  publish out, subscribe and handshake).
• Client sockets automatically reconnect (by default) if they lose the connection.
• Designed to work alongside any database/datastore.

Another new addition (which was scheduled to be implemented in a later version)
is initial implementation of parallel block downloading and batch block
database inserts when syncing. Before v2.4, while syncing, nodes could only
download chunks of blocks from a single other node of the ARK network. With the
introduction of parallel block downloading, we have increased this number to 25
concurrent connections to different nodes throughout ARK network. This means
instead of only using a single node, it now downloads blocks simultaneously from
25 different peers. This not only speeds up the syncing process, but it also
levels out the network traffic by spreading it across multiple nodes.

The transaction pool has been improved to handle faulty data more accurately and
to make it even stricter for bad or malicious data to enter the pool, making it
more efficient and safe.

The JSON-RPC we offer, formerly known as Core-JSON-RPC, has received a rework
to turn it into a real RPC that is easier to use and maintain. The biggest
change is that it now offers programmatic use to make integration into ARK Core
easier while simultaneously allowing it to run as a standalone application
detached from a relay.

Various improvements have also been added to ARK Deployer, making bridgechain
deployment even easier and more user friendly. We have implemented Explorer as a
Core plugin and added a new Wallet API that caters to the needs of developers
and ease of integration in end user products.

We have also implemented replay command to verify blockchain data from block 0,
making it easier to do verification of all data in the database the same way it
is done during sync, but without storing the data again (replay blocks in-memory
from the local database).

What’s Next for Core?

While we just released v2.4, our v2.5 upgrade is almost complete as well and
will go live for testing on the public development network in the upcoming days.
This won’t be as tremendous of an upgrade as v2.3 to v2.4 was, but will focus on
deprecating API v1 support. You can read more on this
here**
— **if you are still using API v1 for anything, make sure to update to API v2 as
soon as possible. Numerous improvements will also be made on making the code
leaner, removing obsolete (legacy) code, and making it even more developer
friendly.

Our next major change will be v2.6, for which we can be certain will be THE
BIGGEST and most VALUABLE upgrade of the ARK network — AIP 11 (new transaction
types), AIP 18 (multisignatures) and full AIP 29 support (easy custom
transaction type integration) are just some of the major improvements coming to
the network, but since we don’t want to spoil everything in this blog post we’ll
cover this in one of our upcoming blog posts. Stay tuned!

All Changes From v2.3 to v2.4

*For node operators of the current ARK Core, in order to update, please follow
these migration instructions (v2.3 to v2.4) before you do the update: *
https://docs.ark.io/releases/v2.4/migrating_2.3_2.4.html

Changes In Numbers

• **11 different developers **contributing to the Core.
• 234 new commits to the Core.
• 1,049 files changed in the Core.
• **47,980 code additions **to the Core.
• **37,603 code deletions **in the Core.

As we move towards 2.6 and the completion of AIP11, AIP18 and AIP29, there will
be various breaking changes. The main concern of most developers will be
breaking changes to @arkecosystem/crypto, so go through the commits listed
below and make sure you adjust everything in your application that is affected
by a change.

Breaking Changes

• Implement WebSockets with SocketCluster
  (#2273)
• Switch transaction expiration from seconds to chain height
  (#2461)
• Implement Block.fromHex, Block.fromBytes and Block.fromData methods in
  @arkecosystem/crypto(#2377)
• Implement BlockFactory in @arkecosystem/crypto
  (#2429)
• Implement TransactionFactory in @arkecosystem/crypto
  (#2437)
• Purge invalid transactions after a milestone change
  (#2499)
• Make transaction amount required
  (#2574)
• Always use crypto identities to work with keys and addresses
  (#2443)
• Enforce BigNumber for satoshi based values
  (#2391)
• Move in-memory wallet logic to core-state
  (#2489)
• Replace bignumify with Utils.BigNumber.make
  (#2416)
• Replace Joi with AJV for internal validation
  (#2426)
• Bind plugin options before registering the plugin
  (#2375)
• Extend the node.js event emitter
  (#2440)
• Move the wallet interfaces to @arkecosystem/core-state
  (#2515)
• Simplify the transaction pool by not using insertion order
  (#2495)
• Drop no longer needed pagination from @arkecosystem/core-webhooks
  (#2424)
• Extract transaction signing and utils out of the model in @arkecosystem/crypto
  (#2514)
• Extract transaction verification out of the model in @arkecosystem/crypto
  (#2506)
• Make all Slots.* methods static in @arkecosystem/crypto
  (#2473)
• Move interfaces, types and models in @arkecosystem/crypto
  (#2379)
• Move mainnet exceptions to config in @arkecosystem/crypto
  (#2529)
• Remove extraneous Client class in @arkecosystem/crypto
  (#2417)
• Split the Crypto class into Hash and Transaction in @arkecosystem/crypto
  (#2444)
• Replaced @arkecosystem/core-json-rpc with
  @arkecosystem/core-exchange-json-rpc (Use
  @arkecosystem/core-exchange-json-rpc programmatically)
  (#2643)
• Remove unused methods to get/set/reset height from Slots
  (#2467)

Added

• Implement in @arkecosystem/core-state to manage the state of in-memory data
  (#2479)
• Implement a blockchain replay command
  (#2526)
• Save blocks in batches during sync
  (#2500)
• Implement v2/node/fees endpoint
  (#2393)
• Allow setting a vendor field for transactions created via
  @arkecosystem/core-json-rpc
  (#2425)
• Limit the number of accepted peers per subnet
  (#2507)
• Parallel block download
  (#2433)
• In-memory storage for last N blocks and transactions
  (#2492)
• Require the user to choose a snapshot if the blocks flag is missing in
  @arkecosystem/core commands
  (#2522)
• Integrate end-to-end tests
  (#2468)
• Initial Implementation of core-wallet-api
  (#2544)
• Accept block height to list block transactions in core-api
  (#2567)
• Functional test matchers for core-jest-matchers
  (#2562)
• Don’t trust headers and verify config and plugin connectivity of peers
  (#2559,
  #2553,
  #2552)
• Proxy API calls to core-api until fully developed
  (#2558)
• Add database configuration command to CLI
  (#2557,
  #2563)
• Add command to generate network configuration CLI
  (#2582)
• Initial implementation of core-explorer
  (#2604)

Fixed

• Insert the genesis block as soon as the database is ready
  (#2376)
• Use public API to auto-configure @arkecosystem/core-tester-cli
  (#2517)
• Parse only the last line of pm2 stdout to avoid parsing faulty input
  (#2484)
• Delete bad rounds after unclean shutdown
  (#2581)
• Divide blocks into smaller chunks for batch processing
  (#2586)
• Remove forged transactions from pool before discarding block
  (#2555)
• Various sync issues with devnet and mainnet
  (#2565)
• Do not suspend peer for AppNotReady
  (#2590)
• Allow use of old and new block IDs via core-json-rpc
  (#2593)
• Assign calculated delegate ranks from temp wallets to prevent wrong ranks on
  boot (#2611)
• Camelize block keys before bignum transformation for snapshots
  (#2615)
• Deserialize transactions before they leave the pool to remove bad ones
  (#2622)
• Require all properties in schema and handle 404 resources for core-webhooks
  (#2634)
• Check if transactions can still be applied before forging
  (#2635)
• Off by one error in transaction confirmations via API
  (#2645)
• Set the correct channel if core was directly installed with @next
  (#2646)
• Invalid orderBy causes Internal Server Error via API ([#2653)
• Avoid trying to INSERT duplicates in rounds via core-snapshots
  (#2651)
• Handle failing optional plugins gracefully
  (#2657)
• Correctly purge invalid transactions from disk on start
  (#2665)
• Don’t append duplicate rounds rows to a snapshot
  (#2662)
• Use temporary wallets for transaction validation
  (#2666)
• Correctly display second signature if available via core-api
  (#2670)
• Missing block confirmations on v2 API endpoints
  (#2674)
• Delay transaction purge on start until after StateBuilder finished
  (#2685)
• Check claimed state of peer
  (#2686)
• Ignore overheight blocks and keep forging
  (#2687)

Changed

• Replace Joi with AJV for internal validation
  (#2426)
• Remove height difference ban for peers
  (#2360)
• Move mainnet exceptions to config in @arkecosystem/crypto
  (#2529)
• Invalidate blocks with expired transactions
  (#2528)
• Transaction type agnostic wallet bootstrap to support AIP29
  (#2539)
• Return all schema errors in hapi-ajv (#2571)
• Clean up SocketCluster shutdown and logging (#2560)
• Remove timeout banning (#2597)
• Use dayjs as it now has official UTC support
  (#2592)
• Require a minimum of 0 as pubKeyHash
  (#2628)
• Expire transactions that don’t have an expiration
  (#2672)

Removed

• Remove peer caching (#2606)
• Remove peer banning (#2612)
• Remove coldstart period
  (#2619)
• Remove whitelist access log
  (#2655)

Special Thanks

Thanks to all who have helped test the newest Core on the Development Network,
prior to moving it to Public Network, by providing valuable feedback, reporting
issues, and helping to resolve issues by opening pull-requests.

Especially delegate Alessio, who reported a few security vulnerabilities in
testing phase of newest Core (prior to Core v2.4 going on Mainnet), making it
even more secure and robust —a big thank you.

For those running your own servers with ARK Core, if you experience any errors
or issues, please open a Github issue report for easier tracking and resolution:
https://github.com/ArkEcosystem/core/issues.

Don’t forget we also have a development program for all who want to help improve
ARK and earn some ARK: https://bounty.ark.io.

---

Follow us on social media ( Twitter | Facebook | Reddit | YouTube), join our community ( Slack | Discord ) and stay tuned to our blog on Medium.