Since our last Infrastructure upgrade EOS Tribe have been busy experimenting with various server configurations. Our goal was to build Enterprise level reliable and scalable EOS API infrastructure.
We have quickly discovered that IO operations and not processor speed that is the most critical component in maintaining a healthy cluster of EOS API nodes as well as producer node.
Thanks to EOS Tribe partnership with our Data Center partner Syptec we were able to build and tear down a complete server infrastructure based on stress test results.
We have rebuilt our entire infrastructure 3 times trying various IO technologies.
At the end we have stopped on bare metal servers with attached NVMe SSD drives by Samsung that deliver consistent top performance with heavy random read/write operations characteristics of Nodeos process.
Currently we are dedicating 3 physical servers to EOS api1, api2 and api3 load balanced on api.eostribe.io and 1 physical server for Worbli API node. And one specialized physical server with server grade 4.0 Ghz processor to Producer Node.
You can see specification for each server on diagram below:
We are running 2 nodeos instances on each server with one using history api plugin. And one nodeos instance on api3 server is also synchronizing with MongoDB plugin while we are working on alternative history plugin initiative :
Below is EOS Tribe overall EOS Infrastructure deployment diagram supported by above servers:
We are running our Primary and Backup Block Producer on secured Wireguard Mesh network directly peered with other top Block Producer nodes (via seed node).
EOS Tribe block producing node is also running latest version of Heartbeat plugin. As one could see from CPU statistics reported by Heartbeat monitor EOS Tribe producer is running on one of the fastest CPU on the network:
EOS Tribe servers are hosted in secure Data Center in Bluffdale, UT on a fast low latency 10G connection. We are running in the same facility Facebook is running it's servers.
It requires multiple biometrics access checkpoints for access to the server room. Photographs are usually forbidden however with permission of our provider we can provide a sneak peak into the facility:
The power generators you see on a last picture along with battery backup power inside a facility is capable to power entire Data Center for up to 36 days in case of total power loss to facility.
EOS Tribe goal is to provide enterprise level support for EOS network as a block producer and provide stable and scalable API to development community.