iguana status - 1 hour BTC sync, from scratch

After a bug was reported about the bitcoin sync stalling, I found a couple of system issues and fixed them. I also made a couple of shell scripts that loop on iguana, so even if it crashes, it just resumes. The change from 2000 blocks to 100 increased the overhead, but I managed to get the mythical 1 hour sync!

The following is the bandwidth chart, it is in kilobytes per second, so multiply by about 10 to get the megabits per second. It used to peak at over 120MB/sec as it is a 1 gigabit/sec connection for the server. But since most people dont have any connection like that, we dont lose much by slowing down the transfer rate. now a 250 megabits/sec connection should be able to replicate this performance.

I used the ./fastgenbtc script on an SSD server with 64GB of RAM. Still it ran out partway through, but it did automatically restart. Can you find in the bandwidth data where it crashed/restarted?

Now this is 1 hour to get all the data into bundle files. Still another hour or so is needed to process it and get to realtime blocks, but with the frontloading of the download, that means even if connection is lost it can continue crunching away and also that a 100 megabit per second connection should come close to the 2 hour time to realtime processing.

I notice at these settings the tmp dir only gets about 2GB, so it uses up RAM like crazy, but saves on HDD space during the sync.

James

it took 75 minutes total to get all the stragglers. At the end you just end up waiting a long time for the last blocks to come in, but at least this only happens at the end and not all the time like a serial sync ends up doing.

01:20:54   3793.74    327.89
01:21:54   2880.15    326.28
01:22:54  21622.71    386.93
01:23:54  31129.05    317.10
01:24:54  30578.95    315.52
01:25:54  11181.91    207.80
01:26:55   2724.29    162.14
01:27:55   5569.86    144.24
01:28:55  22005.42    217.63
01:29:55  28805.29    244.96
01:30:55  33015.54    295.16
01:31:55  31859.43    297.41
01:32:55  31507.55    302.09
01:33:55  33022.51    312.74
  Time           eth0       
HH:MM:SS   KB/s in  KB/s out
01:34:55  30681.66    312.11
01:35:55  40202.57    273.54
01:36:55  17690.79    186.20
01:37:55  33492.91    277.29
01:38:55  29564.62    313.06
01:39:55  36833.52    331.44
01:40:55  30255.90    279.72
01:41:55  33410.82    263.10
01:42:56  29287.15    253.20
01:43:56  32937.38    262.86
01:44:56  23001.50    239.44
01:45:56  39026.84    384.33
01:46:56  40690.34    405.71
01:47:56  24515.06    324.35
01:48:56  27608.76    313.32
01:49:56  25815.32    274.26
01:50:56  28547.83    298.87
  Time           eth0       
HH:MM:SS   KB/s in  KB/s out
01:51:56  35297.50    325.84
01:52:56  37797.67    360.16
01:53:56  39582.91    362.75
01:54:56  35919.24    361.97
01:55:56  36561.06    343.31
01:56:56  39791.59    357.29
01:57:56  46759.03    405.40
01:58:56  30049.32    372.31
01:59:57  24275.48    177.98
02:00:57  20154.10    160.89
02:01:57  21601.22    256.47
02:02:57  29025.60    269.12
02:03:57  34627.89    227.31
02:04:57  28082.29    195.24
02:05:57  29393.04    225.25
02:06:57  26977.68    216.65
02:07:57  41273.89    344.71
  Time           eth0       
HH:MM:SS   KB/s in  KB/s out
02:08:57  24803.07    283.87
02:09:57  22699.78    222.74
02:10:57    839.88     68.85
02:11:57  32025.31    243.54
02:12:57  37059.55    245.45
02:13:57  19033.64    189.71
02:14:57   1012.05     72.38
02:15:57   1308.39     68.14
02:16:58   1437.04     70.02
02:17:58    893.00     65.82
02:18:58   1933.68     81.08
02:19:58   2435.06     86.25
02:20:58   3717.50    150.69
02:21:58   1833.38    138.06
02:22:58   1591.88     54.67
02:23:58   2126.64     95.32
02:24:58    282.25     86.92
  Time           eth0       
HH:MM:SS   KB/s in  KB/s out
02:25:58  24856.70    285.44
02:26:58  34006.65    286.93
02:27:58  25836.59    218.01
02:28:58  11556.06    155.70
02:29:58  33898.82    289.15
02:30:58  42396.82    362.45
02:31:58  46498.98    414.79
02:32:58  44278.07    434.15
02:33:59  25908.16    346.61