Running on hosts: isk-h200gpu001 isk-h200gpu002 isk-h200gpu003 isk-h200gpu004 isk-h200gpu005 isk-h200gpu006 isk-h200gpu007 isk-h200gpu008 isk-h200gpu009 isk-h200gpu010 isk-h200gpu011 isk-h200gpu012 isk-h200gpu013 isk-h200gpu014 isk-h200gpu015 isk-h200gpu016 isk-h200gpu017 isk-h200gpu018 isk-h200gpu019 isk-h200gpu020 isk-h200gpu021 isk-h200gpu022 isk-h200gpu023 isk-h200gpu024 isk-h200gpu025 isk-h200gpu026 isk-h200gpu027 isk-h200gpu028 isk-h200gpu029 isk-h200gpu030 isk-h200gpu031 isk-h200gpu032 isk-h200gpu033 isk-h200gpu034 isk-h200gpu035 isk-h200gpu036 isk-h200gpu037 isk-h200gpu038 isk-h200gpu039 isk-h200gpu040 isk-h200gpu041 isk-h200gpu042 isk-h200gpu043 isk-h200gpu044 isk-h200gpu045 isk-h200gpu046 isk-h200gpu047 isk-h200gpu048 isk-h200gpu049 isk-h200gpu050 isk-h200gpu051 isk-h200gpu052 isk-h200gpu053 isk-h200gpu054 isk-h200gpu055
2025-10-16T22:15:45
========== ENV SUMMARY (host) ==========
---------------- OMP / OpenMP ----------------
---------------- NCCL ----------------
NCCL_DEBUG                      = WARN
NCCL_MIN_NCHANNELS              = 32
NCCL_BUFFSIZE                   = 8388608
---------------- HPL-NVIDIA ----------------
---------------- CUDA ----------------
CUDA_VISIBLE_DEVICES            = 0\,1\,2\,3\,4\,5\,6\,7
---------------- UCX ----------------
---------------- NVSHMEM ----------------
---------------- MPI/PMI/PMIx ----------------
---------------- SLURM ----------------
SLURM_JOB_ID                    = 10105
SLURM_JOB_NAME                  = hpcg
SLURM_NODELIST                  = isk-h200gpu\[001-055\]
SLURM_NNODES                    = 55
SLURM_NTASKS                    = 440
SLURM_PROCID                    = 0
SLURM_LOCALID                   = 0
SLURM_JOB_GPUS                  = 0\,1\,2\,3\,4\,5\,6\,7
SLURM_TOPOLOGY_ADDR             = isk-h200gpu001
--- Raw env grep (prefix scan) ---
CUDA_VISIBLE_DEVICES=0,1,2,3,4,5,6,7
NCCL_BUFFSIZE=8388608
NCCL_DEBUG=WARN
NCCL_DYNAMIC_CHUNK_SIZE=524288
NCCL_IB_QPS_PER_CONNECTION=16
NCCL_MIN_NCHANNELS=32
NCCL_NSOCKS_PERTHREAD=8
NCCL_P2P_NET_CHUNKSIZE=524288
NCCL_SOCKET_NTHREADS=8
PMIX_MCA_gds=^ds12
SLURM_CLUSTER_NAME=isk-cluster
SLURM_CONF=/etc/slurm/slurm.conf
SLURM_CPUS_ON_NODE=120
SLURMD_NODENAME=isk-h200gpu001
SLURM_GPUS_ON_NODE=8
SLURM_GTIDS=0
SLURM_JOB_ACCOUNT=users
SLURM_JOB_CPUS_PER_NODE=120(x55)
SLURM_JOB_GID=2002
SLURM_JOB_GPUS=0,1,2,3,4,5,6,7
SLURM_JOB_ID=10105
SLURM_JOBID=10105
SLURM_JOB_NAME=hpcg
SLURM_JOB_NODELIST=isk-h200gpu[001-055]
SLURM_JOB_NUM_NODES=55
SLURM_JOB_PARTITION=h200
SLURM_JOB_QOS=normal
SLURM_JOB_UID=2002
SLURM_JOB_USER=fumi1ko24
SLURM_LOCALID=0
SLURM_NNODES=55
SLURM_NODE_ALIASES=(null)
SLURM_NODEID=0
SLURM_NODELIST=isk-h200gpu[001-055]
SLURM_NPROCS=440
SLURM_NTASKS=440
SLURM_NTASKS_PER_NODE=8
SLURM_PRIO_PROCESS=0
SLURM_PROCID=0
SLURM_SUBMIT_DIR=/home/2025Benchmark/HPL
SLURM_SUBMIT_HOST=isk-int001
SLURM_TASK_PID=180549
SLURM_TASKS_PER_NODE=8(x55)
SLURM_TOPOLOGY_ADDR=isk-h200gpu001
SLURM_TOPOLOGY_ADDR_PATTERN=node
SLURM_WORKING_CLUSTER=isk-cluster:10.255.255.248:6817:9728:109
UCX_RNDV_THRESH=32k
==================================
HPCG-NVIDIA 25.9.0  -- NVIDIA accelerated HPCG benchmark -- NVIDIA
Build v0.6.0 

Start of application (GPU-Only) ...
Initial Residual = 70194.8
Iteration = 1   Scaled Residual = 0.187153
Iteration = 2   Scaled Residual = 0.102911
Iteration = 3   Scaled Residual = 0.0710946
Iteration = 4   Scaled Residual = 0.0543475
Iteration = 5   Scaled Residual = 0.043972
Iteration = 6   Scaled Residual = 0.0369105
Iteration = 7   Scaled Residual = 0.0318036
Iteration = 8   Scaled Residual = 0.0279541
Iteration = 9   Scaled Residual = 0.0249598
Iteration = 10   Scaled Residual = 0.0225662
Iteration = 11   Scaled Residual = 0.0206027
Iteration = 12   Scaled Residual = 0.0189525
Iteration = 13   Scaled Residual = 0.0175349
Iteration = 14   Scaled Residual = 0.0162994
Iteration = 15   Scaled Residual = 0.0152155
Iteration = 16   Scaled Residual = 0.0142633
Iteration = 17   Scaled Residual = 0.0134287
Iteration = 18   Scaled Residual = 0.012698
Iteration = 19   Scaled Residual = 0.012055
Iteration = 20   Scaled Residual = 0.0114832
Iteration = 21   Scaled Residual = 0.0109666
Iteration = 22   Scaled Residual = 0.0104916
Iteration = 23   Scaled Residual = 0.0100514
Iteration = 24   Scaled Residual = 0.00964463
Iteration = 25   Scaled Residual = 0.00927294
Iteration = 26   Scaled Residual = 0.00893947
Iteration = 27   Scaled Residual = 0.0086452
Iteration = 28   Scaled Residual = 0.00838713
Iteration = 29   Scaled Residual = 0.00815888
Iteration = 30   Scaled Residual = 0.00795105
Iteration = 31   Scaled Residual = 0.00775417
Iteration = 32   Scaled Residual = 0.00756277
Iteration = 33   Scaled Residual = 0.00737534
Iteration = 34   Scaled Residual = 0.00719289
Iteration = 35   Scaled Residual = 0.0070162
Iteration = 36   Scaled Residual = 0.00684227
Iteration = 37   Scaled Residual = 0.00666541
Iteration = 38   Scaled Residual = 0.00648109
Iteration = 39   Scaled Residual = 0.00628922
Iteration = 40   Scaled Residual = 0.00609631
Iteration = 41   Scaled Residual = 0.00591282
Iteration = 42   Scaled Residual = 0.00574784
Iteration = 43   Scaled Residual = 0.00560631
Iteration = 44   Scaled Residual = 0.00548764
Iteration = 45   Scaled Residual = 0.00538626
Iteration = 46   Scaled Residual = 0.00529492
Iteration = 47   Scaled Residual = 0.00520707
Iteration = 48   Scaled Residual = 0.00511851
Iteration = 49   Scaled Residual = 0.00502926
Iteration = 50   Scaled Residual = 0.00494119
GPU Rank Info:
 | cuSPARSE version 12.6
 | Reference CPU memory = 31829.74 MB
 | GPU Name: 'NVIDIA H200' 
 | GPU Memory Use: 60452 MB / 143156 MB
 | Process Grid: 8x5x11
 | Local Domain: 512x512x272
 | Number of CPU Threads: 15
 | Slice Size: 4096
WARNING: PERFORMING UNPRECONDITIONED ITERATIONS
Call [0] Number of Iterations [11] Scaled Residual [1.198e-14]
WARNING: PERFORMING UNPRECONDITIONED ITERATIONS
Call [1] Number of Iterations [11] Scaled Residual [1.198e-14]
Call [0] Number of Iterations [2] Scaled Residual [1.53548e-17]
Call [1] Number of Iterations [2] Scaled Residual [1.53548e-17]
Departure from symmetry (scaled) for SpMV abs(x'*A*y - y'*A*x) = 1.92675e-15
Departure from symmetry (scaled) for MG abs(x'*Minv*y - y'*Minv*x) = 0
SpMV call [0] Residual [0]
SpMV call [1] Residual [0]
Initial Residual = 70194.8
Iteration = 1   Scaled Residual = 0.220755
Iteration = 2   Scaled Residual = 0.119578
Iteration = 3   Scaled Residual = 0.0814141
Iteration = 4   Scaled Residual = 0.0616966
Iteration = 5   Scaled Residual = 0.0497259
Iteration = 6   Scaled Residual = 0.0416496
Iteration = 7   Scaled Residual = 0.0358474
Iteration = 8   Scaled Residual = 0.0314817
Iteration = 9   Scaled Residual = 0.0280673
Iteration = 10   Scaled Residual = 0.0253277
Iteration = 11   Scaled Residual = 0.0230813
Iteration = 12   Scaled Residual = 0.0212028
Iteration = 13   Scaled Residual = 0.0196133
Iteration = 14   Scaled Residual = 0.0182497
Iteration = 15   Scaled Residual = 0.0170659
Iteration = 16   Scaled Residual = 0.0160306
Iteration = 17   Scaled Residual = 0.015118
Iteration = 18   Scaled Residual = 0.0143091
Iteration = 19   Scaled Residual = 0.0135882
Iteration = 20   Scaled Residual = 0.0129424
Iteration = 21   Scaled Residual = 0.0123621
Iteration = 22   Scaled Residual = 0.0118389
Iteration = 23   Scaled Residual = 0.0113654
Iteration = 24   Scaled Residual = 0.0109345
Iteration = 25   Scaled Residual = 0.0105385
Iteration = 26   Scaled Residual = 0.0101719
Iteration = 27   Scaled Residual = 0.00982859
Iteration = 28   Scaled Residual = 0.00950305
Iteration = 29   Scaled Residual = 0.00919167
Iteration = 30   Scaled Residual = 0.00889175
Iteration = 31   Scaled Residual = 0.00860254
Iteration = 32   Scaled Residual = 0.00832542
Iteration = 33   Scaled Residual = 0.00806108
Iteration = 34   Scaled Residual = 0.00781098
Iteration = 35   Scaled Residual = 0.00757675
Iteration = 36   Scaled Residual = 0.00735861
Iteration = 37   Scaled Residual = 0.00715662
Iteration = 38   Scaled Residual = 0.00697028
Iteration = 39   Scaled Residual = 0.00679784
Iteration = 40   Scaled Residual = 0.00663808
Iteration = 41   Scaled Residual = 0.0064894
Iteration = 42   Scaled Residual = 0.00634934
Iteration = 43   Scaled Residual = 0.00621593
Iteration = 44   Scaled Residual = 0.00608697
Iteration = 45   Scaled Residual = 0.00595997
Iteration = 46   Scaled Residual = 0.00583375
Iteration = 47   Scaled Residual = 0.00570782
Iteration = 48   Scaled Residual = 0.00558262
Iteration = 49   Scaled Residual = 0.00545971
Iteration = 50   Scaled Residual = 0.0053405
Iteration = 51   Scaled Residual = 0.00522659
Iteration = 52   Scaled Residual = 0.00511978
Iteration = 53   Scaled Residual = 0.00502059
Iteration = 54   Scaled Residual = 0.00492934
Call [0] Scaled Residual [0.00492934]
Call [1] Scaled Residual [0.00492934]
Call [2] Scaled Residual [0.00492934]
Call [3] Scaled Residual [0.00492934]
Call [4] Scaled Residual [0.00492934]
Call [5] Scaled Residual [0.00492934]
Call [6] Scaled Residual [0.00492934]
Call [7] Scaled Residual [0.00492934]
Call [8] Scaled Residual [0.00492934]
Call [9] Scaled Residual [0.00492934]
Call [10] Scaled Residual [0.00492934]
Call [11] Scaled Residual [0.00492934]
Call [12] Scaled Residual [0.00492934]
Call [13] Scaled Residual [0.00492934]
Call [14] Scaled Residual [0.00492934]
Call [15] Scaled Residual [0.00492934]
Call [16] Scaled Residual [0.00492934]
Call [17] Scaled Residual [0.00492934]
Call [18] Scaled Residual [0.00492934]
Call [19] Scaled Residual [0.00492934]
Call [20] Scaled Residual [0.00492934]
Call [21] Scaled Residual [0.00492934]
Call [22] Scaled Residual [0.00492934]
Call [23] Scaled Residual [0.00492934]
Call [24] Scaled Residual [0.00492934]
Call [25] Scaled Residual [0.00492934]
Call [26] Scaled Residual [0.00492934]
Call [27] Scaled Residual [0.00492934]
Call [28] Scaled Residual [0.00492934]
Call [29] Scaled Residual [0.00492934]
Call [30] Scaled Residual [0.00492934]
Call [31] Scaled Residual [0.00492934]
Call [32] Scaled Residual [0.00492934]
Call [33] Scaled Residual [0.00492934]
Call [34] Scaled Residual [0.00492934]
Call [35] Scaled Residual [0.00492934]
Call [36] Scaled Residual [0.00492934]
Call [37] Scaled Residual [0.00492934]
Call [38] Scaled Residual [0.00492934]
Call [39] Scaled Residual [0.00492934]
Call [40] Scaled Residual [0.00492934]
Call [41] Scaled Residual [0.00492934]
Call [42] Scaled Residual [0.00492934]
Call [43] Scaled Residual [0.00492934]
Call [44] Scaled Residual [0.00492934]
Call [45] Scaled Residual [0.00492934]
Call [46] Scaled Residual [0.00492934]
Call [47] Scaled Residual [0.00492934]
Call [48] Scaled Residual [0.00492934]
Call [49] Scaled Residual [0.00492934]
Call [50] Scaled Residual [0.00492934]
Call [51] Scaled Residual [0.00492934]
Call [52] Scaled Residual [0.00492934]
Call [53] Scaled Residual [0.00492934]
Call [54] Scaled Residual [0.00492934]
Call [55] Scaled Residual [0.00492934]
Call [56] Scaled Residual [0.00492934]
Call [57] Scaled Residual [0.00492934]
Call [58] Scaled Residual [0.00492934]
Call [59] Scaled Residual [0.00492934]
Call [60] Scaled Residual [0.00492934]
Call [61] Scaled Residual [0.00492934]
Call [62] Scaled Residual [0.00492934]
Call [63] Scaled Residual [0.00492934]
Call [64] Scaled Residual [0.00492934]
Call [65] Scaled Residual [0.00492934]
Call [66] Scaled Residual [0.00492934]
Call [67] Scaled Residual [0.00492934]
Call [68] Scaled Residual [0.00492934]
Call [69] Scaled Residual [0.00492934]
Call [70] Scaled Residual [0.00492934]
Call [71] Scaled Residual [0.00492934]
Call [72] Scaled Residual [0.00492934]
Call [73] Scaled Residual [0.00492934]
Call [74] Scaled Residual [0.00492934]
Call [75] Scaled Residual [0.00492934]
Call [76] Scaled Residual [0.00492934]
Call [77] Scaled Residual [0.00492934]
Call [78] Scaled Residual [0.00492934]
Call [79] Scaled Residual [0.00492934]
Call [80] Scaled Residual [0.00492934]
Call [81] Scaled Residual [0.00492934]
Call [82] Scaled Residual [0.00492934]
Call [83] Scaled Residual [0.00492934]
Call [84] Scaled Residual [0.00492934]
Call [85] Scaled Residual [0.00492934]
Call [86] Scaled Residual [0.00492934]
Call [87] Scaled Residual [0.00492934]
Call [88] Scaled Residual [0.00492934]
Call [89] Scaled Residual [0.00492934]
Call [90] Scaled Residual [0.00492934]
Call [91] Scaled Residual [0.00492934]
Call [92] Scaled Residual [0.00492934]
Call [93] Scaled Residual [0.00492934]
Call [94] Scaled Residual [0.00492934]
Call [95] Scaled Residual [0.00492934]
Call [96] Scaled Residual [0.00492934]
Call [97] Scaled Residual [0.00492934]
Call [98] Scaled Residual [0.00492934]
Call [99] Scaled Residual [0.00492934]
Call [100] Scaled Residual [0.00492934]
Call [101] Scaled Residual [0.00492934]
Call [102] Scaled Residual [0.00492934]
Call [103] Scaled Residual [0.00492934]
Call [104] Scaled Residual [0.00492934]
Call [105] Scaled Residual [0.00492934]
Call [106] Scaled Residual [0.00492934]
Call [107] Scaled Residual [0.00492934]
Call [108] Scaled Residual [0.00492934]
Call [109] Scaled Residual [0.00492934]
Call [110] Scaled Residual [0.00492934]
Call [111] Scaled Residual [0.00492934]
Call [112] Scaled Residual [0.00492934]
Call [113] Scaled Residual [0.00492934]
Call [114] Scaled Residual [0.00492934]
Call [115] Scaled Residual [0.00492934]
Call [116] Scaled Residual [0.00492934]
Call [117] Scaled Residual [0.00492934]
Call [118] Scaled Residual [0.00492934]
Call [119] Scaled Residual [0.00492934]
Call [120] Scaled Residual [0.00492934]
Call [121] Scaled Residual [0.00492934]
Call [122] Scaled Residual [0.00492934]
Call [123] Scaled Residual [0.00492934]
Call [124] Scaled Residual [0.00492934]
Call [125] Scaled Residual [0.00492934]
Call [126] Scaled Residual [0.00492934]
Call [127] Scaled Residual [0.00492934]
Call [128] Scaled Residual [0.00492934]
Call [129] Scaled Residual [0.00492934]
Call [130] Scaled Residual [0.00492934]
Call [131] Scaled Residual [0.00492934]
Call [132] Scaled Residual [0.00492934]
Call [133] Scaled Residual [0.00492934]
Call [134] Scaled Residual [0.00492934]
Call [135] Scaled Residual [0.00492934]
Call [136] Scaled Residual [0.00492934]
Call [137] Scaled Residual [0.00492934]
Call [138] Scaled Residual [0.00492934]
Call [139] Scaled Residual [0.00492934]
Call [140] Scaled Residual [0.00492934]
Call [141] Scaled Residual [0.00492934]
Call [142] Scaled Residual [0.00492934]
Call [143] Scaled Residual [0.00492934]
Call [144] Scaled Residual [0.00492934]
Call [145] Scaled Residual [0.00492934]
Call [146] Scaled Residual [0.00492934]
Call [147] Scaled Residual [0.00492934]
Call [148] Scaled Residual [0.00492934]
Call [149] Scaled Residual [0.00492934]
Call [150] Scaled Residual [0.00492934]
Call [151] Scaled Residual [0.00492934]
Call [152] Scaled Residual [0.00492934]
Call [153] Scaled Residual [0.00492934]
Call [154] Scaled Residual [0.00492934]
Call [155] Scaled Residual [0.00492934]
Call [156] Scaled Residual [0.00492934]
Call [157] Scaled Residual [0.00492934]
Call [158] Scaled Residual [0.00492934]
Call [159] Scaled Residual [0.00492934]
Call [160] Scaled Residual [0.00492934]
Call [161] Scaled Residual [0.00492934]
Call [162] Scaled Residual [0.00492934]
Call [163] Scaled Residual [0.00492934]
Call [164] Scaled Residual [0.00492934]
Call [165] Scaled Residual [0.00492934]
Call [166] Scaled Residual [0.00492934]
Call [167] Scaled Residual [0.00492934]
Call [168] Scaled Residual [0.00492934]
Call [169] Scaled Residual [0.00492934]
Call [170] Scaled Residual [0.00492934]
Call [171] Scaled Residual [0.00492934]
Call [172] Scaled Residual [0.00492934]
Call [173] Scaled Residual [0.00492934]
Call [174] Scaled Residual [0.00492934]
Call [175] Scaled Residual [0.00492934]
Call [176] Scaled Residual [0.00492934]
Call [177] Scaled Residual [0.00492934]
Call [178] Scaled Residual [0.00492934]
Call [179] Scaled Residual [0.00492934]
Call [180] Scaled Residual [0.00492934]
Call [181] Scaled Residual [0.00492934]
Call [182] Scaled Residual [0.00492934]
Call [183] Scaled Residual [0.00492934]
Call [184] Scaled Residual [0.00492934]
Call [185] Scaled Residual [0.00492934]
Call [186] Scaled Residual [0.00492934]
Call [187] Scaled Residual [0.00492934]
Call [188] Scaled Residual [0.00492934]
Call [189] Scaled Residual [0.00492934]
Call [190] Scaled Residual [0.00492934]
Call [191] Scaled Residual [0.00492934]
Call [192] Scaled Residual [0.00492934]
Call [193] Scaled Residual [0.00492934]
Call [194] Scaled Residual [0.00492934]
Call [195] Scaled Residual [0.00492934]
Call [196] Scaled Residual [0.00492934]
Call [197] Scaled Residual [0.00492934]
Call [198] Scaled Residual [0.00492934]
Call [199] Scaled Residual [0.00492934]
Call [200] Scaled Residual [0.00492934]
Call [201] Scaled Residual [0.00492934]
Call [202] Scaled Residual [0.00492934]
Call [203] Scaled Residual [0.00492934]
Call [204] Scaled Residual [0.00492934]
Call [205] Scaled Residual [0.00492934]
Call [206] Scaled Residual [0.00492934]
Call [207] Scaled Residual [0.00492934]
Call [208] Scaled Residual [0.00492934]
Call [209] Scaled Residual [0.00492934]
Call [210] Scaled Residual [0.00492934]
Call [211] Scaled Residual [0.00492934]
Call [212] Scaled Residual [0.00492934]
Call [213] Scaled Residual [0.00492934]
Call [214] Scaled Residual [0.00492934]
Call [215] Scaled Residual [0.00492934]
Call [216] Scaled Residual [0.00492934]
Call [217] Scaled Residual [0.00492934]
Call [218] Scaled Residual [0.00492934]
Call [219] Scaled Residual [0.00492934]
Call [220] Scaled Residual [0.00492934]
Call [221] Scaled Residual [0.00492934]
Call [222] Scaled Residual [0.00492934]
Call [223] Scaled Residual [0.00492934]
Call [224] Scaled Residual [0.00492934]
Call [225] Scaled Residual [0.00492934]
Call [226] Scaled Residual [0.00492934]
Call [227] Scaled Residual [0.00492934]
Call [228] Scaled Residual [0.00492934]
Call [229] Scaled Residual [0.00492934]
Call [230] Scaled Residual [0.00492934]
Call [231] Scaled Residual [0.00492934]
Call [232] Scaled Residual [0.00492934]
Call [233] Scaled Residual [0.00492934]
Call [234] Scaled Residual [0.00492934]
Call [235] Scaled Residual [0.00492934]
Call [236] Scaled Residual [0.00492934]
Call [237] Scaled Residual [0.00492934]
Call [238] Scaled Residual [0.00492934]
Call [239] Scaled Residual [0.00492934]
Call [240] Scaled Residual [0.00492934]
Call [241] Scaled Residual [0.00492934]
Call [242] Scaled Residual [0.00492934]
Call [243] Scaled Residual [0.00492934]
Call [244] Scaled Residual [0.00492934]
Call [245] Scaled Residual [0.00492934]
Call [246] Scaled Residual [0.00492934]
Call [247] Scaled Residual [0.00492934]
Call [248] Scaled Residual [0.00492934]
Call [249] Scaled Residual [0.00492934]
Call [250] Scaled Residual [0.00492934]
Call [251] Scaled Residual [0.00492934]
Call [252] Scaled Residual [0.00492934]
Call [253] Scaled Residual [0.00492934]
Call [254] Scaled Residual [0.00492934]
Call [255] Scaled Residual [0.00492934]
Call [256] Scaled Residual [0.00492934]
Call [257] Scaled Residual [0.00492934]
Call [258] Scaled Residual [0.00492934]
Call [259] Scaled Residual [0.00492934]
Call [260] Scaled Residual [0.00492934]
Call [261] Scaled Residual [0.00492934]
Call [262] Scaled Residual [0.00492934]
Call [263] Scaled Residual [0.00492934]
Call [264] Scaled Residual [0.00492934]
Call [265] Scaled Residual [0.00492934]
Call [266] Scaled Residual [0.00492934]
Call [267] Scaled Residual [0.00492934]
Call [268] Scaled Residual [0.00492934]
Call [269] Scaled Residual [0.00492934]
Call [270] Scaled Residual [0.00492934]
Call [271] Scaled Residual [0.00492934]
Call [272] Scaled Residual [0.00492934]
Call [273] Scaled Residual [0.00492934]
Call [274] Scaled Residual [0.00492934]
Call [275] Scaled Residual [0.00492934]
Call [276] Scaled Residual [0.00492934]
Call [277] Scaled Residual [0.00492934]
Call [278] Scaled Residual [0.00492934]
Call [279] Scaled Residual [0.00492934]
Call [280] Scaled Residual [0.00492934]
Call [281] Scaled Residual [0.00492934]
Call [282] Scaled Residual [0.00492934]
Call [283] Scaled Residual [0.00492934]
Call [284] Scaled Residual [0.00492934]
Call [285] Scaled Residual [0.00492934]
Call [286] Scaled Residual [0.00492934]
Call [287] Scaled Residual [0.00492934]
Call [288] Scaled Residual [0.00492934]
Call [289] Scaled Residual [0.00492934]
Call [290] Scaled Residual [0.00492934]
Call [291] Scaled Residual [0.00492934]
Call [292] Scaled Residual [0.00492934]
Call [293] Scaled Residual [0.00492934]
Call [294] Scaled Residual [0.00492934]
Call [295] Scaled Residual [0.00492934]
Call [296] Scaled Residual [0.00492934]
Call [297] Scaled Residual [0.00492934]
Call [298] Scaled Residual [0.00492934]
Call [299] Scaled Residual [0.00492934]
Call [300] Scaled Residual [0.00492934]
Call [301] Scaled Residual [0.00492934]
Call [302] Scaled Residual [0.00492934]
Call [303] Scaled Residual [0.00492934]
Call [304] Scaled Residual [0.00492934]
Call [305] Scaled Residual [0.00492934]
Call [306] Scaled Residual [0.00492934]
Call [307] Scaled Residual [0.00492934]
Call [308] Scaled Residual [0.00492934]
Call [309] Scaled Residual [0.00492934]
Call [310] Scaled Residual [0.00492934]
Call [311] Scaled Residual [0.00492934]
Call [312] Scaled Residual [0.00492934]
Call [313] Scaled Residual [0.00492934]
Call [314] Scaled Residual [0.00492934]
Call [315] Scaled Residual [0.00492934]
Call [316] Scaled Residual [0.00492934]
Call [317] Scaled Residual [0.00492934]
Call [318] Scaled Residual [0.00492934]
Call [319] Scaled Residual [0.00492934]
Call [320] Scaled Residual [0.00492934]
Call [321] Scaled Residual [0.00492934]
Call [322] Scaled Residual [0.00492934]
Call [323] Scaled Residual [0.00492934]
Call [324] Scaled Residual [0.00492934]
Call [325] Scaled Residual [0.00492934]
Call [326] Scaled Residual [0.00492934]
Call [327] Scaled Residual [0.00492934]
Call [328] Scaled Residual [0.00492934]
Call [329] Scaled Residual [0.00492934]
Call [330] Scaled Residual [0.00492934]
Call [331] Scaled Residual [0.00492934]
Call [332] Scaled Residual [0.00492934]
Call [333] Scaled Residual [0.00492934]
Call [334] Scaled Residual [0.00492934]
Call [335] Scaled Residual [0.00492934]
Call [336] Scaled Residual [0.00492934]
Call [337] Scaled Residual [0.00492934]
Call [338] Scaled Residual [0.00492934]
Call [339] Scaled Residual [0.00492934]
Call [340] Scaled Residual [0.00492934]
Call [341] Scaled Residual [0.00492934]
Call [342] Scaled Residual [0.00492934]
Call [343] Scaled Residual [0.00492934]
Call [344] Scaled Residual [0.00492934]
Call [345] Scaled Residual [0.00492934]
Call [346] Scaled Residual [0.00492934]
Call [347] Scaled Residual [0.00492934]
Call [348] Scaled Residual [0.00492934]
Call [349] Scaled Residual [0.00492934]
Call [350] Scaled Residual [0.00492934]
Call [351] Scaled Residual [0.00492934]
Call [352] Scaled Residual [0.00492934]
Call [353] Scaled Residual [0.00492934]
Call [354] Scaled Residual [0.00492934]
Call [355] Scaled Residual [0.00492934]
Call [356] Scaled Residual [0.00492934]
Call [357] Scaled Residual [0.00492934]
Call [358] Scaled Residual [0.00492934]
Call [359] Scaled Residual [0.00492934]
Call [360] Scaled Residual [0.00492934]
Call [361] Scaled Residual [0.00492934]
Call [362] Scaled Residual [0.00492934]
Call [363] Scaled Residual [0.00492934]
Call [364] Scaled Residual [0.00492934]
Call [365] Scaled Residual [0.00492934]
Call [366] Scaled Residual [0.00492934]
Call [367] Scaled Residual [0.00492934]
Call [368] Scaled Residual [0.00492934]
Call [369] Scaled Residual [0.00492934]
Call [370] Scaled Residual [0.00492934]
Call [371] Scaled Residual [0.00492934]
Call [372] Scaled Residual [0.00492934]
Call [373] Scaled Residual [0.00492934]
Call [374] Scaled Residual [0.00492934]
Call [375] Scaled Residual [0.00492934]
Call [376] Scaled Residual [0.00492934]
Call [377] Scaled Residual [0.00492934]
Call [378] Scaled Residual [0.00492934]
Call [379] Scaled Residual [0.00492934]
Call [380] Scaled Residual [0.00492934]
Call [381] Scaled Residual [0.00492934]
Call [382] Scaled Residual [0.00492934]
Call [383] Scaled Residual [0.00492934]
Call [384] Scaled Residual [0.00492934]
Call [385] Scaled Residual [0.00492934]
Call [386] Scaled Residual [0.00492934]
Call [387] Scaled Residual [0.00492934]
Call [388] Scaled Residual [0.00492934]
Call [389] Scaled Residual [0.00492934]
Call [390] Scaled Residual [0.00492934]
Call [391] Scaled Residual [0.00492934]
Call [392] Scaled Residual [0.00492934]
Call [393] Scaled Residual [0.00492934]
Call [394] Scaled Residual [0.00492934]
Call [395] Scaled Residual [0.00492934]
Call [396] Scaled Residual [0.00492934]
Call [397] Scaled Residual [0.00492934]
Call [398] Scaled Residual [0.00492934]
Call [399] Scaled Residual [0.00492934]
Call [400] Scaled Residual [0.00492934]
Call [401] Scaled Residual [0.00492934]
Call [402] Scaled Residual [0.00492934]
Call [403] Scaled Residual [0.00492934]
Call [404] Scaled Residual [0.00492934]
Call [405] Scaled Residual [0.00492934]
Call [406] Scaled Residual [0.00492934]
Call [407] Scaled Residual [0.00492934]
Call [408] Scaled Residual [0.00492934]
Call [409] Scaled Residual [0.00492934]
Call [410] Scaled Residual [0.00492934]
Call [411] Scaled Residual [0.00492934]
Call [412] Scaled Residual [0.00492934]
Call [413] Scaled Residual [0.00492934]
Call [414] Scaled Residual [0.00492934]
Call [415] Scaled Residual [0.00492934]
Call [416] Scaled Residual [0.00492934]
Call [417] Scaled Residual [0.00492934]
Call [418] Scaled Residual [0.00492934]
Call [419] Scaled Residual [0.00492934]
Call [420] Scaled Residual [0.00492934]
Call [421] Scaled Residual [0.00492934]
Call [422] Scaled Residual [0.00492934]
Call [423] Scaled Residual [0.00492934]
Call [424] Scaled Residual [0.00492934]
Call [425] Scaled Residual [0.00492934]
Call [426] Scaled Residual [0.00492934]
Call [427] Scaled Residual [0.00492934]
Call [428] Scaled Residual [0.00492934]
Call [429] Scaled Residual [0.00492934]
Call [430] Scaled Residual [0.00492934]
Call [431] Scaled Residual [0.00492934]
Call [432] Scaled Residual [0.00492934]
Call [433] Scaled Residual [0.00492934]
Call [434] Scaled Residual [0.00492934]
Call [435] Scaled Residual [0.00492934]
Call [436] Scaled Residual [0.00492934]
Call [437] Scaled Residual [0.00492934]
Call [438] Scaled Residual [0.00492934]
Call [439] Scaled Residual [0.00492934]
Call [440] Scaled Residual [0.00492934]
Call [441] Scaled Residual [0.00492934]
Call [442] Scaled Residual [0.00492934]
Call [443] Scaled Residual [0.00492934]
Call [444] Scaled Residual [0.00492934]
Call [445] Scaled Residual [0.00492934]
Call [446] Scaled Residual [0.00492934]
Call [447] Scaled Residual [0.00492934]
Call [448] Scaled Residual [0.00492934]
Call [449] Scaled Residual [0.00492934]
Call [450] Scaled Residual [0.00492934]
Call [451] Scaled Residual [0.00492934]
Call [452] Scaled Residual [0.00492934]
Call [453] Scaled Residual [0.00492934]
Call [454] Scaled Residual [0.00492934]
Call [455] Scaled Residual [0.00492934]
Call [456] Scaled Residual [0.00492934]
Call [457] Scaled Residual [0.00492934]
Call [458] Scaled Residual [0.00492934]
Call [459] Scaled Residual [0.00492934]
Call [460] Scaled Residual [0.00492934]
Call [461] Scaled Residual [0.00492934]
Call [462] Scaled Residual [0.00492934]
Call [463] Scaled Residual [0.00492934]
Call [464] Scaled Residual [0.00492934]
Call [465] Scaled Residual [0.00492934]
Call [466] Scaled Residual [0.00492934]
Call [467] Scaled Residual [0.00492934]
Call [468] Scaled Residual [0.00492934]
Call [469] Scaled Residual [0.00492934]
Call [470] Scaled Residual [0.00492934]
Call [471] Scaled Residual [0.00492934]
Call [472] Scaled Residual [0.00492934]
Call [473] Scaled Residual [0.00492934]
Call [474] Scaled Residual [0.00492934]
Call [475] Scaled Residual [0.00492934]
Call [476] Scaled Residual [0.00492934]
Call [477] Scaled Residual [0.00492934]
Call [478] Scaled Residual [0.00492934]
Call [479] Scaled Residual [0.00492934]
Call [480] Scaled Residual [0.00492934]
Call [481] Scaled Residual [0.00492934]
Call [482] Scaled Residual [0.00492934]
Call [483] Scaled Residual [0.00492934]
Call [484] Scaled Residual [0.00492934]
Call [485] Scaled Residual [0.00492934]
Call [486] Scaled Residual [0.00492934]
Call [487] Scaled Residual [0.00492934]
Call [488] Scaled Residual [0.00492934]
Call [489] Scaled Residual [0.00492934]
Call [490] Scaled Residual [0.00492934]
Call [491] Scaled Residual [0.00492934]
Call [492] Scaled Residual [0.00492934]
Call [493] Scaled Residual [0.00492934]
Call [494] Scaled Residual [0.00492934]
Call [495] Scaled Residual [0.00492934]
Call [496] Scaled Residual [0.00492934]
Call [497] Scaled Residual [0.00492934]
Call [498] Scaled Residual [0.00492934]
Call [499] Scaled Residual [0.00492934]
Call [500] Scaled Residual [0.00492934]
Call [501] Scaled Residual [0.00492934]
Call [502] Scaled Residual [0.00492934]
Call [503] Scaled Residual [0.00492934]
Call [504] Scaled Residual [0.00492934]
Call [505] Scaled Residual [0.00492934]
Call [506] Scaled Residual [0.00492934]
Call [507] Scaled Residual [0.00492934]
Call [508] Scaled Residual [0.00492934]
Call [509] Scaled Residual [0.00492934]
Call [510] Scaled Residual [0.00492934]
Call [511] Scaled Residual [0.00492934]
Call [512] Scaled Residual [0.00492934]
Call [513] Scaled Residual [0.00492934]
Call [514] Scaled Residual [0.00492934]
Call [515] Scaled Residual [0.00492934]
Call [516] Scaled Residual [0.00492934]
Call [517] Scaled Residual [0.00492934]
Call [518] Scaled Residual [0.00492934]
Call [519] Scaled Residual [0.00492934]
Call [520] Scaled Residual [0.00492934]
Call [521] Scaled Residual [0.00492934]
Call [522] Scaled Residual [0.00492934]
Call [523] Scaled Residual [0.00492934]
Call [524] Scaled Residual [0.00492934]
Call [525] Scaled Residual [0.00492934]
Call [526] Scaled Residual [0.00492934]
Call [527] Scaled Residual [0.00492934]
Call [528] Scaled Residual [0.00492934]
Call [529] Scaled Residual [0.00492934]
Call [530] Scaled Residual [0.00492934]
Call [531] Scaled Residual [0.00492934]
Call [532] Scaled Residual [0.00492934]
Call [533] Scaled Residual [0.00492934]
Call [534] Scaled Residual [0.00492934]
Call [535] Scaled Residual [0.00492934]
Call [536] Scaled Residual [0.00492934]
Call [537] Scaled Residual [0.00492934]
Call [538] Scaled Residual [0.00492934]
Call [539] Scaled Residual [0.00492934]
Call [540] Scaled Residual [0.00492934]
Call [541] Scaled Residual [0.00492934]
Call [542] Scaled Residual [0.00492934]
Call [543] Scaled Residual [0.00492934]
Call [544] Scaled Residual [0.00492934]
Call [545] Scaled Residual [0.00492934]
Call [546] Scaled Residual [0.00492934]
Call [547] Scaled Residual [0.00492934]
Call [548] Scaled Residual [0.00492934]
Call [549] Scaled Residual [0.00492934]
Call [550] Scaled Residual [0.00492934]
Call [551] Scaled Residual [0.00492934]
Call [552] Scaled Residual [0.00492934]
Call [553] Scaled Residual [0.00492934]
Call [554] Scaled Residual [0.00492934]
Call [555] Scaled Residual [0.00492934]
Call [556] Scaled Residual [0.00492934]
Call [557] Scaled Residual [0.00492934]
Call [558] Scaled Residual [0.00492934]
Call [559] Scaled Residual [0.00492934]
Call [560] Scaled Residual [0.00492934]
Call [561] Scaled Residual [0.00492934]
Call [562] Scaled Residual [0.00492934]
Call [563] Scaled Residual [0.00492934]
Call [564] Scaled Residual [0.00492934]
Call [565] Scaled Residual [0.00492934]
Call [566] Scaled Residual [0.00492934]
Call [567] Scaled Residual [0.00492934]
Call [568] Scaled Residual [0.00492934]
Call [569] Scaled Residual [0.00492934]
Call [570] Scaled Residual [0.00492934]
Call [571] Scaled Residual [0.00492934]
Call [572] Scaled Residual [0.00492934]
Call [573] Scaled Residual [0.00492934]
Call [574] Scaled Residual [0.00492934]
Call [575] Scaled Residual [0.00492934]
Call [576] Scaled Residual [0.00492934]
Call [577] Scaled Residual [0.00492934]
Call [578] Scaled Residual [0.00492934]
Call [579] Scaled Residual [0.00492934]
Call [580] Scaled Residual [0.00492934]
Call [581] Scaled Residual [0.00492934]
Call [582] Scaled Residual [0.00492934]
Call [583] Scaled Residual [0.00492934]
Call [584] Scaled Residual [0.00492934]
Call [585] Scaled Residual [0.00492934]
Call [586] Scaled Residual [0.00492934]
Call [587] Scaled Residual [0.00492934]
Call [588] Scaled Residual [0.00492934]
Call [589] Scaled Residual [0.00492934]
Call [590] Scaled Residual [0.00492934]
Call [591] Scaled Residual [0.00492934]
Call [592] Scaled Residual [0.00492934]
Call [593] Scaled Residual [0.00492934]
Call [594] Scaled Residual [0.00492934]
Call [595] Scaled Residual [0.00492934]
Call [596] Scaled Residual [0.00492934]
Call [597] Scaled Residual [0.00492934]
Call [598] Scaled Residual [0.00492934]
Call [599] Scaled Residual [0.00492934]
Call [600] Scaled Residual [0.00492934]
Call [601] Scaled Residual [0.00492934]
Call [602] Scaled Residual [0.00492934]
Call [603] Scaled Residual [0.00492934]
Call [604] Scaled Residual [0.00492934]
Call [605] Scaled Residual [0.00492934]
Call [606] Scaled Residual [0.00492934]
Call [607] Scaled Residual [0.00492934]
Call [608] Scaled Residual [0.00492934]
Call [609] Scaled Residual [0.00492934]
Call [610] Scaled Residual [0.00492934]
Call [611] Scaled Residual [0.00492934]
Call [612] Scaled Residual [0.00492934]
Call [613] Scaled Residual [0.00492934]
Call [614] Scaled Residual [0.00492934]
Call [615] Scaled Residual [0.00492934]
Call [616] Scaled Residual [0.00492934]
Call [617] Scaled Residual [0.00492934]
Call [618] Scaled Residual [0.00492934]
Call [619] Scaled Residual [0.00492934]
Call [620] Scaled Residual [0.00492934]
Call [621] Scaled Residual [0.00492934]
Call [622] Scaled Residual [0.00492934]
Call [623] Scaled Residual [0.00492934]
Call [624] Scaled Residual [0.00492934]
Call [625] Scaled Residual [0.00492934]
Call [626] Scaled Residual [0.00492934]
Call [627] Scaled Residual [0.00492934]
Call [628] Scaled Residual [0.00492934]
Call [629] Scaled Residual [0.00492934]
Call [630] Scaled Residual [0.00492934]
Call [631] Scaled Residual [0.00492934]
Call [632] Scaled Residual [0.00492934]
Call [633] Scaled Residual [0.00492934]
Call [634] Scaled Residual [0.00492934]
Call [635] Scaled Residual [0.00492934]
Call [636] Scaled Residual [0.00492934]
Call [637] Scaled Residual [0.00492934]
Call [638] Scaled Residual [0.00492934]
Call [639] Scaled Residual [0.00492934]
Call [640] Scaled Residual [0.00492934]
Call [641] Scaled Residual [0.00492934]
Call [642] Scaled Residual [0.00492934]
Call [643] Scaled Residual [0.00492934]
Call [644] Scaled Residual [0.00492934]
Call [645] Scaled Residual [0.00492934]
Call [646] Scaled Residual [0.00492934]
Call [647] Scaled Residual [0.00492934]
Call [648] Scaled Residual [0.00492934]
Call [649] Scaled Residual [0.00492934]
Call [650] Scaled Residual [0.00492934]
Call [651] Scaled Residual [0.00492934]
Call [652] Scaled Residual [0.00492934]
Call [653] Scaled Residual [0.00492934]
Call [654] Scaled Residual [0.00492934]
Call [655] Scaled Residual [0.00492934]
Call [656] Scaled Residual [0.00492934]
Call [657] Scaled Residual [0.00492934]
Call [658] Scaled Residual [0.00492934]
Call [659] Scaled Residual [0.00492934]
Call [660] Scaled Residual [0.00492934]
Call [661] Scaled Residual [0.00492934]
Call [662] Scaled Residual [0.00492934]
Call [663] Scaled Residual [0.00492934]
Call [664] Scaled Residual [0.00492934]
Call [665] Scaled Residual [0.00492934]
Call [666] Scaled Residual [0.00492934]
Call [667] Scaled Residual [0.00492934]
Call [668] Scaled Residual [0.00492934]
Call [669] Scaled Residual [0.00492934]
Call [670] Scaled Residual [0.00492934]
Call [671] Scaled Residual [0.00492934]
Call [672] Scaled Residual [0.00492934]
Call [673] Scaled Residual [0.00492934]
Call [674] Scaled Residual [0.00492934]
Call [675] Scaled Residual [0.00492934]
Call [676] Scaled Residual [0.00492934]
Call [677] Scaled Residual [0.00492934]
Call [678] Scaled Residual [0.00492934]
Call [679] Scaled Residual [0.00492934]
Call [680] Scaled Residual [0.00492934]
Call [681] Scaled Residual [0.00492934]
Call [682] Scaled Residual [0.00492934]
Call [683] Scaled Residual [0.00492934]
Call [684] Scaled Residual [0.00492934]
Call [685] Scaled Residual [0.00492934]
Call [686] Scaled Residual [0.00492934]
Call [687] Scaled Residual [0.00492934]
Call [688] Scaled Residual [0.00492934]
Call [689] Scaled Residual [0.00492934]
Call [690] Scaled Residual [0.00492934]
Call [691] Scaled Residual [0.00492934]
Call [692] Scaled Residual [0.00492934]
Call [693] Scaled Residual [0.00492934]
Call [694] Scaled Residual [0.00492934]
Call [695] Scaled Residual [0.00492934]
Call [696] Scaled Residual [0.00492934]
Call [697] Scaled Residual [0.00492934]
Call [698] Scaled Residual [0.00492934]
Call [699] Scaled Residual [0.00492934]
Call [700] Scaled Residual [0.00492934]
Call [701] Scaled Residual [0.00492934]
Call [702] Scaled Residual [0.00492934]
Call [703] Scaled Residual [0.00492934]
Call [704] Scaled Residual [0.00492934]
Call [705] Scaled Residual [0.00492934]
Call [706] Scaled Residual [0.00492934]
Call [707] Scaled Residual [0.00492934]
Call [708] Scaled Residual [0.00492934]
Call [709] Scaled Residual [0.00492934]
Call [710] Scaled Residual [0.00492934]
Call [711] Scaled Residual [0.00492934]
Call [712] Scaled Residual [0.00492934]
Call [713] Scaled Residual [0.00492934]
Call [714] Scaled Residual [0.00492934]
Call [715] Scaled Residual [0.00492934]
Call [716] Scaled Residual [0.00492934]
Call [717] Scaled Residual [0.00492934]
Call [718] Scaled Residual [0.00492934]
Call [719] Scaled Residual [0.00492934]
Call [720] Scaled Residual [0.00492934]
Call [721] Scaled Residual [0.00492934]
Call [722] Scaled Residual [0.00492934]
Call [723] Scaled Residual [0.00492934]
Call [724] Scaled Residual [0.00492934]
Call [725] Scaled Residual [0.00492934]
Call [726] Scaled Residual [0.00492934]
Call [727] Scaled Residual [0.00492934]
Call [728] Scaled Residual [0.00492934]
Call [729] Scaled Residual [0.00492934]
Call [730] Scaled Residual [0.00492934]
Call [731] Scaled Residual [0.00492934]
Call [732] Scaled Residual [0.00492934]
Call [733] Scaled Residual [0.00492934]
Call [734] Scaled Residual [0.00492934]
Call [735] Scaled Residual [0.00492934]
Call [736] Scaled Residual [0.00492934]
Call [737] Scaled Residual [0.00492934]
Call [738] Scaled Residual [0.00492934]
Call [739] Scaled Residual [0.00492934]
Call [740] Scaled Residual [0.00492934]
Call [741] Scaled Residual [0.00492934]
Call [742] Scaled Residual [0.00492934]
Call [743] Scaled Residual [0.00492934]
Call [744] Scaled Residual [0.00492934]
Call [745] Scaled Residual [0.00492934]
Call [746] Scaled Residual [0.00492934]
Call [747] Scaled Residual [0.00492934]
Call [748] Scaled Residual [0.00492934]
Call [749] Scaled Residual [0.00492934]
Call [750] Scaled Residual [0.00492934]
Call [751] Scaled Residual [0.00492934]
Call [752] Scaled Residual [0.00492934]
Call [753] Scaled Residual [0.00492934]
Call [754] Scaled Residual [0.00492934]
Call [755] Scaled Residual [0.00492934]
Call [756] Scaled Residual [0.00492934]
Call [757] Scaled Residual [0.00492934]
Call [758] Scaled Residual [0.00492934]
Call [759] Scaled Residual [0.00492934]
Call [760] Scaled Residual [0.00492934]
Call [761] Scaled Residual [0.00492934]
Call [762] Scaled Residual [0.00492934]
Call [763] Scaled Residual [0.00492934]
Call [764] Scaled Residual [0.00492934]
Call [765] Scaled Residual [0.00492934]
Call [766] Scaled Residual [0.00492934]
Call [767] Scaled Residual [0.00492934]
Call [768] Scaled Residual [0.00492934]
Call [769] Scaled Residual [0.00492934]
Call [770] Scaled Residual [0.00492934]
Call [771] Scaled Residual [0.00492934]
Call [772] Scaled Residual [0.00492934]
Call [773] Scaled Residual [0.00492934]
Call [774] Scaled Residual [0.00492934]
Call [775] Scaled Residual [0.00492934]
Call [776] Scaled Residual [0.00492934]
Call [777] Scaled Residual [0.00492934]
Call [778] Scaled Residual [0.00492934]
Call [779] Scaled Residual [0.00492934]
Call [780] Scaled Residual [0.00492934]
Call [781] Scaled Residual [0.00492934]
Call [782] Scaled Residual [0.00492934]
Call [783] Scaled Residual [0.00492934]
Call [784] Scaled Residual [0.00492934]
Call [785] Scaled Residual [0.00492934]
Call [786] Scaled Residual [0.00492934]
Call [787] Scaled Residual [0.00492934]
Call [788] Scaled Residual [0.00492934]
Call [789] Scaled Residual [0.00492934]
Call [790] Scaled Residual [0.00492934]
Call [791] Scaled Residual [0.00492934]
Call [792] Scaled Residual [0.00492934]
Call [793] Scaled Residual [0.00492934]
Call [794] Scaled Residual [0.00492934]
Call [795] Scaled Residual [0.00492934]
Call [796] Scaled Residual [0.00492934]
Call [797] Scaled Residual [0.00492934]
Call [798] Scaled Residual [0.00492934]
Call [799] Scaled Residual [0.00492934]
Call [800] Scaled Residual [0.00492934]
Call [801] Scaled Residual [0.00492934]
Call [802] Scaled Residual [0.00492934]
Call [803] Scaled Residual [0.00492934]
Call [804] Scaled Residual [0.00492934]
Call [805] Scaled Residual [0.00492934]
Call [806] Scaled Residual [0.00492934]
Call [807] Scaled Residual [0.00492934]
Call [808] Scaled Residual [0.00492934]
Call [809] Scaled Residual [0.00492934]
Call [810] Scaled Residual [0.00492934]
Call [811] Scaled Residual [0.00492934]
Call [812] Scaled Residual [0.00492934]
Call [813] Scaled Residual [0.00492934]
Call [814] Scaled Residual [0.00492934]
Call [815] Scaled Residual [0.00492934]
Call [816] Scaled Residual [0.00492934]
Call [817] Scaled Residual [0.00492934]
Call [818] Scaled Residual [0.00492934]
Call [819] Scaled Residual [0.00492934]
Call [820] Scaled Residual [0.00492934]
Call [821] Scaled Residual [0.00492934]
Call [822] Scaled Residual [0.00492934]
Call [823] Scaled Residual [0.00492934]
Call [824] Scaled Residual [0.00492934]
Call [825] Scaled Residual [0.00492934]
Call [826] Scaled Residual [0.00492934]
Call [827] Scaled Residual [0.00492934]
Call [828] Scaled Residual [0.00492934]
Call [829] Scaled Residual [0.00492934]
Call [830] Scaled Residual [0.00492934]
Call [831] Scaled Residual [0.00492934]
Call [832] Scaled Residual [0.00492934]
Call [833] Scaled Residual [0.00492934]
Call [834] Scaled Residual [0.00492934]
Call [835] Scaled Residual [0.00492934]
Call [836] Scaled Residual [0.00492934]
Call [837] Scaled Residual [0.00492934]
Call [838] Scaled Residual [0.00492934]
Call [839] Scaled Residual [0.00492934]
Call [840] Scaled Residual [0.00492934]
Call [841] Scaled Residual [0.00492934]
Call [842] Scaled Residual [0.00492934]
Call [843] Scaled Residual [0.00492934]
Call [844] Scaled Residual [0.00492934]
Call [845] Scaled Residual [0.00492934]
Call [846] Scaled Residual [0.00492934]
Call [847] Scaled Residual [0.00492934]
Call [848] Scaled Residual [0.00492934]
Call [849] Scaled Residual [0.00492934]
Call [850] Scaled Residual [0.00492934]
Call [851] Scaled Residual [0.00492934]
Call [852] Scaled Residual [0.00492934]
Call [853] Scaled Residual [0.00492934]
Call [854] Scaled Residual [0.00492934]
Call [855] Scaled Residual [0.00492934]
Call [856] Scaled Residual [0.00492934]
Call [857] Scaled Residual [0.00492934]
Call [858] Scaled Residual [0.00492934]
Call [859] Scaled Residual [0.00492934]
Call [860] Scaled Residual [0.00492934]
Call [861] Scaled Residual [0.00492934]
Call [862] Scaled Residual [0.00492934]
Call [863] Scaled Residual [0.00492934]
Call [864] Scaled Residual [0.00492934]
Call [865] Scaled Residual [0.00492934]
Call [866] Scaled Residual [0.00492934]
Call [867] Scaled Residual [0.00492934]
Call [868] Scaled Residual [0.00492934]
Call [869] Scaled Residual [0.00492934]
Call [870] Scaled Residual [0.00492934]
Call [871] Scaled Residual [0.00492934]
Call [872] Scaled Residual [0.00492934]
Call [873] Scaled Residual [0.00492934]
Call [874] Scaled Residual [0.00492934]
Call [875] Scaled Residual [0.00492934]
Call [876] Scaled Residual [0.00492934]
Call [877] Scaled Residual [0.00492934]
Call [878] Scaled Residual [0.00492934]
Call [879] Scaled Residual [0.00492934]
Call [880] Scaled Residual [0.00492934]
Call [881] Scaled Residual [0.00492934]
Call [882] Scaled Residual [0.00492934]
Call [883] Scaled Residual [0.00492934]
Call [884] Scaled Residual [0.00492934]
Call [885] Scaled Residual [0.00492934]
Call [886] Scaled Residual [0.00492934]
Call [887] Scaled Residual [0.00492934]
Call [888] Scaled Residual [0.00492934]
Call [889] Scaled Residual [0.00492934]
Call [890] Scaled Residual [0.00492934]
Call [891] Scaled Residual [0.00492934]
Call [892] Scaled Residual [0.00492934]
Call [893] Scaled Residual [0.00492934]
Call [894] Scaled Residual [0.00492934]
Call [895] Scaled Residual [0.00492934]
Call [896] Scaled Residual [0.00492934]
Call [897] Scaled Residual [0.00492934]
Call [898] Scaled Residual [0.00492934]
Call [899] Scaled Residual [0.00492934]
Call [900] Scaled Residual [0.00492934]
Call [901] Scaled Residual [0.00492934]
Call [902] Scaled Residual [0.00492934]
Call [903] Scaled Residual [0.00492934]
Call [904] Scaled Residual [0.00492934]
Call [905] Scaled Residual [0.00492934]
Call [906] Scaled Residual [0.00492934]
Call [907] Scaled Residual [0.00492934]
Call [908] Scaled Residual [0.00492934]
Call [909] Scaled Residual [0.00492934]
Call [910] Scaled Residual [0.00492934]
Call [911] Scaled Residual [0.00492934]
Call [912] Scaled Residual [0.00492934]
Call [913] Scaled Residual [0.00492934]
Call [914] Scaled Residual [0.00492934]
Call [915] Scaled Residual [0.00492934]
Call [916] Scaled Residual [0.00492934]
Call [917] Scaled Residual [0.00492934]
Call [918] Scaled Residual [0.00492934]
Call [919] Scaled Residual [0.00492934]
Call [920] Scaled Residual [0.00492934]
Call [921] Scaled Residual [0.00492934]
Call [922] Scaled Residual [0.00492934]
Call [923] Scaled Residual [0.00492934]
Call [924] Scaled Residual [0.00492934]
Call [925] Scaled Residual [0.00492934]
Call [926] Scaled Residual [0.00492934]
Call [927] Scaled Residual [0.00492934]
Call [928] Scaled Residual [0.00492934]
Call [929] Scaled Residual [0.00492934]
Call [930] Scaled Residual [0.00492934]
Call [931] Scaled Residual [0.00492934]
Call [932] Scaled Residual [0.00492934]
Call [933] Scaled Residual [0.00492934]
Call [934] Scaled Residual [0.00492934]
Call [935] Scaled Residual [0.00492934]
Call [936] Scaled Residual [0.00492934]
Call [937] Scaled Residual [0.00492934]
Call [938] Scaled Residual [0.00492934]
Call [939] Scaled Residual [0.00492934]
Call [940] Scaled Residual [0.00492934]
Call [941] Scaled Residual [0.00492934]
Call [942] Scaled Residual [0.00492934]
Call [943] Scaled Residual [0.00492934]
Call [944] Scaled Residual [0.00492934]
Call [945] Scaled Residual [0.00492934]
Call [946] Scaled Residual [0.00492934]
Call [947] Scaled Residual [0.00492934]
Call [948] Scaled Residual [0.00492934]
Call [949] Scaled Residual [0.00492934]
Call [950] Scaled Residual [0.00492934]
Call [951] Scaled Residual [0.00492934]
Call [952] Scaled Residual [0.00492934]
Call [953] Scaled Residual [0.00492934]
Call [954] Scaled Residual [0.00492934]
Call [955] Scaled Residual [0.00492934]
Call [956] Scaled Residual [0.00492934]
Call [957] Scaled Residual [0.00492934]
Call [958] Scaled Residual [0.00492934]
Call [959] Scaled Residual [0.00492934]
Call [960] Scaled Residual [0.00492934]
Call [961] Scaled Residual [0.00492934]
Call [962] Scaled Residual [0.00492934]
Call [963] Scaled Residual [0.00492934]
Call [964] Scaled Residual [0.00492934]
Call [965] Scaled Residual [0.00492934]
Call [966] Scaled Residual [0.00492934]
Call [967] Scaled Residual [0.00492934]
Call [968] Scaled Residual [0.00492934]
Call [969] Scaled Residual [0.00492934]
Call [970] Scaled Residual [0.00492934]
Call [971] Scaled Residual [0.00492934]
Call [972] Scaled Residual [0.00492934]
Call [973] Scaled Residual [0.00492934]
Call [974] Scaled Residual [0.00492934]
Call [975] Scaled Residual [0.00492934]
Call [976] Scaled Residual [0.00492934]
Call [977] Scaled Residual [0.00492934]
Call [978] Scaled Residual [0.00492934]
Call [979] Scaled Residual [0.00492934]
Call [980] Scaled Residual [0.00492934]
Call [981] Scaled Residual [0.00492934]
Call [982] Scaled Residual [0.00492934]
Call [983] Scaled Residual [0.00492934]
Call [984] Scaled Residual [0.00492934]
Call [985] Scaled Residual [0.00492934]
Call [986] Scaled Residual [0.00492934]
Call [987] Scaled Residual [0.00492934]
Call [988] Scaled Residual [0.00492934]
Call [989] Scaled Residual [0.00492934]
Call [990] Scaled Residual [0.00492934]
Call [991] Scaled Residual [0.00492934]
Call [992] Scaled Residual [0.00492934]
Call [993] Scaled Residual [0.00492934]
Call [994] Scaled Residual [0.00492934]
Call [995] Scaled Residual [0.00492934]
Call [996] Scaled Residual [0.00492934]
Call [997] Scaled Residual [0.00492934]
Call [998] Scaled Residual [0.00492934]
Call [999] Scaled Residual [0.00492934]
Call [1000] Scaled Residual [0.00492934]
Call [1001] Scaled Residual [0.00492934]
Call [1002] Scaled Residual [0.00492934]
Call [1003] Scaled Residual [0.00492934]
Call [1004] Scaled Residual [0.00492934]
Call [1005] Scaled Residual [0.00492934]
Call [1006] Scaled Residual [0.00492934]
Call [1007] Scaled Residual [0.00492934]
Call [1008] Scaled Residual [0.00492934]
Call [1009] Scaled Residual [0.00492934]
Call [1010] Scaled Residual [0.00492934]
Call [1011] Scaled Residual [0.00492934]
Call [1012] Scaled Residual [0.00492934]
Call [1013] Scaled Residual [0.00492934]
Call [1014] Scaled Residual [0.00492934]
Call [1015] Scaled Residual [0.00492934]
Call [1016] Scaled Residual [0.00492934]
Call [1017] Scaled Residual [0.00492934]
Call [1018] Scaled Residual [0.00492934]
Call [1019] Scaled Residual [0.00492934]
Call [1020] Scaled Residual [0.00492934]
Call [1021] Scaled Residual [0.00492934]
Call [1022] Scaled Residual [0.00492934]
Call [1023] Scaled Residual [0.00492934]
Call [1024] Scaled Residual [0.00492934]
Call [1025] Scaled Residual [0.00492934]
Call [1026] Scaled Residual [0.00492934]
Call [1027] Scaled Residual [0.00492934]
Call [1028] Scaled Residual [0.00492934]
Call [1029] Scaled Residual [0.00492934]
Call [1030] Scaled Residual [0.00492934]
Call [1031] Scaled Residual [0.00492934]
Call [1032] Scaled Residual [0.00492934]
Call [1033] Scaled Residual [0.00492934]
Call [1034] Scaled Residual [0.00492934]
Call [1035] Scaled Residual [0.00492934]
Call [1036] Scaled Residual [0.00492934]
Call [1037] Scaled Residual [0.00492934]
HPCG-Benchmark
version=3.1
Release date=March 28, 2019
Machine Summary=
Machine Summary::Distributed Processes=440
Machine Summary::Threads per processes=15
Global Problem Dimensions=
Global Problem Dimensions::Global nx=4096
Global Problem Dimensions::Global ny=2560
Global Problem Dimensions::Global nz=2992
Processor Dimensions=
Processor Dimensions::npx=8
Processor Dimensions::npy=5
Processor Dimensions::npz=11
Local Domain Dimensions=
Local Domain Dimensions::nx=512
Local Domain Dimensions::ny=512
########## Problem Summary  ##########=
Setup Information=
Setup Information::Setup Time=0.198342
Linear System Information=
Linear System Information::Number of Equations=31373393920
Linear System Information::Number of Nonzero Terms=846534542392
Multigrid Information=
Multigrid Information::Number of coarse grid levels=3
Multigrid Information::Coarse Grids=
Multigrid Information::Coarse Grids::Grid Level=1
Multigrid Information::Coarse Grids::Number of Equations=3921674240
Multigrid Information::Coarse Grids::Number of Nonzero Terms=105748460056
Multigrid Information::Coarse Grids::Number of Presmoother Steps=1
Multigrid Information::Coarse Grids::Number of Postsmoother Steps=1
Multigrid Information::Coarse Grids::Grid Level=2
Multigrid Information::Coarse Grids::Number of Equations=490209280
Multigrid Information::Coarse Grids::Number of Nonzero Terms=13201478920
Multigrid Information::Coarse Grids::Number of Presmoother Steps=1
Multigrid Information::Coarse Grids::Number of Postsmoother Steps=1
Multigrid Information::Coarse Grids::Grid Level=3
Multigrid Information::Coarse Grids::Number of Equations=61276160
Multigrid Information::Coarse Grids::Number of Nonzero Terms=1645920640
Multigrid Information::Coarse Grids::Number of Presmoother Steps=1
Multigrid Information::Coarse Grids::Number of Postsmoother Steps=1
########## Memory Use Summary  ##########=
Memory Use Information=
Memory Use Information::Total memory used for data (Gbytes)=22427.3
Memory Use Information::Memory used for OptimizeProblem data (Gbytes)=0
Memory Use Information::Bytes per equation (Total memory / Number of Equations)=714.851
Memory Use Information::Memory used for linear system and CG (Gbytes)=19737.7
Memory Use Information::Coarse Grids=
Memory Use Information::Coarse Grids::Grid Level=1
Memory Use Information::Coarse Grids::Memory used=2357.88
Memory Use Information::Coarse Grids::Grid Level=2
Memory Use Information::Coarse Grids::Memory used=294.856
Memory Use Information::Coarse Grids::Grid Level=3
Memory Use Information::Coarse Grids::Memory used=36.8877
########## V&V Testing Summary  ##########=
Spectral Convergence Tests=
Spectral Convergence Tests::Result=PASSED
Spectral Convergence Tests::Unpreconditioned=
Spectral Convergence Tests::Unpreconditioned::Maximum iteration count=11
Spectral Convergence Tests::Unpreconditioned::Expected iteration count=12
Spectral Convergence Tests::Preconditioned=
Spectral Convergence Tests::Preconditioned::Maximum iteration count=2
Spectral Convergence Tests::Preconditioned::Expected iteration count=2
Departure from Symmetry |x'Ay-y'Ax|/(2*||x||*||A||*||y||)/epsilon=
Departure from Symmetry |x'Ay-y'Ax|/(2*||x||*||A||*||y||)/epsilon::Result=PASSED
Departure from Symmetry |x'Ay-y'Ax|/(2*||x||*||A||*||y||)/epsilon::Departure for SpMV=1.92675e-15
Departure from Symmetry |x'Ay-y'Ax|/(2*||x||*||A||*||y||)/epsilon::Departure for MG=0
########## Iterations Summary  ##########=
Iteration Count Information=
Iteration Count Information::Result=PASSED
Iteration Count Information::Reference CG iterations per set=50
Iteration Count Information::Optimized CG iterations per set=54
Iteration Count Information::Total number of reference iterations=51900
Iteration Count Information::Total number of optimized iterations=56052
########## Reproducibility Summary  ##########=
Reproducibility Information=
Reproducibility Information::Result=PASSED
Reproducibility Information::Scaled residual mean=0.00492934
Reproducibility Information::Scaled residual variance=0
########## Performance Summary (times in sec) ##########=
Benchmark Time Summary=
Benchmark Time Summary::Optimization phase=0.230962
Benchmark Time Summary::DDOT=105.622
Benchmark Time Summary::WAXPBY=66.8185
Benchmark Time Summary::SpMV=334.095
Benchmark Time Summary::MG=1452.54
Benchmark Time Summary::Total=1959.39
Floating Point Operations Summary=
Floating Point Operations Summary::Raw DDOT=1.06164e+16
Floating Point Operations Summary::Raw WAXPBY=1.06164e+16
Floating Point Operations Summary::Raw SpMV=9.66573e+16
Floating Point Operations Summary::Raw MG=5.41542e+17
Floating Point Operations Summary::Total=6.59432e+17
Floating Point Operations Summary::Total with convergence overhead=6.10586e+17
GB/s Summary=
GB/s Summary::Raw Read B/W=2.07287e+06
GB/s Summary::Raw Write B/W=479101
GB/s Summary::Raw Total B/W=2.55197e+06
GB/s Summary::Total with convergence and optimization phase overhead=2.31039e+06
GFLOP/s Summary=
GFLOP/s Summary::Raw DDOT=100513
GFLOP/s Summary::Raw WAXPBY=158884
GFLOP/s Summary::Raw SpMV=289311
GFLOP/s Summary::Raw MG=372824
GFLOP/s Summary::Raw Total=336550
GFLOP/s Summary::Total with convergence overhead=311620
GFLOP/s Summary::Total with convergence and optimization phase overhead=304691
User Optimization Overheads=
User Optimization Overheads::Optimization phase time (sec)=0.230962
User Optimization Overheads::Optimization phase time vs reference SpMV+MG time=0.0184064
DDOT Timing Variations=
DDOT Timing Variations::Min DDOT MPI_Allreduce time=23.7861
DDOT Timing Variations::Max DDOT MPI_Allreduce time=64.666
DDOT Timing Variations::Avg DDOT MPI_Allreduce time=37.2468
Final Summary=
Final Summary::HPCG result is VALID with a GFLOP/s rating of=304691
Final Summary::HPCG 2.4 rating for historical reasons is=307854
Final Summary::Please upload results from the YAML file contents to=http://hpcg-benchmark.org
Done
2025-10-16T22:15:45
