012.5 task2附录：NINETOOTHED 调参实验

调参实验

1. NINETOOTHED_AUTOTUNE 自动调优实验

当默认 BLOCK_SIZE 为 1024，对比是否开启自动调优。

实验结果：耗时相近。
现象分析：对 vector add，最优往往就在 1024 附近，搜完经常还是接近默认值，所以两条曲线几乎重合。

实验记录

# 不开自动调优
 
(base) root@78633864f968:/data/huggingface_home/hub/gollamago/assignment/task2# python benchmark_ninetoothed_add.py
size    correct NineToothed(ms) PyTorch(ms)
262144  PASS    0.0283          0.0292
524288  PASS    0.0311          0.0315
1048576 PASS    0.0368          0.0353
2097152 PASS    0.0452          0.0403
4194304 PASS    0.0626          0.0484
8388608 PASS    0.0798          0.0652
16777216        PASS    0.1258          0.0981
33554432        PASS    0.2179          0.1638
67108864        PASS    0.4031          0.2968
134217728       PASS    0.7755          0.5607
 
# 开启自动调优 NINETOOTHED_AUTOTUNE=1
 
(base) root@78633864f968:/data/huggingface_home/hub/gollamago/assignment/task2# NINETOOTHED_AUTOTUNE=1 python benchmark_ninetoothed_add.py
size    correct NineToothed(ms) PyTorch(ms)
262144  PASS    0.0320          0.0322
524288  PASS    0.0339          0.0335
1048576 PASS    0.0378          0.0356
2097152 PASS    0.0447          0.0399
4194304 PASS    0.0552          0.0538
8388608 PASS    0.0797          0.0636
16777216        PASS    0.1223          0.0939
33554432        PASS    0.2178          0.1595
67108864        PASS    0.4023          0.2964
134217728       PASS    0.7710          0.5583

2. BLOCK_SIZE 调参实验

把默认块大小从1024改为256，再对比是否开启自动调优。

BLOCK_SIZE = (
    ninetoothed.block_size(lower_bound=256, upper_bound=1024)
    if AUTOTUNE
    else 256
)

实验结果：自动调优显著优化耗时，越长批次优化越明显。
现象分析：对于小size，向量还不长时，真正搬运的数据少，GPU 也未必吃满。BLOCK_SIZE 好坏会被固定开销盖住，所以差距拉不开。对于大size，向量越长size / BLOCK_SIZE 越碎越多，访存延迟越高，自动调优的效果越好。

实验记录

 
# 重复两次实验，获取无自动调优基准
 
(base) root@78633864f968:/data/huggingface_home/hub/gollamago/assignment/task2# python benchmark_ninetoothed_add.py
size    correct NineToothed(ms) PyTorch(ms)
262144  PASS    0.0400          0.0362
524288  PASS    0.0464          0.0360
1048576 PASS    0.0636          0.0460
2097152 PASS    0.0926          0.0464
4194304 PASS    0.1326          0.0509
8388608 PASS    0.2272          0.0693
16777216        PASS    0.4189          0.1016
33554432        PASS    0.7848          0.1667
67108864        PASS    1.5410          0.3045
134217728       PASS    3.0976          0.5671
(base) root@78633864f968:/data/huggingface_home/hub/gollamago/assignment/task2# python benchmark_ninetoothed_add.py
size    correct NineToothed(ms) PyTorch(ms)
262144  PASS    0.0383          0.0340
524288  PASS    0.0453          0.0330
1048576 PASS    0.0533          0.0355
2097152 PASS    0.0821          0.0394
4194304 PASS    0.1394          0.0438
8388608 PASS    0.2203          0.0635
16777216        PASS    0.4218          0.1056
33554432        PASS    0.7957          0.1680
67108864        PASS    1.5604          0.2953
134217728       PASS    3.0785          0.5572
 
 
# 开启自动调优
 
(base) root@78633864f968:/data/huggingface_home/hub/gollamago/assignment/task2# NINETOOTHED_AUTOTUNE=1 python benchmark_ninetoothed_add.py
size    correct NineToothed(ms) PyTorch(ms)
262144  PASS    0.0282          0.0282
524288  PASS    0.0343          0.0294
1048576 PASS    0.0398          0.0373
2097152 PASS    0.0471          0.0416
4194304 PASS    0.0583          0.0492
8388608 PASS    0.0816          0.0628
16777216        PASS    0.1312          0.1038
33554432        PASS    0.2235          0.1651
67108864        PASS    0.4114          0.3039
134217728       PASS    0.7872          0.5622