AMD APU Benchmark (AMD A9-9410 RADEON R5, 5 COMPUTE CORES 2C+3G)

[trholding]

Inference:

./dlprim_benchmark 0:0 ../docs/nets_for_benchmark/resnet18-b16.js 
Using: STONEY (stoney, LLVM 14.0.6, DRM 3.47, 5.19.10-arch1-1) on Clover
Inputs
- data: (16,3,224,224)
Outputs
- loss: (16,1000)
Step -5    753.283
Step -4    817.717
Step -3    751.294
Step -2    734.901
Step -1    814.137
Step  0    744.462
Step  1    750.251
Step  2    736.538
Step  3    754.376
Step  4    743.783
Step  5    748.411
Step  6    735.602
Step  7    747.700
Step  8    755.288
Step  9    741.004
Step 10    748.873
Step 11    746.445
Step 12    753.553
Step 13    743.892
Step 14    746.026
Step 15    745.455
Step 16    753.993
Step 17    744.365
Step 18    745.327
Step 19    743.223
Time per sample: 46.652 ms
TOT time per batch:  746.428 ms

Training

./dlprim_benchmark -b 0:0 ../docs/nets_for_benchmark/resnet18-b16.js 
Using: STONEY (stoney, LLVM 14.0.6, DRM 3.47, 5.19.10-arch1-1) on Clover
Inputs
- data: (16,3,224,224)
Outputs
- loss: (16,1000)
Step -5   2786.765   982.025  1804.740
Step -4   2515.683   830.553  1685.130
Step -3   2506.575   829.354  1677.221
Step -2   2495.544   812.345  1683.198
Step -1   2513.398   831.204  1682.194
Step  0   2495.929   814.021  1681.908
Step  1   2509.957   831.687  1678.269
Step  2   2500.654   814.149  1686.505
Step  3   2514.796   835.242  1679.555
Step  4   2505.049   820.404  1684.646
Step  5   2495.013   821.157  1673.856
Step  6   2504.354   820.759  1683.595
Step  7   2498.988   823.010  1675.978
Step  8   2504.039   822.625  1681.414
Step  9   2498.310   820.414  1677.896
Step 10   2517.657   825.044  1692.613
Step 11   2515.839   836.715  1679.125
Step 12   2497.379   820.033  1677.345
Step 13   2526.872   841.701  1685.170
Step 14   2494.049   814.485  1679.564
Step 15   2521.818   833.189  1688.630
Step 16   2513.063   816.292  1696.772
Step 17   2518.752   832.467  1686.285
Step 18   2517.444   828.530  1688.915
Step 19   2511.166   828.520  1682.647
Time per sample: 156.754 ms
FWD time per batch:  825.022 ms
BWD time per batch:  1683.034 ms
TOT time per batch:  2508.056 ms

System Info:

cat /proc/cpuinfo 
processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 112
model name      : AMD A9-9410 RADEON R5, 5 COMPUTE CORES 2C+3G
stepping        : 0
microcode       : 0x6006705
cpu MHz         : 3432.985
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 0
cpu cores       : 2
apicid          : 16
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good acc_power nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm perfctr_core perfctr_nb bpext ptsc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 xsaveopt arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov
bugs            : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed
bogomips        : 5791.17
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro acc_power [13]

processor       : 1
vendor_id       : AuthenticAMD
cpu family      : 21
model           : 112
model name      : AMD A9-9410 RADEON R5, 5 COMPUTE CORES 2C+3G
stepping        : 0
microcode       : 0x6006705
cpu MHz         : 3421.891
cache size      : 1024 KB
physical id     : 0
siblings        : 2
core id         : 1
cpu cores       : 2
apicid          : 17
initial apicid  : 1
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good acc_power nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw ibs xop skinit wdt lwp fma4 tce nodeid_msr tbm perfctr_core perfctr_nb bpext ptsc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 xsaveopt arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov
bugs            : fxsave_leak sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass retbleed
bogomips        : 5791.17
TLB size        : 1536 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 48 bits physical, 48 bits virtual
power management: ts ttp tm 100mhzsteps hwpstate cpb eff_freq_ro acc_power [13]

Linux:

5.19.10-arch1-1 #1 SMP PREEMPT_DYNAMIC x86_64 GNU/Linux

CL Info:

clinfo
Number of platforms                               1
  Platform Name                                   Clover
  Platform Vendor                                 Mesa
  Platform Version                                OpenCL 1.1 Mesa 22.2.1
  Platform Profile                                FULL_PROFILE
  Platform Extensions                             cl_khr_icd
  Platform Extensions function suffix             MESA

  Platform Name                                   Clover
Number of devices                                 1
  Device Name                                     STONEY (stoney, LLVM 14.0.6, DRM 3.47, 5.19.10-arch1-1)
  Device Vendor                                   AMD
  Device Vendor ID                                0x1002
  Device Version                                  OpenCL 1.1 Mesa 22.2.1
  Device Numeric Version                          0x401000 (1.1.0)
  Driver Version                                  22.2.1
  Device OpenCL C Version                         OpenCL C 1.1 
  Device Type                                     GPU
  Device Profile                                  FULL_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Max compute units                               3
  Max clock frequency                             800MHz
  Max work item dimensions                        3
  Max work item sizes                             256x256x256
  Max work group size                             256
  Preferred work group size multiple (kernel)     64
  Preferred / native vector sizes                 
    char                                                16 / 16      
    short                                                8 / 8       
    int                                                  4 / 4       
    long                                                 2 / 2       
    half                                                 0 / 0        (n/a)
    float                                                4 / 4       
    double                                               2 / 2        (cl_khr_fp64)
  Half-precision Floating-point support           (n/a)
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 No
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Double-precision Floating-point support         (cl_khr_fp64)
    Denormals                                     Yes
    Infinity and NANs                             Yes
    Round to nearest                              Yes
    Round to zero                                 Yes
    Round to infinity                             Yes
    IEEE754-2008 fused multiply-add               Yes
    Support is emulated in software               No
  Address bits                                    64, Little-Endian
  Global memory size                              8102260736 (7.546GiB)
  Error Correction support                        No
  Max memory allocation                           2025565184 (1.886GiB)
  Unified memory for Host and Device              No
  Minimum alignment for any data type             128 bytes
  Alignment of base address                       32768 bits (4096 bytes)
  Global Memory cache type                        None
  Image support                                   No
  Local memory type                               Local
  Local memory size                               65536 (64KiB)
  Max number of constant args                     16
  Max constant buffer size                        67108864 (64MiB)
  Max size of kernel argument                     1024
  Queue properties                                
    Out-of-order execution                        No
    Profiling                                     Yes
  Profiling timer resolution                      0ns
  Execution capabilities                          
    Run OpenCL kernels                            Yes
    Run native kernels                            No
    ILs with version                              (n/a)
  Built-in kernels with version                   (n/a)
  Device Extensions                               cl_khr_byte_addressable_store cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_int64_base_atomics cl_khr_int64_extended_atomics cl_khr_fp64 cl_khr_extended_versioning
  Device Extensions with Version                  cl_khr_byte_addressable_store                                    0x400000 (1.0.0)
                                                  cl_khr_global_int32_base_atomics                                 0x400000 (1.0.0)
                                                  cl_khr_global_int32_extended_atomics                             0x400000 (1.0.0)
                                                  cl_khr_local_int32_base_atomics                                  0x400000 (1.0.0)
                                                  cl_khr_local_int32_extended_atomics                              0x400000 (1.0.0)
                                                  cl_khr_int64_base_atomics                                        0x400000 (1.0.0)
                                                  cl_khr_int64_extended_atomics                                    0x400000 (1.0.0)
                                                  cl_khr_fp64                                                      0x400000 (1.0.0)
                                                  cl_khr_extended_versioning                                       0x400000 (1.0.0)

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  Clover
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [MESA]
  clCreateContext(NULL, ...) [default]            Success [MESA]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 Clover
    Device Name                                   STONEY (stoney, LLVM 14.0.6, DRM 3.47, 5.19.10-arch1-1)
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 Clover
    Device Name                                   STONEY (stoney, LLVM 14.0.6, DRM 3.47, 5.19.10-arch1-1)
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 Clover
    Device Name                                   STONEY (stoney, LLVM 14.0.6, DRM 3.47, 5.19.10-arch1-1)

ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.3.1
  ICD loader Profile                              OpenCL 3.0

[artyom-beilis]

For such an old GPU it seems to be quite impressive. It looks like 300 GFlops device with 3 comute units and 800MHz clock. It is less than Intel 400GFlops GPU and still it performs ~30% faster than Intel GPU..

Can you run dlprim_flops on this device to see what is the base line and how each operation performs?

[trholding]

It doesn't do 300, max ~100 on a few, but even then dlprimitives is still awesome as it blows away the CPU.

Here is the log:

/opt/dlprim/bin/dlprim_flops 0:0 0.5
Testing on STONEY (stoney, LLVM 14.0.6, DRM 3.47, 5.19.10-arch1-1) on Clover
Testing memory speed
- Vector size 1
-- Warming 
-- Running   8.52597 GB/s
- Vector size 2
-- Warming 
-- Running   6.826 GB/s
- Vector size 4
-- Warming 
-- Running   8.64148 GB/s
- Vector size 8
-- Warming 
-- Running   7.36691 GB/s
- Vector size 16
-- Warming 
-- Running   5.32196 GB/s
Testing flops float
- Vector size 1
-- Warming 
-- Running   140.41 GFlops
- Vector size 2
-- Warming 
-- Running   141.694 GFlops
- Vector size 4
-- Warming 
-- Running   138.562 GFlops
- Vector size 8
-- Warming 
-- Running   128.822 GFlops
- Vector size 16
-- Warming 
-- Running   114.076 GFlops
Summray for STONEY (stoney, LLVM 14.0.6, DRM 3.47, 5.19.10-arch1-1) on Clover
Peak GFlops for float 141.694
Peak memory 8.64148 GB/s
GEMM
  NN  0:  512,  512,  512       72.6 GFlops (51.24%)      0.9 GB/s (16.00%) limited by gflops 51.24%
  NN  1: 1024, 1024, 1024       84.8 GFlops (59.83%)      0.5 GB/s ( 9.34%) limited by gflops 59.83%
  NN  2: 1025, 1025, 1025       70.8 GFlops (49.95%)      0.4 GB/s ( 7.79%) limited by gflops 49.95%
  NN  3: 2048, 2048, 2048       88.3 GFlops (62.32%)      0.3 GB/s ( 4.86%) limited by gflops 62.32%
  NN  4: 2049, 2049, 2049       78.5 GFlops (55.39%)      0.2 GB/s ( 4.32%) limited by gflops 55.39%
  NN  5:   64, 2048,   64       32.1 GFlops (22.62%)      2.1 GB/s (38.53%) limited by memory 38.53%
  NN  6: 2048,   64, 2048       50.8 GFlops (35.84%)      1.7 GB/s (31.69%) limited by gflops 35.84%
  NN  7: 2048, 2048,   64       66.8 GFlops (47.15%)      2.2 GB/s (42.01%) limited by gflops 47.15%
  NN  8: 2048,   64,   64       29.8 GFlops (21.06%)      1.9 GB/s (35.87%) limited by memory 35.87%
  NN  9:   64, 2048, 2048       52.9 GFlops (37.36%)      1.8 GB/s (33.04%) limited by gflops 37.36%
  NN 10:   64,   64, 2048       25.5 GFlops (18.00%)      1.6 GB/s (30.42%) limited by memory 30.42%
  NT  0:  512,  512,  512       68.2 GFlops (48.14%)      0.8 GB/s (15.03%) limited by gflops 48.14%
  NT  1: 1024, 1024, 1024       85.9 GFlops (60.60%)      0.5 GB/s ( 9.46%) limited by gflops 60.60%
  NT  2: 1025, 1025, 1025       71.0 GFlops (50.09%)      0.4 GB/s ( 7.81%) limited by gflops 50.09%
  NT  3: 2048, 2048, 2048       89.7 GFlops (63.31%)      0.3 GB/s ( 4.94%) limited by gflops 63.31%
  NT  4: 2049, 2049, 2049       80.1 GFlops (56.53%)      0.2 GB/s ( 4.41%) limited by gflops 56.53%
  NT  5:   64, 2048,   64       37.7 GFlops (26.58%)      2.4 GB/s (45.28%) limited by memory 45.28%
  NT  6: 2048,   64, 2048       65.0 GFlops (45.90%)      2.2 GB/s (40.59%) limited by gflops 45.90%
  NT  7: 2048, 2048,   64       69.7 GFlops (49.21%)      2.3 GB/s (43.85%) limited by gflops 49.21%
  NT  8: 2048,   64,   64       36.4 GFlops (25.70%)      2.3 GB/s (43.78%) limited by memory 43.78%
  NT  9:   64, 2048, 2048       66.6 GFlops (47.02%)      2.2 GB/s (41.57%) limited by gflops 47.02%
  NT 10:   64,   64, 2048       22.6 GFlops (15.94%)      1.4 GB/s (26.94%) limited by memory 26.94%
  TN  0:  512,  512,  512       65.8 GFlops (46.41%)      0.8 GB/s (14.49%) limited by gflops 46.41%
  TN  1: 1024, 1024, 1024       82.7 GFlops (58.34%)      0.5 GB/s ( 9.11%) limited by gflops 58.34%
  TN  2: 1025, 1025, 1025       68.8 GFlops (48.57%)      0.4 GB/s ( 7.57%) limited by gflops 48.57%
  TN  3: 2048, 2048, 2048       86.6 GFlops (61.09%)      0.3 GB/s ( 4.77%) limited by gflops 61.09%
  TN  4: 2049, 2049, 2049       77.2 GFlops (54.50%)      0.2 GB/s ( 4.25%) limited by gflops 54.50%
  TN  5:   64, 2048,   64       29.7 GFlops (20.96%)      1.9 GB/s (35.71%) limited by memory 35.71%
  TN  6: 2048,   64, 2048       46.8 GFlops (33.04%)      1.6 GB/s (29.21%) limited by gflops 33.04%
  TN  7: 2048, 2048,   64       65.5 GFlops (46.22%)      2.2 GB/s (41.18%) limited by gflops 46.22%
  TN  8: 2048,   64,   64       29.4 GFlops (20.74%)      1.9 GB/s (35.32%) limited by memory 35.32%
  TN  9:   64, 2048, 2048       47.5 GFlops (33.55%)      1.6 GB/s (29.67%) limited by gflops 33.55%
  TN 10:   64,   64, 2048       26.4 GFlops (18.61%)      1.7 GB/s (31.46%) limited by memory 31.46%
  TT  0:  512,  512,  512       66.6 GFlops (46.98%)      0.8 GB/s (14.67%) limited by gflops 46.98%
  TT  1: 1024, 1024, 1024       82.8 GFlops (58.44%)      0.5 GB/s ( 9.12%) limited by gflops 58.44%
  TT  2: 1025, 1025, 1025       69.1 GFlops (48.77%)      0.4 GB/s ( 7.60%) limited by gflops 48.77%
  TT  3: 2048, 2048, 2048       86.6 GFlops (61.10%)      0.3 GB/s ( 4.77%) limited by gflops 61.10%
  TT  4: 2049, 2049, 2049       77.5 GFlops (54.71%)      0.2 GB/s ( 4.27%) limited by gflops 54.71%
  TT  5:   64, 2048,   64       33.3 GFlops (23.53%)      2.1 GB/s (40.08%) limited by memory 40.08%
  TT  6: 2048,   64, 2048       55.7 GFlops (39.34%)      1.9 GB/s (34.78%) limited by gflops 39.34%
  TT  7: 2048, 2048,   64       67.3 GFlops (47.47%)      2.3 GB/s (42.30%) limited by gflops 47.47%
  TT  8: 2048,   64,   64       29.2 GFlops (20.60%)      1.9 GB/s (35.10%) limited by memory 35.10%
  TT  9:   64, 2048, 2048       55.5 GFlops (39.15%)      1.8 GB/s (34.62%) limited by gflops 39.15%
  TT 10:   64,   64, 2048       25.8 GFlops (18.18%)      1.6 GB/s (30.73%) limited by memory 30.73%
Convolution
   0     effnet  forward b=64 k=3  p=1 s=1 in=480  out=480  g=480 D=14       10.9 GFlops ( 7.70%)      4.9 GB/s (91.15%) limited by memory 91.15% algo=depthwise_separable
   0     effnet bwd-data b=64 k=3  p=1 s=1 in=480  out=480  g=480 D=14        1.1 GFlops ( 0.74%)      0.5 GB/s ( 8.80%) limited by memory  8.80% algo=depthwise_separable
   0     effnet bwd-filt b=64 k=3  p=1 s=1 in=480  out=480  g=480 D=14        0.9 GFlops ( 0.67%)      0.4 GB/s ( 7.93%) limited by memory  7.93% algo=depthwise_separable
   1    alexnet  forward b=64 k=11 p=2 s=4 in=3    out=64   g=1   D=224      67.3 GFlops (47.48%)      0.7 GB/s (12.39%) limited by gflops 47.48% algo=gemm
   1    alexnet bwd-data b=64 k=11 p=2 s=4 in=3    out=64   g=1   D=224      43.6 GFlops (30.79%)      0.4 GB/s ( 8.04%) limited by gflops 30.79% algo=gemm
   1    alexnet bwd-filt b=64 k=11 p=2 s=4 in=3    out=64   g=1   D=224      51.2 GFlops (36.15%)      0.5 GB/s ( 9.45%) limited by gflops 36.15% algo=gemm
   2    alexnet  forward b=64 k=5  p=2 s=1 in=96   out=192  g=2   D=27       53.2 GFlops (37.52%)      0.1 GB/s ( 2.54%) limited by gflops 37.52% algo=gemm
   2    alexnet bwd-data b=64 k=5  p=2 s=1 in=96   out=192  g=2   D=27       54.0 GFlops (38.13%)      0.1 GB/s ( 2.58%) limited by gflops 38.13% algo=gemm
   2    alexnet bwd-filt b=64 k=5  p=2 s=1 in=96   out=192  g=2   D=27       51.1 GFlops (36.03%)      0.1 GB/s ( 2.48%) limited by gflops 36.03% algo=gemm
   3    alexnet  forward b=64 k=5  p=2 s=1 in=64   out=192  g=1   D=27       69.8 GFlops (49.25%)      0.1 GB/s ( 2.24%) limited by gflops 49.25% algo=gemm
   3    alexnet bwd-data b=64 k=5  p=2 s=1 in=64   out=192  g=1   D=27       64.3 GFlops (45.40%)      0.1 GB/s ( 2.07%) limited by gflops 45.40% algo=gemm
   3    alexnet bwd-filt b=64 k=5  p=2 s=1 in=64   out=192  g=1   D=27       69.3 GFlops (48.88%)      0.1 GB/s ( 2.28%) limited by gflops 48.88% algo=gemm
   4    alexnet  forward b=64 k=3  p=1 s=1 in=384  out=256  g=1   D=13      103.9 GFlops (73.31%)      0.2 GB/s ( 3.18%) limited by gflops 73.31% algo=winograd
   4    alexnet bwd-data b=64 k=3  p=1 s=1 in=384  out=256  g=1   D=13       89.4 GFlops (63.10%)      0.1 GB/s ( 2.74%) limited by gflops 63.10% algo=winograd
   4    alexnet bwd-filt b=64 k=3  p=1 s=1 in=384  out=256  g=1   D=13      106.2 GFlops (74.95%)      0.2 GB/s ( 3.62%) limited by gflops 74.95% algo=winograd
   5     resnet  forward b=64 k=7  p=3 s=2 in=3    out=64   g=1   D=224      61.4 GFlops (43.32%)      1.0 GB/s (18.64%) limited by gflops 43.32% algo=gemm
   5     resnet bwd-data b=64 k=7  p=3 s=2 in=3    out=64   g=1   D=224      41.1 GFlops (29.01%)      0.7 GB/s (12.48%) limited by gflops 29.01% algo=gemm
   5     resnet bwd-filt b=64 k=7  p=3 s=2 in=3    out=64   g=1   D=224      49.4 GFlops (34.86%)      0.8 GB/s (15.00%) limited by gflops 34.86% algo=gemm
   6     resnet  forward b=64 k=1  p=0 s=1 in=64   out=256  g=1   D=56       34.5 GFlops (24.38%)      1.3 GB/s (25.36%) limited by memory 25.36% algo=gemm
   6     resnet bwd-data b=64 k=1  p=0 s=1 in=64   out=256  g=1   D=56       63.6 GFlops (44.92%)      2.5 GB/s (46.72%) limited by memory 46.72% algo=gemm
   6     resnet bwd-filt b=64 k=1  p=0 s=1 in=64   out=256  g=1   D=56       38.0 GFlops (26.81%)      1.5 GB/s (27.90%) limited by memory 27.90% algo=gemm
   7     resnet  forward b=64 k=1  p=0 s=1 in=64   out=64   g=1   D=56       62.1 GFlops (43.85%)      3.9 GB/s (72.97%) limited by memory 72.97% algo=gemm
   7     resnet bwd-data b=64 k=1  p=0 s=1 in=64   out=64   g=1   D=56       44.3 GFlops (31.27%)      2.8 GB/s (52.04%) limited by memory 52.04% algo=gemm
   7     resnet bwd-filt b=64 k=1  p=0 s=1 in=64   out=64   g=1   D=56       54.5 GFlops (38.46%)      3.4 GB/s (64.02%) limited by memory 64.02% algo=gemm
   8     resnet  forward b=64 k=3  p=1 s=1 in=64   out=64   g=1   D=56       94.3 GFlops (66.52%)      0.7 GB/s (12.32%) limited by gflops 66.52% algo=winograd
   8     resnet bwd-data b=64 k=3  p=1 s=1 in=64   out=64   g=1   D=56       60.5 GFlops (42.69%)      0.4 GB/s ( 7.90%) limited by gflops 42.69% algo=winograd
   8     resnet bwd-filt b=64 k=3  p=1 s=1 in=64   out=64   g=1   D=56       80.2 GFlops (56.62%)      0.6 GB/s (10.50%) limited by gflops 56.62% algo=winograd
   9     resnet  forward b=64 k=1  p=0 s=2 in=1024 out=2048 g=1   D=14       45.6 GFlops (32.19%)      0.3 GB/s ( 5.57%) limited by gflops 32.19% algo=gemm
   9     resnet bwd-data b=64 k=1  p=0 s=2 in=1024 out=2048 g=1   D=14       67.7 GFlops (47.79%)      0.4 GB/s ( 8.27%) limited by gflops 47.79% algo=gemm
   9     resnet bwd-filt b=64 k=1  p=0 s=2 in=1024 out=2048 g=1   D=14       66.9 GFlops (47.25%)      0.5 GB/s ( 8.98%) limited by gflops 47.25% algo=gemm
  10     resnet  forward b=64 k=1  p=0 s=1 in=1024 out=256  g=1   D=14       50.2 GFlops (35.42%)      0.5 GB/s ( 9.36%) limited by gflops 35.42% algo=gemm
  10     resnet bwd-data b=64 k=1  p=0 s=1 in=1024 out=256  g=1   D=14       55.9 GFlops (39.47%)      0.6 GB/s (10.43%) limited by gflops 39.47% algo=gemm
  10     resnet bwd-filt b=64 k=1  p=0 s=1 in=1024 out=256  g=1   D=14       66.7 GFlops (47.04%)      0.7 GB/s (12.63%) limited by gflops 47.04% algo=gemm
  11     resnet  forward b=64 k=3  p=1 s=1 in=256  out=256  g=1   D=14      109.4 GFlops (77.18%)      0.2 GB/s ( 3.90%) limited by gflops 77.18% algo=winograd
  11     resnet bwd-data b=64 k=3  p=1 s=1 in=256  out=256  g=1   D=14       98.5 GFlops (69.51%)      0.2 GB/s ( 3.51%) limited by gflops 69.51% algo=winograd
  11     resnet bwd-filt b=64 k=3  p=1 s=1 in=256  out=256  g=1   D=14      110.6 GFlops (78.03%)      0.2 GB/s ( 4.27%) limited by gflops 78.03% algo=winograd
  12        vgg  forward b=64 k=3  p=1 s=1 in=3    out=64   g=1   D=224      43.1 GFlops (30.45%)      3.3 GB/s (62.86%) limited by memory 62.86% algo=gemm
  12        vgg bwd-data b=64 k=3  p=1 s=1 in=3    out=64   g=1   D=224      12.2 GFlops ( 8.58%)      0.9 GB/s (17.71%) limited by memory 17.71% algo=gemm
  12        vgg bwd-filt b=64 k=3  p=1 s=1 in=3    out=64   g=1   D=224      11.3 GFlops ( 7.99%)      0.9 GB/s (16.51%) limited by memory 16.51% algo=winograd
  13        vgg  forward b=64 k=3  p=1 s=1 in=64   out=64   g=1   D=224      96.4 GFlops (68.03%)      0.7 GB/s (12.58%) limited by gflops 68.03% algo=winograd
  13        vgg bwd-data b=64 k=3  p=1 s=1 in=64   out=64   g=1   D=224      60.7 GFlops (42.81%)      0.4 GB/s ( 7.92%) limited by gflops 42.81% algo=winograd
  13        vgg bwd-filt b=64 k=3  p=1 s=1 in=64   out=64   g=1   D=224      84.9 GFlops (59.90%)      0.6 GB/s (11.08%) limited by gflops 59.90% algo=winograd
  14        vgg  forward b=64 k=3  p=1 s=1 in=512  out=512  g=1   D=28      122.8 GFlops (86.65%)      0.1 GB/s ( 2.09%) limited by gflops 86.65% algo=winograd
  14        vgg bwd-data b=64 k=3  p=1 s=1 in=512  out=512  g=1   D=28      118.7 GFlops (83.74%)      0.1 GB/s ( 2.02%) limited by gflops 83.74% algo=winograd
  14        vgg bwd-filt b=64 k=3  p=1 s=1 in=512  out=512  g=1   D=28      118.3 GFlops (83.51%)      0.1 GB/s ( 2.11%) limited by gflops 83.51% algo=winograd
  15     mobile  forward b=64 k=3  p=1 s=2 in=3    out=32   g=1   D=224      34.9 GFlops (24.62%)      3.6 GB/s (66.77%) limited by memory 66.77% algo=gemm
  15     mobile bwd-data b=64 k=3  p=1 s=2 in=3    out=32   g=1   D=224      11.4 GFlops ( 8.04%)      1.2 GB/s (21.79%) limited by memory 21.79% algo=gemm
  15     mobile bwd-filt b=64 k=3  p=1 s=2 in=3    out=32   g=1   D=224      17.6 GFlops (12.41%)      1.8 GB/s (33.65%) limited by memory 33.65% algo=gemm
  16     mobile  forward b=64 k=3  p=1 s=1 in=144  out=144  g=144 D=56        6.9 GFlops ( 4.87%)      3.1 GB/s (57.65%) limited by memory 57.65% algo=depthwise_separable
  16     mobile bwd-data b=64 k=3  p=1 s=1 in=144  out=144  g=144 D=56        1.1 GFlops ( 0.78%)      0.5 GB/s ( 9.26%) limited by memory  9.26% algo=depthwise_separable
  17     mobile  forward b=64 k=3  p=1 s=2 in=144  out=144  g=144 D=56        0.3 GFlops ( 0.23%)      0.4 GB/s ( 6.75%) limited by memory  6.75% algo=gemm
  17     mobile bwd-data b=64 k=3  p=1 s=2 in=144  out=144  g=144 D=56        0.3 GFlops ( 0.19%)      0.3 GB/s ( 5.55%) limited by memory  5.55% algo=gemm
  17     mobile bwd-filt b=64 k=3  p=1 s=2 in=144  out=144  g=144 D=56        0.6 GFlops ( 0.46%)      0.7 GB/s (13.57%) limited by memory 13.57% algo=gemm
  18     mobile  forward b=64 k=1  p=0 s=1 in=144  out=24   g=1   D=56       20.3 GFlops (14.34%)      2.0 GB/s (37.11%) limited by memory 37.11% algo=gemm
  18     mobile bwd-data b=64 k=1  p=0 s=1 in=144  out=24   g=1   D=56       12.0 GFlops ( 8.48%)      1.2 GB/s (21.96%) limited by memory 21.96% algo=gemm
  18     mobile bwd-filt b=64 k=1  p=0 s=1 in=144  out=24   g=1   D=56       22.5 GFlops (15.90%)      2.2 GB/s (41.16%) limited by memory 41.16% algo=gemm
  19     mobile  forward b=64 k=1  p=0 s=1 in=24   out=144  g=1   D=56       32.8 GFlops (23.18%)      3.2 GB/s (60.01%) limited by memory 60.01% algo=gemm
  19     mobile bwd-data b=64 k=1  p=0 s=1 in=24   out=144  g=1   D=56       17.5 GFlops (12.35%)      1.7 GB/s (31.98%) limited by memory 31.98% algo=gemm
  19     mobile bwd-filt b=64 k=1  p=0 s=1 in=24   out=144  g=1   D=56       22.3 GFlops (15.74%)      2.2 GB/s (40.76%) limited by memory 40.76% algo=gemm
  20     mobile  forward b=64 k=1  p=0 s=1 in=960  out=160  g=1   D=7        58.2 GFlops (41.05%)      0.9 GB/s (16.64%) limited by gflops 41.05% algo=gemm
  20     mobile bwd-data b=64 k=1  p=0 s=1 in=960  out=160  g=1   D=7        43.9 GFlops (31.01%)      0.7 GB/s (12.57%) limited by gflops 31.01% algo=gemm
  20     mobile bwd-filt b=64 k=1  p=0 s=1 in=960  out=160  g=1   D=7        57.1 GFlops (40.30%)      0.9 GB/s (17.02%) limited by gflops 40.30% algo=gemm
  21     mobile  forward b=64 k=1  p=0 s=1 in=960  out=320  g=1   D=7        47.2 GFlops (33.28%)      0.4 GB/s ( 7.95%) limited by gflops 33.28% algo=gemm
  21     mobile bwd-data b=64 k=1  p=0 s=1 in=960  out=320  g=1   D=7        58.8 GFlops (41.52%)      0.5 GB/s ( 9.92%) limited by gflops 41.52% algo=gemm
  21     mobile bwd-filt b=64 k=1  p=0 s=1 in=960  out=320  g=1   D=7        61.9 GFlops (43.71%)      0.6 GB/s (11.18%) limited by gflops 43.71% algo=gemm
  22     mobile  forward b=64 k=3  p=1 s=1 in=960  out=960  g=960 D=7         7.8 GFlops ( 5.47%)      3.5 GB/s (64.85%) limited by memory 64.85% algo=depthwise_separable
  22     mobile bwd-data b=64 k=3  p=1 s=1 in=960  out=960  g=960 D=7         1.1 GFlops ( 0.75%)      0.5 GB/s ( 8.86%) limited by memory  8.86% algo=depthwise_separable
  22     mobile bwd-filt b=64 k=3  p=1 s=1 in=960  out=960  g=960 D=7         2.1 GFlops ( 1.45%)      0.9 GB/s (17.22%) limited by memory 17.22% algo=depthwise_separable
  23      scale  forward b=64 k=1  p=0 s=1 in=256  out=256  g=256 D=56        1.5 GFlops ( 1.06%)      6.0 GB/s (113.30%) limited by memory 113.30% algo=depthwise_separable
  23      scale bwd-data b=64 k=1  p=0 s=1 in=256  out=256  g=256 D=56        1.1 GFlops ( 0.75%)      4.3 GB/s (79.95%) limited by memory 79.95% algo=depthwise_separable
  23      scale bwd-filt b=64 k=1  p=0 s=1 in=256  out=256  g=256 D=56        1.1 GFlops ( 0.81%)      4.6 GB/s (85.99%) limited by memory 85.99% algo=depthwise_separable
  24      scale  forward b=64 k=1  p=0 s=1 in=1024 out=1024 g=1024 D=7         1.3 GFlops ( 0.91%)      5.2 GB/s (97.19%) limited by memory 97.19% algo=depthwise_separable
  24      scale bwd-data b=64 k=1  p=0 s=1 in=1024 out=1024 g=1024 D=7         0.6 GFlops ( 0.45%)      2.5 GB/s (47.43%) limited by memory 47.43% algo=depthwise_separable
  24      scale bwd-filt b=64 k=1  p=0 s=1 in=1024 out=1024 g=1024 D=7         0.3 GFlops ( 0.18%)      1.0 GB/s (19.32%) limited by memory 19.32% algo=depthwise_separable
Broadcast/Reduce
     float (64,512,24,24)  (64,512,24,24)  (64,512,24,24)         1.9 GFlops ( 1.35%)     11.4 GB/s (214.99%) limited by memory 214.99%
     float (64,512,24,24)  (512,1,1)       (64,512,24,24)         1.3 GFlops ( 0.90%)      5.1 GB/s (95.55%) limited by memory 95.55%
     float (64,512,24,24)  (1,512,1,1)     (1,512,1,1)            1.6 GFlops ( 1.14%)      2.2 GB/s (40.52%) limited by memory 40.52%
     float (64,512,24,24)  (64,512,24,24)  (1,512,1,1)            1.5 GFlops ( 1.08%)      4.1 GB/s (76.84%) limited by memory 76.84%
     float (64,512,24,24)  (64,512,24,24)  (64,1,1,1)             1.3 GFlops ( 0.95%)      3.6 GB/s (67.60%) limited by memory 67.60%
     float (256,1000)      (256,1)         (1)                    2.7 GFlops ( 1.92%)      3.6 GB/s (68.30%) limited by memory 68.30%
      long (64,512,24,24)  (64,512,24,24)  (64,512,24,24)         1.0 GFlops ( 0.71%)     12.1 GB/s (227.36%) limited by memory 227.36%
      long (64,512,24,24)  (512,1,1)       (64,512,24,24)         1.1 GFlops ( 0.79%)      8.9 GB/s (167.43%) limited by memory 167.43%
      long (64,512,24,24)  (1,512,1,1)     (1,512,1,1)            1.5 GFlops ( 1.06%)      4.0 GB/s (74.96%) limited by memory 74.96%
      long (64,512,24,24)  (64,512,24,24)  (1,512,1,1)            1.4 GFlops ( 0.96%)      7.3 GB/s (136.46%) limited by memory 136.46%
      long (64,512,24,24)  (64,512,24,24)  (64,1,1,1)             0.5 GFlops ( 0.38%)      2.8 GB/s (53.43%) limited by memory 53.43%
      long (256,1000)      (256,1)         (1)                    2.2 GFlops ( 1.57%)      5.9 GB/s (111.55%) limited by memory 111.55%
     short (64,512,24,24)  (64,512,24,24)  (64,512,24,24)         2.9 GFlops ( 2.07%)      8.8 GB/s (165.51%) limited by memory 165.51%
     short (64,512,24,24)  (512,1,1)       (64,512,24,24)         1.3 GFlops ( 0.92%)      2.6 GB/s (49.22%) limited by memory 49.22%
     short (64,512,24,24)  (1,512,1,1)     (1,512,1,1)            1.6 GFlops ( 1.15%)      1.1 GB/s (20.35%) limited by memory 20.35%
     short (64,512,24,24)  (64,512,24,24)  (1,512,1,1)            1.6 GFlops ( 1.15%)      2.2 GB/s (40.72%) limited by memory 40.72%
     short (64,512,24,24)  (64,512,24,24)  (64,1,1,1)             4.9 GFlops ( 3.43%)      6.5 GB/s (121.67%) limited by memory 121.67%
     short (256,1000)      (256,1)         (1)                    2.8 GFlops ( 1.96%)      1.9 GB/s (34.81%) limited by memory 34.81%

Took a long time to finish

[artyom-beilis]

Looks like it gets to ~140GFlops, maybe at this architecture FMA runs in two cicles rather than one so it is clock * number of threads > 800*3*64 /1000 -> 153GFlops

Still it seems to reach very good utilization of the maximal performance.

[trholding]

Yes I am impressed, you made old hardware learn new tricks :)

On the other hand I tested dlprimitives on the same hardware as above but with opencl via pocl on cpu and it triggered a segfault midway and a dump of the kernel it was last running.

I'll paste it in a while, right now compiling a few drivers and stuff, so that system is not free.

pocl is something very interesting and important generally, testing and assuring it works with pocl has advantages to dlprimitives - for example you could utilize both cpu and gpu, fpga, and custom accelerators (pocl ports for fpga based compute and custom silicon with 100 to a lot of gflops) thru opencl & pocl.

I think this is a bug in pocl. Let me paste the log as soon as I can so you can get a better idea.

[artyom-beilis]

pocl on cpu and it triggered a segfault midway and a dump of the kernel it was last running.

I never managed to run with pocl - it crashes all the time even on trivial kernels.

I did run on CPU using Intel OpenCL runtime for CPU but the performance is far from optimal - OpenBLAS + im2col works almost twice as fast. Once again I hadn't optimized GEMM for CPU.

Also the code isn't optimized for CPU. DLPrimitives provide CPU support using OpenBLAS/GEMM + im2col. That isn't very fast (Intel's implementation using MKLDNN is faster) but yet is similar in performance of CPU on Caffe.

[artyom-beilis]

 ./dlprim_benchmark -b cpu ../docs/nets_for_benchmark/resnet18-b16.js 
Using: CPU
Inputs
- data: (16,3,224,224)
Outputs
- loss: (16,1000)
Step -5   2223.291   813.627  1409.663
Step -4   2190.685   773.928  1416.757

$ ./dlprim_benchmark -b 3:0 ../docs/nets_for_benchmark/resnet18-b16.js 
Using: Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz on Intel(R) OpenCL
Inputs
- data: (16,3,224,224)
Outputs
- loss: (16,1000)
Step -5   5709.108  2019.580  3689.528
Step -4   5441.142  1897.707  3543.435

$ ./dlprim_benchmark -b 4:0 ../docs/nets_for_benchmark/resnet18-b16.js 
Using: pthread-Intel(R) Core(TM) i5-6600 CPU @ 3.30GHz on Portable Computing Language
Inputs
- data: (16,3,224,224)
Outputs
- loss: (16,1000)
Step -5  71967.467 19909.680 52057.787

So CPU OpenCL is either not optimal yet and POCL at all.

[trholding]

Do you think it's possible to crosscompile this to Raspberry Pi Zero? That way we could see how it would run on the most minimum of hardware, and can serve as an estimate to people how much compute and memory they need for their various models. (I'll try the cross compiling thing)

OpenCL for raspi: https://github.com/doe300/VC4CL

See this link for info: https://kenny-peng.com/2021/09/14/raspi_zero_opencl.html

The clinfo output is like this: (copied it from the site above)

pi@raspberrypi:~ $ sudo clinfo
Number of platforms                               1
  Platform Name                                   OpenCL for the Raspberry Pi VideoCore IV GPU
  Platform Vendor                                 doe300
  Platform Version                                OpenCL 1.2 VC4CL 0.4.9999 (2cf1d93)
  Platform Profile                                EMBEDDED_PROFILE
  Platform Extensions                             cl_khr_il_program cl_khr_spir cl_khr_create_command_queue cl_altera_device_temperature cl_altera_live_object_tracking cl_khr_icd cl_khr_extended_versioning cl_khr_spirv_no_integer_wrap_decoration cl_khr_suggested_local_work_size cl_vc4cl_performance_counters
  Platform Extensions function suffix             VC4CL

  Platform Name                                   OpenCL for the Raspberry Pi VideoCore IV GPU
Number of devices                                 1
  Device Name                                     VideoCore IV GPU
  Device Vendor                                   Broadcom
  Device Vendor ID                                0x14e4
  Device Version                                  OpenCL 1.2 VC4CL 0.4.9999 (2cf1d93)
  Driver Version                                  0.4.9999
  Device OpenCL C Version                         OpenCL C 1.2
  Device Type                                     GPU
  Device Profile                                  EMBEDDED_PROFILE
  Device Available                                Yes
  Compiler Available                              Yes
  Linker Available                                Yes
  Max compute units                               1
  Max clock frequency                             300MHz
  Core Temperature (Altera)                       31 C
  Device Partition                                (core)
    Max number of sub-devices                     0
    Supported partition types                     None
    Supported affinity domains                    (n/a)
  Max work item dimensions                        3
  Max work item sizes                             12x12x12
  Max work group size                             12
  Preferred work group size multiple              1
  Preferred / native vector sizes
    char                                                16 / 16
    short                                               16 / 16
    int                                                 16 / 16
    long                                                 0 / 0
    half                                                 0 / 0        (n/a)
    float                                               16 / 16
    double                                               0 / 0        (n/a)
  Half-precision Floating-point support           (n/a)
  Single-precision Floating-point support         (core)
    Denormals                                     No
    Infinity and NANs                             No
    Round to nearest                              No
    Round to zero                                 Yes
    Round to infinity                             No
    IEEE754-2008 fused multiply-add               No
    Support is emulated in software               No
    Correctly-rounded divide and sqrt operations  No
  Double-precision Floating-point support         (n/a)
  Address bits                                    32, Little-Endian
  Global memory size                              67108864 (64MiB)
  Error Correction support                        No
  Max memory allocation                           67108864 (64MiB)
  Unified memory for Host and Device              Yes
  Minimum alignment for any data type             64 bytes
  Alignment of base address                       512 bits (64 bytes)
  Global Memory cache type                        Read/Write
  Global Memory cache size                        32768 (32KiB)
  Global Memory cache line size                   64 bytes
  Image support                                   No
  Local memory type                               Global
  Local memory size                               67108864 (64MiB)
  Max number of constant args                     32
  Max constant buffer size                        67108864 (64MiB)
  Max size of kernel argument                     256
  Queue properties
    Out-of-order execution                        No
    Profiling                                     Yes
  Prefer user sync for interop                    Yes
  Profiling timer resolution                      1ns
  Execution capabilities
    Run OpenCL kernels                            Yes
    Run native kernels                            No
    IL version                                    SPIR-V_1.5 SPIR_1.2
    SPIR versions                                 1.2
  printf() buffer size                            0
  Built-in kernels                                (n/a)
  Device Extensions                               cl_khr_global_int32_base_atomics cl_khr_global_int32_extended_atomics cl_khr_local_int32_base_atomics cl_khr_local_int32_extended_atomics cl_khr_byte_addressable_store cl_nv_pragma_unroll cl_arm_core_id cl_ext_atomic_counters_32 cl_khr_initialize_memory cl_arm_integer_dot_product_int8 cl_arm_integer_dot_product_accumulate_int8 cl_arm_integer_dot_product_accumulate_int16 cl_arm_integer_dot_product_accumulate_saturate_int8 cl_khr_il_program cl_khr_spir cl_khr_create_command_queue cl_altera_device_temperature cl_altera_live_object_tracking cl_khr_icd cl_khr_extended_versioning cl_khr_spirv_no_integer_wrap_decoration cl_khr_suggested_local_work_size cl_vc4cl_performance_counters

NULL platform behavior
  clGetPlatformInfo(NULL, CL_PLATFORM_NAME, ...)  OpenCL for the Raspberry Pi VideoCore IV GPU
  clGetDeviceIDs(NULL, CL_DEVICE_TYPE_ALL, ...)   Success [VC4CL]
  clCreateContext(NULL, ...) [default]            Success [VC4CL]
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_DEFAULT)  Success (1)
    Platform Name                                 OpenCL for the Raspberry Pi VideoCore IV GPU
    Device Name                                   VideoCore IV GPU
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CPU)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_GPU)  Success (1)
    Platform Name                                 OpenCL for the Raspberry Pi VideoCore IV GPU
    Device Name                                   VideoCore IV GPU
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ACCELERATOR)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_CUSTOM)  No devices found in platform
  clCreateContextFromType(NULL, CL_DEVICE_TYPE_ALL)  Success (1)
    Platform Name                                 OpenCL for the Raspberry Pi VideoCore IV GPU
    Device Name                                   VideoCore IV GPU

ICD loader properties
  ICD loader Name                                 OpenCL ICD Loader
  ICD loader Vendor                               OCL Icd free software
  ICD loader Version                              2.2.12
  ICD loader Profile                              OpenCL 2.2

[artyom-beilis]

Max work item dimensions 3
Max work item sizes 12x12x12
Max work group size 12

This is quite problematic, that means most of algorithms wouldn't work as expected. Parallel reductions need different algorithms, GEMM in current implementation would not work. It needs different architecture. It runs (at least once was tested) on Mali and performance wasn't very optimal.

So unless centeral algorithms will be tuned/rewritten for this GPU I don't thing this would work. I can't say right now how much work is it to adapt the code to this GPU and restrictions.

[trholding]

I guess there is no point in supporting ultra low end GPUs then as the algorithms itself have minimum requirements to be effective.