GPU attempting to create too many threads

[RicksterB57]

I have been attempting to run a 500x500x200 meter model on a 256x256x128 grid on a GPU and have been encountering this error:
ERROR: LoadError: Number of threads per block exceeds kernel limit (896 > 768).
I have been able to run this model on the same GPU in the past. Any tips on limiting the amount of threads that the model tries to open during the simulation?

[glwagner]

Can you show us the code you are trying to run and also let us know what GPU you are using?

[RicksterB57]

This is the error that I get on Julia 1.11 when I try to run the model on my machine's GPU:

ERROR: LoadError: Number of threads per block exceeds kernel limit (896 > 768).
Stacktrace:
  [1] error(s::String)
    @ Base .\error.jl:35
  [2] diagnose_launch_failure(f::CuFunction, err::CuError; blockdim::CuDim3, threaddim::CuDim3, shmem::Int64)
    @ CUDA C:\Users\C2PO\.julia\packages\CUDA\Tl08O\lib\cudadrv\execution.jl:120
  [3] launch(::CuFunction, ::CUDA.KernelState, ::CartesianIndices{…}, ::CartesianIndices{…}, ::CuDeviceArray{…}, ::KernelFunctionOperation{…}; blocks::Int64, threads::Int64, cooperative::Bool, shmem::Int64, stream::CuStream)
    @ CUDA C:\Users\C2PO\.julia\packages\CUDA\Tl08O\lib\cudadrv\execution.jl:73
  [4] launch
    @ C:\Users\C2PO\.julia\packages\CUDA\Tl08O\lib\cudadrv\execution.jl:52 [inlined]
  [5] #972
    @ C:\Users\C2PO\.julia\packages\CUDA\Tl08O\lib\cudadrv\execution.jl:189 [inlined]
  [6] macro expansion
    @ C:\Users\C2PO\.julia\packages\CUDA\Tl08O\lib\cudadrv\execution.jl:149 [inlined]
  [7] macro expansion
    @ .\none:0 [inlined]
  [8] convert_arguments
    @ .\none:0 [inlined]
  [9] #cudacall#971
    @ C:\Users\C2PO\.julia\packages\CUDA\Tl08O\lib\cudadrv\execution.jl:191 [inlined]
 [10] cudacall
    @ C:\Users\C2PO\.julia\packages\CUDA\Tl08O\lib\cudadrv\execution.jl:187 [inlined]
 [11] macro expansion
    @ C:\Users\C2PO\.julia\packages\CUDA\Tl08O\src\compiler\execution.jl:268 [inlined]
 [12] macro expansion
    @ .\none:0 [inlined]
 [13] call(::CUDA.HostKernel{…}, ::typeof(identity), ::typeof(min), ::Nothing, ::CartesianIndices{…}, ::CartesianIndices{…}, ::Val{…}, ::CuDeviceArray{…}, ::KernelFunctionOperation{…}; call_kwargs::@Kwargs{…})
    @ CUDA .\none:0
 [14] call
    @ .\none:0 [inlined]
 [15] (::CUDA.HostKernel{…})(::Function, ::Function, ::Nothing, ::CartesianIndices{…}, ::CartesianIndices{…}, ::Val{…}, ::CuArray{…}, ::KernelFunctionOperation{…}; threads::Int64, blocks::Int64, kwargs::@Kwargs{…})
    @ CUDA C:\Users\C2PO\.julia\packages\CUDA\Tl08O\src\compiler\execution.jl:390
 [16] macro expansion
    @ C:\Users\C2PO\.julia\packages\CUDA\Tl08O\src\compiler\execution.jl:114 [inlined]
 [17] mapreducedim!(f::typeof(identity), op::typeof(min), R::SubArray{…}, A::KernelFunctionOperation{…}; init::Nothing)
    @ CUDA C:\Users\C2PO\.julia\packages\CUDA\Tl08O\src\mapreduce.jl:271
 [18] mapreducedim!(f::typeof(identity), op::typeof(min), R::SubArray{…}, A::KernelFunctionOperation{…})
    @ CUDA C:\Users\C2PO\.julia\packages\CUDA\Tl08O\src\mapreduce.jl:169
 [19] mapreducedim!(f::Function, op::Function, R::SubArray{…}, A::KernelFunctionOperation{…})
    @ GPUArrays C:\Users\C2PO\.julia\packages\GPUArrays\qt4ax\src\host\mapreduce.jl:10
 [20] #minimum!#969
    @ .\reducedim.jl:1006 [inlined]
 [21] minimum!(f::Function, r::Field{…}, a::KernelFunctionOperation{…}; condition::Nothing, mask::Float64, kwargs::@Kwargs{…})
    @ Oceananigans.Fields C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Fields\field.jl:684
 [22] minimum(f::Function, c::KernelFunctionOperation{…}; condition::Nothing, mask::Float64, dims::Function)
    @ Oceananigans.Fields C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Fields\field.jl:714
 [23] minimum
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Fields\field.jl:703 [inlined]
 [24] minimum
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Fields\field.jl:723 [inlined]
 [25] cell_advection_timescale
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Advection\cell_advection_timescale.jl:16 [inlined]
 [26] cell_advection_timescale
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Models\NonhydrostaticModels\NonhydrostaticModels.jl:80 [inlined]
 [27] new_time_step
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Simulations\time_step_wizard.jl:102 [inlined]
 [28] TimeStepWizard
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Simulations\time_step_wizard.jl:116 [inlined]
 [29] (::Callback{…})(sim::Simulation{…})
    @ Oceananigans.Simulations C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Simulations\callback.jl:15
 [30] initialize!(sim::Simulation{…})
    @ Oceananigans.Simulations C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Simulations\run.jl:215
 [31] time_step!(sim::Simulation{…})
    @ Oceananigans.Simulations C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Simulations\run.jl:112
 [32] run!(sim::Simulation{…}; pickup::Bool)
    @ Oceananigans.Simulations C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Simulations\run.jl:97
 [33] run!(sim::Simulation{…})
    @ Oceananigans.Simulations C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Simulations\run.jl:85
 [34] top-level scope
    @ C:\Users\C2PO\Desktop\LES-main\idealized_storm_model_cpu.jl:233
 [35] include(fname::String)
    @ Main .\sysimg.jl:38
 [36] top-level scope
    @ REPL[2]:1
in expression starting at C:\Users\C2PO\Desktop\LES-main\idealized_storm_model_cpu.jl:233

[glwagner]

Oh sorry, I did not realize that you are using Julia 1.11! I don't think Julia 1.11 works yet. At least, we have not yet been able to upgrade our CI to use Julia 1.11, see #3836.

We could have an action item to throw a warning if one is using Julia 1.11. cc @navidcy

[glwagner]

It might help to share your code too. I just noticed that the error comes from minimum. So perhaps the code will work without adaptive time-stepping? Are you also using an ImmersedBoundaryGrid?

Here's the critical part of the stack trace:

 [19] mapreducedim!(f::Function, op::Function, R::SubArray{…}, A::KernelFunctionOperation{…})
    @ GPUArrays C:\Users\C2PO\.julia\packages\GPUArrays\qt4ax\src\host\mapreduce.jl:10
 [20] #minimum!#969
    @ .\reducedim.jl:1006 [inlined]
 [21] minimum!(f::Function, r::Field{…}, a::KernelFunctionOperation{…}; condition::Nothing, mask::Float64, kwargs::@Kwargs{…})
    @ Oceananigans.Fields C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Fields\field.jl:684
 [22] minimum(f::Function, c::KernelFunctionOperation{…}; condition::Nothing, mask::Float64, dims::Function)
    @ Oceananigans.Fields C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Fields\field.jl:714
 [23] minimum
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Fields\field.jl:703 [inlined]
 [24] minimum
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Fields\field.jl:723 [inlined]
 [25] cell_advection_timescale
    @ C:\Users\C2PO\.julia\packages\Oceananigans\7twKV\src\Advection\cell_advection_timescale.jl:16 [inlined]

cell_advection_timescale is used for adaptive time-stepping. The error ultimately originates from within mapreducedim! I'm at a loss for why you get that particular error. Maybe try something like

using Oceananigans
grid = RectilinearGrid(size=(16, 16, 16), extent=(1, 1, 1))
c = CenterField(grid)
c_min = minimum(c)

And add stuff to that snippet until you get your error. Then I can try to run it on GPUs I have, and we can figure out if it is a GPU-specific issue.

[RicksterB57]

No I am not using an ImmersedBoundaryGrid. In terms of the grid and model that I am running

arch = GPU()
grid = RectilinearGrid(arch, size = (Nx, Nx, Nz),
                          x = (0, Lx),
                          y = (0, Ly),
                          z = (-Lz, 0));

# Defining the buoyancy model
buoyancy = SeawaterBuoyancy(equation_of_state=LinearEquationOfState(thermal_expansion = 2e-4,
                haline_contraction = 8e-4));
u_top_bc = FluxBoundaryCondition(wind_drag, discrete_form=true, parameters=(time_index, ρₐ, ρ₀, cᴰ, u10_gpu))
T_top_bc = FluxBoundaryCondition(surface_heat_flux, discrete_form=true, parameters=(time_index, ρ₀, cᴾ, latent_gpu, sensible_gpu))
u_bcs = FieldBoundaryConditions(top = u_top_bc );
T_bcs = FieldBoundaryConditions(top = T_top_bc);

# Define your model here
model = NonhydrostaticModel(; grid, buoyancy, 
                            advection = UpwindBiased(order=5),
                            timestepper = :RungeKutta3,
                            tracers = (:T, :S),
                            coriolis = FPlane(rotation_rate=7.2921e-5, latitude=72),
                            closure = AnisotropicMinimumDissipation(),
                            boundary_conditions = (u=u_bcs, T=T_bcs))

[glwagner]

Okay so what about these:

using Oceananigans
grid = RectilinearGrid(GPU(), size=(16, 16, 16), extent=(1, 1, 1))
c = CenterField(grid)
c_min = minimum(c) # test minimum on Field
c_min = minimum(c - 2)  # test minimum on AbstractOperation

your error comes from KernelFunctionOperation so we can try that too if the above works