Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

How long should it take to preprocess for global direction? #95

Open
Mehrab-Tanjim opened this issue May 16, 2022 · 11 comments
Open

How long should it take to preprocess for global direction? #95

Mehrab-Tanjim opened this issue May 16, 2022 · 11 comments

Comments

@Mehrab-Tanjim
Copy link

I think in the paper it is mentioned it takes ~4 hours to preprocess the global direction for FFHQ. However, when I tried to proprocess for the same FFHQ model, it is taking ~24 hours (using V100 GPU). I did not change any settings. Is it expected? Or am I missing something?
@betterze
Copy link
Collaborator

Would you mind checking you gpu usage when you run the 'SingleChannel.py'?

@Mehrab-Tanjim
Copy link
Author

Sure! Right now I am using T4, and I am seeing a constant 75% usage (screenshot below).
image

@betterze
Copy link
Collaborator

you are using TensorFlow, version 1.14 or 1.15?

@Mehrab-Tanjim
Copy link
Author

I should add that I am using google colab to train. It is using Tensorflow 1.15.2

@betterze
Copy link
Collaborator

betterze commented May 16, 2022 •
edited
Loading

To be honest, this never happens to me. I use a gtx 1080 gpu on my local computer. It takes around 4 hours. Your gpu is better than mine, it should be faster.

Could you provide the log when you run the codes?

dataset_name='ffhq' 
# input prepare data 
python GetCode.py --dataset_name $dataset_name --code_type 'w'
python GetCode.py --dataset_name $dataset_name --code_type 's'
python GetCode.py --dataset_name $dataset_name --code_type 's_mean_std'

# preprocess (this may take a few hours). 
python SingleChannel.py --dataset_name $dataset_name

The 'SingleChannel.py' will take 4 hours.

python GetGUIData.py --dataset_name $dataset_name
python PlayInteractively.py --dataset_name $dataset_name

'PlayInteractively.py' is for the GUI, I am not sure if the GUI could be run in Colab. Which dataset or SyleGAN model you are using?

@Mehrab-Tanjim
Copy link
Author

Yes, I was surprised too when it is taking more than 4 hours. I am running exactly the same set of codes, just in Google Colab:

dataset_name='ffhq' 
# input prepare data 
python GetCode.py --dataset_name $dataset_name --code_type 'w'
python GetCode.py --dataset_name $dataset_name --code_type 's'
python GetCode.py --dataset_name $dataset_name --code_type 's_mean_std'

# preprocess (this may take a few hours). 
python SingleChannel.py --dataset_name $dataset_name

The StyleGAN model should be whatever is the default one (i.e. StyleGAN2)

@Mehrab-Tanjim
Copy link
Author

The log is long. But here it is:

  100%|███████████████████████████████████████| 338M/338M [00:04<00:00, 79.4MiB/s]
  2022-05-16 16:10:17.065738: I tensorflow/core/platform/profile_utils/cpu_utils.cc:94] CPU Frequency: 2299995000 Hz
  2022-05-16 16:10:17.065961: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x78a61c0 initialized for platform Host (this does not guarantee that XLA will be used). Devices:
  2022-05-16 16:10:17.065998: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Host, Default Version
  2022-05-16 16:10:17.067149: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcuda.so.1
  2022-05-16 16:10:17.067545: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.068264: I tensorflow/compiler/xla/service/service.cc:168] XLA service 0x78a6000 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
  2022-05-16 16:10:17.068297: I tensorflow/compiler/xla/service/service.cc:176]   StreamExecutor device (0): Tesla T4, Compute Capability 7.5
  2022-05-16 16:10:17.068470: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.069075: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
  name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
  pciBusID: 0000:00:04.0
  2022-05-16 16:10:17.071739: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
  2022-05-16 16:10:17.083818: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
  2022-05-16 16:10:17.087048: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
  2022-05-16 16:10:17.092396: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
  2022-05-16 16:10:17.104587: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
  2022-05-16 16:10:17.108108: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
  2022-05-16 16:10:17.132946: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
  2022-05-16 16:10:17.133102: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.133790: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.134307: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
  2022-05-16 16:10:17.134394: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
  2022-05-16 16:10:17.135480: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
  2022-05-16 16:10:17.135516: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
  2022-05-16 16:10:17.135534: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
  2022-05-16 16:10:17.135644: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.136162: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.136676: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/job:localhost/replica:0/task:0/device:GPU:0 with 13016 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
  Setting up TensorFlow plugin "fused_bias_act.cu": 2022-05-16 16:10:17.683423: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.684362: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
  name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
  pciBusID: 0000:00:04.0
  2022-05-16 16:10:17.684524: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
  2022-05-16 16:10:17.684588: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
  2022-05-16 16:10:17.684619: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
  2022-05-16 16:10:17.684648: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
  2022-05-16 16:10:17.684670: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
  2022-05-16 16:10:17.684694: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
  2022-05-16 16:10:17.684719: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
  2022-05-16 16:10:17.684806: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.685609: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.686164: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
  2022-05-16 16:10:17.686208: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
  2022-05-16 16:10:17.686233: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
  2022-05-16 16:10:17.686244: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
  2022-05-16 16:10:17.686328: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.686958: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.687518: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/device:GPU:0 with 13016 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
  Loading... Done.
  Setting up TensorFlow plugin "upfirdn_2d.cu": 2022-05-16 16:10:17.823879: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.824553: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1639] Found device 0 with properties: 
  name: Tesla T4 major: 7 minor: 5 memoryClockRate(GHz): 1.59
  pciBusID: 0000:00:04.0
  2022-05-16 16:10:17.824668: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudart.so.10.1
  2022-05-16 16:10:17.824700: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
  2022-05-16 16:10:17.824728: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcufft.so.10
  2022-05-16 16:10:17.824755: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcurand.so.10
  2022-05-16 16:10:17.824780: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusolver.so.10
  2022-05-16 16:10:17.824803: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcusparse.so.10
  2022-05-16 16:10:17.824826: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
  2022-05-16 16:10:17.824902: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.825477: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.825976: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1767] Adding visible gpu devices: 0
  2022-05-16 16:10:17.826028: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1180] Device interconnect StreamExecutor with strength 1 edge matrix:
  2022-05-16 16:10:17.826043: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1186]      0 
  2022-05-16 16:10:17.826054: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1199] 0:   N 
  2022-05-16 16:10:17.826135: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.826679: I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:983] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
  2022-05-16 16:10:17.827179: I tensorflow/core/common_runtime/gpu/gpu_device.cc:1325] Created TensorFlow device (/device:GPU:0 with 13016 MB memory) -> physical GPU (device: 0, name: Tesla T4, pci bus id: 0000:00:04.0, compute capability: 7.5)
  Loading... Done.
  
  Gs                              Params    OutputShape          WeightShape     
  ---                             ---       ---                  ---             
  latents_in                      -         (?, 512)             -               
  labels_in                       -         (?, 0)               -               
  lod                             -         ()                   -               
  dlatent_avg                     -         (512,)               -               
  G_mapping/Normalize             -         (?, 512)             -               
  G_mapping/Dense0                262656    (?, 512)             (512, 512)      
  G_mapping/Dense1                262656    (?, 512)             (512, 512)      
  G_mapping/Dense2                262656    (?, 512)             (512, 512)      
  G_mapping/Dense3                262656    (?, 512)             (512, 512)      
  G_mapping/Dense4                262656    (?, 512)             (512, 512)      
  G_mapping/Dense5                262656    (?, 512)             (512, 512)      
  G_mapping/Dense6                262656    (?, 512)             (512, 512)      
  G_mapping/Dense7                262656    (?, 512)             (512, 512)      
  G_mapping/Broadcast             -         (?, 18, 512)         -               
  Truncation/Lerp                 -         (?, 18, 512)         -               
  G_synthesis/4x4/Const           8192      (?, 512, 4, 4)       (1, 512, 4, 4)  
  G_synthesis/4x4/Conv            2622465   (?, 512, 4, 4)       (3, 3, 512, 512)
  G_synthesis/4x4/ToRGB           264195    (?, 3, 4, 4)         (1, 1, 512, 3)  
  G_synthesis/8x8/Conv0_up        2622465   (?, 512, 8, 8)       (3, 3, 512, 512)
  G_synthesis/8x8/Conv1           2622465   (?, 512, 8, 8)       (3, 3, 512, 512)
  G_synthesis/8x8/Upsample        -         (?, 3, 8, 8)         -               
  G_synthesis/8x8/ToRGB           264195    (?, 3, 8, 8)         (1, 1, 512, 3)  
  G_synthesis/16x16/Conv0_up      2622465   (?, 512, 16, 16)     (3, 3, 512, 512)
  G_synthesis/16x16/Conv1         2622465   (?, 512, 16, 16)     (3, 3, 512, 512)
  G_synthesis/16x16/Upsample      -         (?, 3, 16, 16)       -               
  G_synthesis/16x16/ToRGB         264195    (?, 3, 16, 16)       (1, 1, 512, 3)  
  G_synthesis/32x32/Conv0_up      2622465   (?, 512, 32, 32)     (3, 3, 512, 512)
  G_synthesis/32x32/Conv1         2622465   (?, 512, 32, 32)     (3, 3, 512, 512)
  G_synthesis/32x32/Upsample      -         (?, 3, 32, 32)       -               
  G_synthesis/32x32/ToRGB         264195    (?, 3, 32, 32)       (1, 1, 512, 3)  
  G_synthesis/64x64/Conv0_up      2622465   (?, 512, 64, 64)     (3, 3, 512, 512)
  G_synthesis/64x64/Conv1         2622465   (?, 512, 64, 64)     (3, 3, 512, 512)
  G_synthesis/64x64/Upsample      -         (?, 3, 64, 64)       -               
  G_synthesis/64x64/ToRGB         264195    (?, 3, 64, 64)       (1, 1, 512, 3)  
  G_synthesis/128x128/Conv0_up    1442561   (?, 256, 128, 128)   (3, 3, 512, 256)
  G_synthesis/128x128/Conv1       721409    (?, 256, 128, 128)   (3, 3, 256, 256)
  G_synthesis/128x128/Upsample    -         (?, 3, 128, 128)     -               
  G_synthesis/128x128/ToRGB       132099    (?, 3, 128, 128)     (1, 1, 256, 3)  
  G_synthesis/256x256/Conv0_up    426369    (?, 128, 256, 256)   (3, 3, 256, 128)
  G_synthesis/256x256/Conv1       213249    (?, 128, 256, 256)   (3, 3, 128, 128)
  G_synthesis/256x256/Upsample    -         (?, 3, 256, 256)     -               
  G_synthesis/256x256/ToRGB       66051     (?, 3, 256, 256)     (1, 1, 128, 3)  
  G_synthesis/512x512/Conv0_up    139457    (?, 64, 512, 512)    (3, 3, 128, 64) 
  G_synthesis/512x512/Conv1       69761     (?, 64, 512, 512)    (3, 3, 64, 64)  
  G_synthesis/512x512/Upsample    -         (?, 3, 512, 512)     -               
  G_synthesis/512x512/ToRGB       33027     (?, 3, 512, 512)     (1, 1, 64, 3)   
  G_synthesis/1024x1024/Conv0_up  51297     (?, 32, 1024, 1024)  (3, 3, 64, 32)  
  G_synthesis/1024x1024/Conv1     25665     (?, 32, 1024, 1024)  (3, 3, 32, 32)  
  G_synthesis/1024x1024/Upsample  -         (?, 3, 1024, 1024)   -               
  G_synthesis/1024x1024/ToRGB     16515     (?, 3, 1024, 1024)   (1, 1, 32, 3)   
  ---                             ---       ---                  ---             
  Total                           30370060                                       
  
  2022-05-16 16:10:21.765556: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcublas.so.10
  2022-05-16 16:10:22.071804: I tensorflow/stream_executor/platform/default/dso_loader.cc:44] Successfully opened dynamic library libcudnn.so.7
  ffhq
    0%

@betterze
Copy link
Collaborator

The log seems to be correct. Maybe this is related to colab. Maybe colab allocate the same gpu to multiple users? I am not an experter in colab.

@Mehrab-Tanjim
Copy link
Author

I recently tried to preprocess in my own pc which has 1080 gpu (tf 1.14). I am facing two problems... with the current settings of 100 images, it is facing out of memories when it tried to calculate features using clip. Second problem: when I don't calculate features and only generate images, it shows me per layer it will take 2 hours...and there are 11 layers...and so back to ~24 hours 😅

Just out of curiosity how many images and steps were you able to run for in 1080?

@Panghema
Copy link

Panghema commented Jun 7, 2022

actually, my 2080ti ran for about 24 hours
it seems to me that the CPU limits the processing speed. it takes up 100% at a time and 3% at a time

@asheroin asheroin mentioned this issue Sep 29, 2022
Open
@ShristiDasBiswas
Copy link

which code can i use to find the global directions on a custom dataset?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
4 participants
@Mehrab-Tanjim @betterze @Panghema @ShristiDasBiswas