"AssertionError: datasets should not be an empty iterable" when training FeatureExtractor

[vbalog]

Hello,

I think this is a different issue than in thread #131 but please correct me if I am wrong!

I am using the beta upgraded version of DeepEthogram. I was able to train the FlowGenerator successfully, but attempting to infer or train the FeatureExtractor isn't going well.

When I attempt to train my FeatureExtractor using the GUI, I get the following error and output:

[2023-03-23 16:19:48,320] INFO [deepethogram.gui.main.get_selected_models:1136] {'flow_generator': {'no pretrained weights': None, '200221_115158_TinyMotionNet': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt', '230322_140309_flow_generator_train': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_140309_flow_generator_train\lightning_checkpoints\epoch=0-step=999.ckpt', '230322_211503_flow_generator_train': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt'}, 'feature_extractor': {'no pretrained weights': None, '200415_125824_hidden_two_stream_kinetics_degf': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200415_125824_hidden_two_stream_kinetics_degf\checkpoint.pt'}, 'sequence': {'': None}}
{'flow_generator': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt', 'feature_extractor': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200415_125824_hidden_two_stream_kinetics_degf\checkpoint.pt', 'sequence': None}
[2023-03-23 16:19:48,322] INFO [deepethogram.gui.main.get_selected_models:1136] {'flow_generator': {'no pretrained weights': None, '200221_115158_TinyMotionNet': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt', '230322_140309_flow_generator_train': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_140309_flow_generator_train\lightning_checkpoints\epoch=0-step=999.ckpt', '230322_211503_flow_generator_train': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt'}, 'feature_extractor': {'no pretrained weights': None, '200415_125824_hidden_two_stream_kinetics_degf': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200415_125824_hidden_two_stream_kinetics_degf\checkpoint.pt'}, 'sequence': {'': None}}
[2023-03-23 16:19:48,323] INFO [deepethogram.gui.main.get_selected_models:1136] {'flow_generator': {'no pretrained weights': None, '200221_115158_TinyMotionNet': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200221_115158_TinyMotionNet\checkpoint.pt', '230322_140309_flow_generator_train': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_140309_flow_generator_train\lightning_checkpoints\epoch=0-step=999.ckpt', '230322_211503_flow_generator_train': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt'}, 'feature_extractor': {'no pretrained weights': None, '200415_125824_hidden_two_stream_kinetics_degf': 'C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200415_125824_hidden_two_stream_kinetics_degf\checkpoint.pt'}, 'sequence': {'': None}}
[2023-03-23 16:19:48,324] INFO [deepethogram.gui.main.featureextractor_train:385] feature extractor train called with args: ['python', '-m', 'deepethogram.feature_extractor.train', 'project.path=C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram', 'feature_extractor.weights=C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200415_125824_hidden_two_stream_kinetics_degf\checkpoint.pt', 'flow_generator.weights=C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt']
[2023-03-23 16:19:51,632] INFO [main.feature_extractor_train:66] args: C:\Users\Bizon.conda\envs\deg2\lib\site-packages\deepethogram\feature_extractor\train.py project.path=C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram feature_extractor.weights=C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200415_125824_hidden_two_stream_kinetics_degf\checkpoint.pt flow_generator.weights=C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt
[2023-03-23 16:19:51,632] INFO [main.feature_extractor_train:71] configuration used ~~~~~
[2023-03-23 16:19:51,653] INFO [main.feature_extractor_train:72] split:
reload: true
file: null
train_val_test:

• 0.8
• 0.2
• 0.0
  compute:
  fp16: false
  num_workers: 8
  batch_size: 32
  min_batch_size: 8
  max_batch_size: 512
  distributed: false
  gpu_id: 0
  dali: false
  metrics_workers: 0
  reload:
  overwrite_cfg: false
  latest: false
  notes: null
  log:
  level: info
  augs:
  brightness: 0.25
  contrast: 0.1
  hue: 0.1
  saturation: 0.1
  color_p: 0.5
  grayscale: 0.5
  crop_size: null
  resize:
• 224
• 224
  dali: false
  random_resize: false
  pad: null
  LR: 0.5
  UD: 0.0
  degrees: 10
  normalization:
  'N': 119808000
  mean:
  • 0.3220345943051363
  • 0.3220345943051363
  • 0.3220345943051363
    std:
  • 0.16833273961261574
  • 0.16833273961261574
  • 0.16833273961261574
    train:
    lr: 0.0001
    scheduler: plateau
    num_epochs: 20
    steps_per_epoch:
    train: 1000
    val: 1000
    test: null
    min_lr: 5.0e-07
    stopping_type: learning_rate
    milestones:
• 50
• 100
• 150
• 200
• 250
• 300
  weight_loss: true
  patience: 3
  early_stopping_begins: 0
  viz_metrics: true
  viz_examples: 10
  reduction_factor: 0.1
  loss_weight_exp: 1.0
  loss_gamma: 1.0
  label_smoothing: 0.05
  oversampling_exp: 0.0
  regularization:
  style: l2_sp
  alpha: 1.0e-05
  beta: 0.001
  flow_generator:
  type: flow_generator
  flow_loss: MotionNet
  flow_max: 10
  input_images: 11
  flow_sparsity: false
  smooth_weight_multiplier: 1.0
  sparsity_weight: 0.0
  loss: MotionNet
  max: 5
  n_rgb: 11
  arch: TinyMotionNet
  weights: C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt
  feature_extractor:
  arch: resnet18
  fusion: average
  sampler: null
  final_bn: false
  sampling_ratio: null
  final_activation: sigmoid
  dropout_p: 0.25
  n_flows: 10
  n_rgb: 1
  curriculum: false
  inputs: both
  weights: C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models\200415_125824_hidden_two_stream_kinetics_degf\checkpoint.pt
  cmap: deepethogram
  control_arrow_jump: 31
  label_view_width: 31
  postprocessor:
  min_bout_length: 1
  type: min_bout_per_behavior
  prediction_opacity: 0.2
  project:
  class_names:
• background
• freezing
  config_file: project_config.yaml
  data_path: C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\DATA
  labeler: null
  model_path: C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models
  name: CFC-Cohort1
  path: C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram
  pretrained_path: C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\pretrained_models
  run:
  type: train
  model: feature_extractor
  dir: C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230323_161951_feature_extractor_train
  sequence:
  filter_length: 15
  unlabeled_alpha: 0.1
  vertical_arrow_jump: 3

[2023-03-23 16:19:52,098] INFO [deepethogram.projects.get_weightfile_from_cfg:1079] loading specified weights: C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt
[2023-03-23 16:19:52,098] INFO [main.feature_extractor_train:79] loading flow generator from file C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt
[2023-03-23 16:19:52,099] INFO [deepethogram.utils.load_state:341] loading from checkpoint file C:\Users\Bizon\Desktop\Valerie DeepEthogram Video CFC Testing\VB_DE_Try3\CFC-Cohort1_deepethogram\models\230322_211503_flow_generator_train\lightning_checkpoints\epoch=4-step=4999.ckpt...
[2023-03-23 16:19:52,181] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_150958 20230221
[2023-03-23 16:19:52,181] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_163107 2
[2023-03-23 16:19:52,181] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_164648 3
[2023-03-23 16:19:52,182] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_165946 4
[2023-03-23 16:19:52,182] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_171150
[2023-03-23 16:19:52,182] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_172800 6
[2023-03-23 16:19:52,183] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_174912 7
[2023-03-23 16:19:52,183] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_180715 8
[2023-03-23 16:19:52,184] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_182416
[2023-03-23 16:19:52,184] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022223_184414 10
[2023-03-23 16:19:52,184] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022323_155654 6
[2023-03-23 16:19:52,184] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022323_160830 7
[2023-03-23 16:19:52,185] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022323_161728 8
[2023-03-23 16:19:52,185] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022323_162420 9
[2023-03-23 16:19:52,185] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022323_163221 10
[2023-03-23 16:19:52,185] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_145231 1
[2023-03-23 16:19:52,186] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_151352 2
[2023-03-23 16:19:52,186] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_152053 3
[2023-03-23 16:19:52,186] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_153744 4
[2023-03-23 16:19:52,187] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_154512 5
[2023-03-23 16:19:52,187] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_155404 6
[2023-03-23 16:19:52,187] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_160400 7
[2023-03-23 16:19:52,187] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_161312 8
[2023-03-23 16:19:52,188] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_162148 9
[2023-03-23 16:19:52,188] WARNING [deepethogram.projects.filter_records_for_filetypes:663] No label file found in record: VF022423_163036 10
Traceback (most recent call last):
File "C:\Users\Bizon.conda\envs\deg2\lib\runpy.py", line 193, in _run_module_as_main
"main", mod_spec)
File "C:\Users\Bizon.conda\envs\deg2\lib\runpy.py", line 85, in _run_code
exec(code, run_globals)
File "C:\Users\Bizon.conda\envs\deg2\lib\site-packages\deepethogram\feature_extractor\train.py", line 646, in
feature_extractor_train(cfg)
File "C:\Users\Bizon.conda\envs\deg2\lib\site-packages\deepethogram\feature_extractor\train.py", line 85, in feature_extractor_train
input_images=cfg.feature_extractor.n_flows + 1)
File "C:\Users\Bizon.conda\envs\deg2\lib\site-packages\deepethogram\data\datasets.py", line 1186, in get_datasets_from_cfg
mean_by_channels=cfg.augs.normalization.mean)
File "C:\Users\Bizon.conda\envs\deg2\lib\site-packages\deepethogram\data\datasets.py", line 934, in get_video_datasets
mean_by_channels=mean_by_channels)
File "C:\Users\Bizon.conda\envs\deg2\lib\site-packages\deepethogram\data\datasets.py", line 403, in init
self.dataset = data.ConcatDataset(datasets)
File "C:\Users\Bizon.conda\envs\deg2\lib\site-packages\torch\utils\data\dataset.py", line 290, in init
assert len(self.datasets) > 0, 'datasets should not be an empty iterable' # type: ignore[arg-type]
AssertionError: datasets should not be an empty iterable
[2023-03-23 16:19:53,361] INFO [deepethogram.gui.main.featureextractor_train:395] Training finished. If you see error messages above, training did not complete successfully.
[2023-03-23 16:19:53,362] INFO [deepethogram.gui.main.featureextractor_train:400] ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
[2023-03-23 16:19:53,445] INFO [deepethogram.gui.main.project_loaded_buttons:175] Number finalized labels: 5

Any help/workaround for this would be greatly appreciated. Thanks!

[vsomesula]

Hello,

I ran into the same error and I resolved it by making a restarting, relaunching my environment, opening DeepEthogram fresh, and starting new project. I also made sure I did not have a space in the project title when I made my project. I'm not sure which of these steps caused the fix but I hope this helps.