problem about StreamMapNet -Hivt trajectory prediction

[JT-Sun]

Hello, I have successfully trained and tested StreamMapNet, but when I [Merge Map and Trajectory Dataset], the code shows that the predicted_map has been successfully generated,

but during the trajectory prediction phase, it still shows that the predicted_map is missing.

The information shows that the error only occurred after processing 4 scenes. Do you have any suggestions for this?

[alfredgu001324]

Hi, may I ask what is the command you use to run the code? And can you maybe loop through the merging results to see if each scene actually contains the predicted_map key? Thank you so much!

I recall someone has encountered similar problems in other Issues. Maybe check them out as well?

[JT-Sun]

python adaptor_meanc_detac.py --version trainval --split train --map_model StreamMapNet --dataroot /home/xx/data/nuscenes/ --index_file /home/xx/data/adaptor_files/traj_scene_frame_full_train.pkl --map_file /home/xx/data/adaptor_files/mapping_results_train_streamnew1_1e-4.pickle --gt_map_file /home/xx/data/adaptor_files/gt_full_train.pickle --save_path /home/xx/data/trj_data/streamnew_1e-4
This is my command

I find that like scene-2254.pkl , which sample_token is '53f5977684e14cb0a28f383fee1dd433' do not contain predicted_map. There are some pkl have same problem

[JT-Sun]

And I also search this sample_token in my mapping_results_train_streamnew1_1e-4.pickle, it also doesn't contain this token. What's wrong with this problem, please help me! thanks!

[JT-Sun]

I confuse that whether i get the mapping_results_train.pickle from the following config:
data_root = '/home/xx/data/nuscenes/'
data_ann = '/home/xx/data/processed/stream/'
data_ann_new = '/home/xx/data/processed/stream_new/'
data = dict(
samples_per_gpu=batch_size,
workers_per_gpu=4,
train=dict(
type='NuscDataset',
data_root=data_root,
ann_file=data_ann_new + 'nuscenes_map_infos_train_newsplit.pkl',
meta=meta,
roi_size=roi_size,
cat2id=cat2id,
pipeline=train_pipeline,
seq_split_num=1,
),
val=dict(
type='NuscDataset',
data_root=data_root,
ann_file=data_ann + 'nuscenes_map_infos_val.pkl',
meta=meta,
roi_size=roi_size,
cat2id=cat2id,
pipeline=test_pipeline,
eval_config=eval_config,
test_mode=True,
seq_split_num=1,
),
test=dict(
type='NuscDataset',
data_root=data_root,
#ann_file=data_ann + 'nuscenes_map_infos_val.pkl',
ann_file=data_ann_new + 'nuscenes_map_infos_train_newsplit.pkl',
meta=meta,
roi_size=roi_size,
cat2id=cat2id,
pipeline=test_pipeline,
eval_config=eval_config,
test_mode=True,
seq_split_num=1,

[alfredgu001324]

Can you maybe try changing ann_file=data_ann_new + 'nuscenes_map_infos_train_newsplit.pkl', to ann_file=data_ann_new + 'nuscenes_map_infos_train.pkl',? I think the problem is that even though we are using the newsplit to train StreamMapNet, to ensure all the mapping models to produce the same mapping results data, we should still use the original train split. Trajdata is only compatible with the original train/val split of nuscenes.

My guess might be wrong (sorry, it's been a while) but can you give it a quick try?

[JT-Sun]

I have try this way, but the answer is also sample_token is not '53f5977684e14cb0a28f383fee1dd433' contained in new mapping_results_train.pickle.

[alfredgu001324]

So you are using the annotation files in the original split (data_ann + 'nuscenes_map_infos_train.pkl') but that sample token is still not present?

[JT-Sun]

I wanted to make sure you said retrain, then train=dict(
type='NuscDataset',
data_root=data_root,
ann_file= data_ann + 'nuscenes_map_infos_train.pkl'
meta=meta,
roi_size=roi_size,
cat2id=cat2id,
pipeline=train_pipeline,
seq_split_num=1,
),
Then test=dict(
type='NuscDataset',
data_root=data_root,
#ann_file=data_ann + 'nuscenes_map_infos_val.pkl',
ann_file=data_ann + 'nuscenes_map_infos_train.pkl'
meta=meta,
roi_size=roi_size,
cat2id=cat2id,
pipeline=test_pipeline,
eval_config=eval_config,
test_mode=True,
seq_split_num=1,

Or am I still trained ann_file=data_ann_new + 'nuscenes_map_infos_train_newsplit.pkl', Just change test=dict(ann_file=data_ann_new + 'nuscenes_map_infos_train_newsplit.pkl' instead of ann_file=data_ann_new + 'nuscenes_map_infos_train.pkl',

I tried the second method，not re-training

[JT-Sun]

export PYTHONPATH="${PYTHONPATH}:/MapUncertaintyPrediction/StreamMapNet_modified"

python tools/train.py
plugin/configs/nusc_newsplit_480_60x30_24e.py
--deterministic
--no-validate
because I use the command as you list，so I just do the test about change the test——config dict

[alfredgu001324]

Or am I still trained ann_file=data_ann_new + 'nuscenes_map_infos_train_newsplit.pkl', Just change test=dict(ann_file=data_ann_new + 'nuscenes_map_infos_train_newsplit.pkl' instead of ann_file=data_ann_new + 'nuscenes_map_infos_train.pkl',

Nuscenes has an original train/val/test split. In StreamMapNet, they proposed using a new train/val/test split to avoid overlap between different splits, thus preventing overfitting. In our case, we need to first train our model using stream_new/nuscenes_infos_temporal_train_newsplit.pkl, then to generate the dataset that downstream prediction models use, we need to evaluate the trained model using stream/nuscenes_infos_temporal_train.pkl. This is to ensure that the mapping scenarios produced by StreamMapNet/MapTR are the same.

If you have already done this, I would suggest checking

1. First, ensure that StreamMapNet does produce the desired mapping data for all the scenarios (Basically loop over the pkl files to check each scenario contains a vector map).
2. If you have successfully trained MapTR, do a cross-check between MapTR and StreamMapNet to see if they have the same available scenarios after evaluating on the train split.
3. The last resort might be that inserting breakpoints in the merging process to see when 'predicted_map' is empty.
4. If it still does not work I can find my file (not sure if it still exists in my workstation or not lol) and share with you for cross-validation.

[JT-Sun]

Thank you for your patience!
python tools/nuscenes_converter.py --data-root ../nuscenes --newsplit --dest_path ../processed/stream_new
python tools/nuscenes_converter.py --data-root ../nuscenes --dest_path ../processed/stream

MapUncertaintyPrediction
├── nuscenes/
├── processed/
│ ├── maptr/
│ ├── maptrv2/
│ ├── stream/
| | ├── nuscenes_infos_temporal_train.pkl
| | ├── nuscenes_infos_temporal_val.pkl
│ ├── stream_new/
| | ├── nuscenes_infos_temporal_train_newsplit.pkl
| | ├── nuscenes_infos_temporal_val_newsplit.pkl

The problem I found was that neither nuscenes_infos_temporal_train.pkl nor nuscenes_infos_temporal_train_newsplit.pkl generated during the data processing phase contained this Token '53 f5977684e14cb0a28f383fee1dd433, compared with the series Mpatr nuscenes_map_infos_temporal_train.pkl, streamMapNet hundreds of token less content.

[alfredgu001324]

Sorry for the late response... was attending ECCV last week. You are actually right, now that you mentioned I finally remember (sorry it's been a long time). Yes StreamMapNet's train split somehow contains fewer scenarios than MapTR series. Originally I wanted to reach out their authors and check but forgot it afterwards.

As a reference, StreamMapNet's training set will contain 15069 scenarios, MapTR series would contain 15113 scenarios. So indeed there are some scenarios missing in StreamMapNet. Unfortunately I think you need to reach out their authors and see what they have done in their data preprocessing step. Sorry about that!

[JT-Sun]

Thank you very much for your reply. Since StreamMapNet's training set contains fewer scenes, I would like to know what was done when your method trained HiVT and denseTNT, because if I continue to follow the code you put in, after merging Map estimation, The trajectory prediction phase reports missing predicted map errors.

[alfredgu001324]

Uhmmm I think I did not do anything special during the merging phase. I remember there are some try-except statements in the code that should catch the missing sample tokens?

[alfredgu001324]

Can you please check how many scenarios are generated with StreamMapNet and MapTR after the merging process? And then maybe loop through them to see if the predicted_map is present?

[JT-Sun]

Uhmmm I think I did not do anything special during the merging phase. I remember there are some try-except statements in the code that should catch the missing sample tokens?

Sorry for the late reply. Yes, there is no problem in the merge stage, but StreamMapNet lacks some scenes, so the predicted map for this part of scenes is also lacking. Errors will be reported when predicting HiVT and DenseTNT trajectory. it still shows that the predicted_map is missing.

[alfredgu001324]

Can you check your merging results in the following: "StreamMapNet's training set will contain 15069 scenarios, MapTR series would contain 15113 scenarios"?

Technically this line ensures that the predicted map is present according to the sample token of the scene.

[JT-Sun]

emm, I find that the after adaptor.py both StreamMapNet and MapTR series train/data will up to contain scene-15190.pkl

[alfredgu001324]

Interesting, let me see if I can reproduce this. For now, can you just add a check during the adaptor merging process? If 'predicted_map' is empty, just skip to the next one?