Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Using new dataset #16

Open
lovelyyunamars opened this issue Mar 19, 2021 · 9 comments
Open

Using new dataset #16

lovelyyunamars opened this issue Mar 19, 2021 · 9 comments

Comments

@lovelyyunamars
Copy link

Dear Dr. Nguyen

Thank you for your awesome work!

Right now, I can get results based on the data you provided. However, when I use my own dataset, there are some problems.

First of all, I listed all the setups I made:
Lat_min=50.6, Lat_max=54.6, Lon_min=-180.0, Lon_max=-174.0
Lat_bins=400, Lon_bins=600
the data I used is from 2017 Jan 1st to Jan 31th, 10 days for training, 10 days for valid, 11 days for tset.

Then I run cvs2pkl, dataset_preprocessing, calculate_AIS_mean, I got the data file similar with the data you provided, but I did not use the ''Loading coastline polygon'' function in the dataset_preprocessing.py

then I start to train the model, it works, but I got two files under the chkpt path with different data_dim, one is 602, one is 1102, I believe 1102 should me mine dataset.

then I run save_logprob, it did not creat errors too

However when I run local_logprob, I can not get any result, even that data/ct_2017010203_10_20/local_logprob-ct_2017010203_10_20_train.pkl-ct_2017010203_10_20_valid.pkl-100-missing_data-False-step-80002 and other results.

So I want to know, why this error happened? is that because I used not enough data?

I will be so grateful if you can reply!

Regards

Yu
@mariamartinmarin
Copy link

Incredible work, Dr. Nguyen,

I am also having some problems. I don't know if it could be because of the format of my dataset.

Lovelyyunamars, could you tell me the original format of your dataset? I have downloaded several datasets from MarineCadastre, but when I pass it csv2pkl, it does not work well, because some fields are missing that are not in the csv file provided by MarineCadastre.

Thank you very much for everything!

@lovelyyunamars
Copy link
Author

Incredible work, Dr. Nguyen,

I am also having some problems. I don't know if it could be because of the format of my dataset.

Lovelyyunamars, could you tell me the original format of your dataset? I have downloaded several datasets from MarineCadastre, but when I pass it csv2pkl, it does not work well, because some fields are missing that are not in the csv file provided by MarineCadastre.

Thank you very much for everything!
Hi, Maria

I think you should adjust the code line 170-176 in csv2pkl.py according to your original csv file.

@dnguyengithub
Copy link
Collaborator

Hello,

Sorry for the late reply, I'm currently not working on this project.

@lovelyyunamars there is a bug in this version, the sizes of the 4-hot vectors are hard-coded in datasets. You should manually modify those values. I'm sorry for this bug. I'll update a newer version when I have time.

Can you successfully run the code on the provided dataset?

@mariamartinmarin You dont need all the attributes in the *.csv files (we use HEADING, ROT, NAV, etc. for other applications). GeoTrackNet requires only LAT, LON, COG, SOG, MMSI, and TIMESTAMP.
If you want to apply GeoTrackNet on your own datasets, you can re-write the preprocessing part (highly recommend, the preprocessing code of this version is very bad).

@lovelyyunamars
Copy link
Author

lovelyyunamars commented Apr 2, 2021 via email

@mariamartinmarin
Copy link

@lovelyyunamars Thank you for your answer.

How could you do the training? When I try to do the training, I get the following error:

An error occurred while training the embedding layer：ValueError：Dimensions must be equal,but are 702 and 602 for 'sub' with input shapes:[?,??,702],[1,1,602].

It seems that I have to make some changes in the dataset.py file, but I don't really know what to do. Can you help me, please?

Thank you very much!

@lovelyyunamars
Copy link
Author

lovelyyunamars commented Apr 29, 2021 via email

@mariamartinmarin
Copy link

Thank you so much @lovelyyunamars !!!

I was able to fix it by modifying some files!
One question I have now (maybe I haven't done enough research) is how to apply the generated model to perform detections. could you help me?

Thank you very much for your help.

@hariiiseldon
Copy link

@mariamartinmarin Hello，I have the same error when I run the GeoTrackNet.py:
An error occurred while training the embedding layer：ValueError：Dimensions must be equal,but are 702 and 602 for 'sub' with input shapes:[?,??,702],[1,1,602].
I would appreciate it if you could tell me how to modify the files.
Thank you so much for your help!

@yche-sflscientific
Copy link

@hariiiseldon hope it's still relevant. I had the same issue and succeeded getting pass the mismatched dimension error.
My understanding is that the 602 is from the where you set using the calculate_AIS_mean.py, and the 702 is from where you input when calling geotracknet.py, specifically calculated from lat_min/lat_max and lon_min/lon_max, divide the range by 0.01 deg, should be your lat_bins and lon_bins set in the dataset.py and calculate_AIS_mean.py. Good luck!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@mariamartinmarin @dnguyengithub @hariiiseldon @lovelyyunamars @yche-sflscientific