train problem

[lckdouble]

No description provided.

[lckdouble]

[=====================================>................................................] Step: 19ms | Tot: 10m10s���������������������������������������� 4374/10000
| WARNING: ran out of memory
[=====================================>................................................] Step: 26ms | Tot: 10m10s���������������������������������������� 4375/10000
| WARNING: ran out of memory
[=====================================>................................................] Step: 28ms | Tot: 10m10s���������������������������������������� 4376/10000
| WARNING: ran out of memory
[=====================================>................................................] Step: 31ms | Tot: 10m10s���������������������������������������� 4377/10000
| WARNING: ran out of memory
[=====================================>................................................] Step: 28ms | Tot: 10m10s���������������������������������������� 4378/10000
| WARNING: ran out of memory
[=====================================>................................................] Step: 18ms | Tot: 10m10s���������������������������������������� 4379/10000
| WARNING: ran out of memory
[=====================================>................................................] Step: 16ms | Tot: 10m10s���������������������������������������� 4380/10000
/data1/shgpu/sh/new/ex_ab_summtxt/Global-Encoding-master/utils/misc_utils.py:13: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
return AttrDict(yaml.load(open(path, 'r')))
Decaying learning rate to 9.15527e-09
Traceback (most recent call last):
File "train.py", line 322, in
main()
File "train.py", line 316, in main
print_log("Best %s score: %.2f\n" % (metric, max(params[metric])))
ValueError: max() arg is an empty sequence

[lckdouble]

max()的问题我解决了一下,但是运行第一个epoch运行一段时间后，频繁出现run out of memory,最后报错，想问一下·有人遇到同样的情况吗

[lckdouble]

RuntimeError: cuDNN error: CUDNN_STATUS_EXECUTION_FAILED
这个是报的错误，开始我以为是pytorch，后来改成cuda 9.0 pytorch 1.1.0，还是会出现这种问题。
我用的数据集是nlpcc,原文是经过抽取的，摘要是真实摘要，分字后放入数据集

[lckdouble]

jian'xiao'le减小了batchsize，

[1397981186]

请问怎么解决max的问题的

[lckdouble]

改了部分代码，你可以参考一下abstract，至于抽取器无参考价值…… [email protected] From: qq1397981186 Date: 2020-02-28 14:26 To: lancopku/Global-Encoding CC: lckdouble; Author Subject: Re: [lancopku/Global-Encoding] train problem (#30) 请问怎么解决max的问题的 — You are receiving this because you authored the thread. Reply to this email directly, view it on GitHub, or unsubscribe.

[kentleung1992]

max()的问题我解决了一下,但是运行第一个epoch运行一段时间后，频繁出现run out of memory,最后报错，想问一下·有人遇到同样的情况吗

求问如何解决max()这个问题，谢谢

[ghost]

max()的问题我解决了一下,但是运行第一个epoch运行一段时间后，频繁出现run out of memory,最后报错，想问一下·有人遇到同样的情况吗

求问如何解决max()这个问题，谢谢

同问你解决了吗？谢谢