-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Fix: Evernote .enex image import #7785
Fix: Evernote .enex image import #7785
Conversation
The latest updates on your projects. Learn more about Vercel for Git ↗︎
|
TODO: Import 390M .enex on HTML wiki is smooth, but on NodeJS it quickly become 100% usage on server side, I need to figure out what is consuming CPU later. If open in chrome, it will cause chrome 100%, so it is actually a frontend issue, maybe the import table is too long? But this only happen on my wiki, can't reproduce on Official wiki. But this PR is OK to merge. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Hi @linonetwo this looks good, apart from one minor issue. The use of a random number in the title seems clumsy. I'm not sure what options there might be, but a simple one would be if we could use a meaningful string (perhaps the truncated and possibly slugified text field), with a hash of the text field as a disambiguator.
I forgot I can use the hash provided by Evernote. |
It's probably a Node.js file IO problem that causes problems. I did create 20000+ tiddlers from the debug console with a node-server active and it caused problems after about 10-20 seconds on the filesystem side. The tiddlers itself where created immediately. May be there is a problem in the node.js server backend if many files are written at the same time. may be memory leak or something similar. |
I don't have more things to add to this PR. I've done using it importing all my Evernote content. Now I have 26k tiddlers, with size 2GB+ (including .git backup, hope tiddlywiki nodejs won't look into .git ...) |
No, I find it's smooth with or without It is just on my wiki. Maybe is cause by some plugin, but I don't have time to bi-sect it by hand. ENEX import can be a good reproducible example for this performance issue. I will look into it one day maybe. |
Thanks @linonetwo |
fixes #3176
Tested with 200M+ .enex file full of website clipped by Evernote WebClipper...All images in website are preserved.
A .enex that has wikihow clipped for test:
wikihow.enex.zip
Only thing to note is that you need to use vzhd1701/evernote-backup#54 to export .enex file, otherwise .enex that resources don't have hash can't be processed properly.