Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Fix: Evernote .enex image import #7785

Merged
merged 11 commits into from
Oct 15, 2023

Conversation

linonetwo
Copy link
Contributor

@linonetwo linonetwo commented Oct 14, 2023

fixes #3176

Tested with 200M+ .enex file full of website clipped by Evernote WebClipper...All images in website are preserved.

截屏2023-10-15 00 09 12

A .enex that has wikihow clipped for test:

wikihow.enex.zip

Only thing to note is that you need to use vzhd1701/evernote-backup#54 to export .enex file, otherwise .enex that resources don't have hash can't be processed properly.

@vercel
Copy link

vercel bot commented Oct 14, 2023

The latest updates on your projects. Learn more about Vercel for Git ↗︎

Name Status Preview Updated (UTC)
tiddlywiki5 ✅ Ready (Inspect) Visit Preview Oct 15, 2023 4:51am

@linonetwo linonetwo marked this pull request as ready for review October 14, 2023 16:06
@linonetwo
Copy link
Contributor Author

linonetwo commented Oct 14, 2023

TODO: Import 390M .enex on HTML wiki is smooth, but on NodeJS it quickly become 100% usage on server side, I need to figure out what is consuming CPU later.

If open in chrome, it will cause chrome 100%, so it is actually a frontend issue, maybe the import table is too long? But this only happen on my wiki, can't reproduce on Official wiki.

But this PR is OK to merge.

Copy link
Member

@Jermolene Jermolene left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Hi @linonetwo this looks good, apart from one minor issue. The use of a random number in the title seems clumsy. I'm not sure what options there might be, but a simple one would be if we could use a meaningful string (perhaps the truncated and possibly slugified text field), with a hash of the text field as a disambiguator.

@linonetwo
Copy link
Contributor Author

random number in the title

I forgot I can use the hash provided by Evernote.

@pmario
Copy link
Member

pmario commented Oct 15, 2023

but on NodeJS it quickly become 100% usage on server side, I need to figure out what is consuming CPU later.

It's probably a Node.js file IO problem that causes problems. I did create 20000+ tiddlers from the debug console with a node-server active and it caused problems after about 10-20 seconds on the filesystem side.

The tiddlers itself where created immediately. May be there is a problem in the node.js server backend if many files are written at the same time. may be memory leak or something similar.

@linonetwo
Copy link
Contributor Author

linonetwo commented Oct 15, 2023

I don't have more things to add to this PR. I've done using it importing all my Evernote content. Now I have 26k tiddlers, with size 2GB+ (including .git backup, hope tiddlywiki nodejs won't look into .git ...)

@linonetwo
Copy link
Contributor Author

linonetwo commented Oct 15, 2023

probably a Node.js file IO problem

No, I find it's smooth with or without "tiddlywiki/filesystem", and "tiddlywiki/tiddlyweb",. On Official wiki.

It is just on my wiki. Maybe is cause by some plugin, but I don't have time to bi-sect it by hand.

ENEX import can be a good reproducible example for this performance issue. I will look into it one day maybe.

@Jermolene
Copy link
Member

Thanks @linonetwo

@Jermolene Jermolene merged commit d3f5695 into TiddlyWiki:master Oct 15, 2023
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

evernote plugin does not import resources
3 participants