Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Post processing of extraction result and adding new urls to the queue #4

Open
pbvahlst opened this issue Jan 30, 2020 · 0 comments
Open
Labels
enhancement New feature or request

Comments

@pbvahlst
Copy link
Contributor

pbvahlst commented Jan 30, 2020

Add optional afterExtractor: function(data, options) which can be used for modifying data after successful extraction.

Furthermore it can be used for adding new urls to the extraction queue.

Should return one of undefined, null, [], {} or an object which should be saved:

The options object has the structure
{urls: []} where you can new urls which should be added to the queue-

If an object is returned test if data lives up the requirements of saving the data.
If new urls add them to extraction queue. The extraction queue should probably be
a Set instead of an array to avoid duplicates

@pbvahlst pbvahlst added the enhancement New feature or request label Jan 30, 2020
@pbvahlst pbvahlst reopened this Jun 11, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
enhancement New feature or request
Projects
None yet
Development

No branches or pull requests

1 participant