Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update magda dev crawl sources #2544

Merged
merged 2 commits into from
Oct 21, 2019
Merged

Conversation

AlexGilleran
Copy link
Contributor

What this PR does

The dev.magda.io crawl sources have gotten a bit out of date with data.gov.au, so some data.gov.au bugs can't be replicated on it. This just takes recent changes to the data.gov.au crawl list and ports them across.

@AlexGilleran AlexGilleran force-pushed the update-magda-dev-crawl-sources branch from 348894e to 08c1b52 Compare October 16, 2019 06:21
@t83714 t83714 self-assigned this Oct 16, 2019
deploy/helm/magda-dev.yml Show resolved Hide resolved
@t83714
Copy link
Contributor

t83714 commented Oct 17, 2019

Had another try in the cluster and seems BOM connector needs a new crawling url (getting 503 access denied error):

Failed to GET http://www.bom.gov.au/geonetwork/srv/eng/csw?service=CSW&version=2.0.2&request=GetRecords&constraintLanguage=FILTER&constraint_language_version=1.1.0&resultType=results&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&startPosition=1&maxRecords=100. 1 retries left. Exception:
Error: Invalid Server Response or Empty result returned!
Requesting http://www.bom.gov.au/geonetwork/srv/eng/csw?service=CSW&version=2.0.2&request=GetRecords&constraintLanguage=FILTER&constraint_language_version=1.1.0&resultType=results&elementsetname=full&outputschema=http%3A%2F%2Fwww.isotc211.org%2F2005%2Fgmd&typeNames=gmd%3AMD_Metadata&startPosition=1&maxRecords=100
[xmldom warning]	unclosed xml attribute 
@#[line:6,col:144]
Error: Invalid Server Response or Empty result returned!
    at Request.request_1.default [as _callback] (/usr/src/app/component/dist/Csw.js:207:31)
    at Request.self.callback (/usr/src/app/node_modules/request/request.js:185:22)
    at emitTwo (events.js:126:13)
    at Request.emit (events.js:214:7)
    at Request.<anonymous> (/usr/src/app/node_modules/request/request.js:1161:10)
    at emitOne (events.js:116:13)
    at Request.emit (events.js:211:7)
    at IncomingMessage.<anonymous> (/usr/src/app/node_modules/request/request.js:1083:12)
    at Object.onceWrapper (events.js:313:30)
    at emitNone (events.js:111:20)

But I guess it probably should be handled by a seperate issue.
Do we want to remove it or leave it in for now? @AlexGilleran

@AlexGilleran
Copy link
Contributor Author

Should probably raise an issue on magda-config-data.gov.au for it :)

@t83714
Copy link
Contributor

t83714 commented Oct 21, 2019

Created a separate issue for BOM connector here:
magda-io/magda-config-data.gov.au#30

@t83714 t83714 merged commit 952119f into master Oct 21, 2019
@t83714 t83714 deleted the update-magda-dev-crawl-sources branch October 21, 2019 03:04
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants