Skip to content

Commit

Permalink
IOP Spider: improve and add tests
Browse files Browse the repository at this point in the history 
This adds test records from IOP and fixes some simple
issues with IOP spider, to make the tests pass.

Introduces a functional tests of the IOP spider.

Signed-off-by: Szymon Łopaciuk <[email protected]>
  • Loading branch information
@szymonlopaciuk
szymonlopaciuk committed Jan 12, 2018
1 parent 8514ac8 commit 2173f2c
Show file tree
Hide file tree
Showing 11 changed files with 407 additions and 8 deletions.
1 change: 1 addition & 0 deletions .travis.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -31,6 +31,7 @@ env:
- SUITE=functional_desy
- SUITE=functional_cds
- SUITE=functional_pos
- SUITE=functional_iop

matrix:
fast_finish: true
Expand Down
7 changes: 7 additions & 0 deletions docker-compose.test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -70,6 +70,13 @@ services:
http-server.local:
condition: service_healthy

functional_iop:
<<: *service_base
command: py.test -vv tests/functional/iop
depends_on:
scrapyd:
condition: service_healthy

unit:
<<: *service_base
command: bash -c "py.test tests/unit -vv && make -C docs clean && make -C docs html && python setup.py sdist && ls dist/*"
Expand Down
6 changes: 3 additions & 3 deletions hepcrawl/extractors/nlm.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -146,10 +146,10 @@ def get_page_numbers(node):

fpage = node.xpath(".//FirstPage/text()").extract_first()
lpage = node.xpath(".//LastPage/text()").extract_first()
if fpage and lpage:
try:
page_nr = str(int(lpage) - int(fpage) + 1)
else:
page_nr = ''
except (ValueError, TypeError):
page_nr = None

return (
fpage,
Expand Down
1 change: 1 addition & 0 deletions hepcrawl/spiders/iop_spider.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -159,6 +159,7 @@ def add_document(self, file_path, hidden, fulltext):
"fulltext": fulltext,
"description": self.name.upper(),
"url": file_path,
"key": os.path.basename(file_path)
}
return file_dict

Expand Down
2 changes: 1 addition & 1 deletion hepcrawl/tohep.py
View file
Original file line number Diff line number Diff line change
Expand Up @@ -243,7 +243,7 @@ def _filter_affiliation(affiliations):
for author in crawler_record.get('authors', []):
builder.add_author(builder.make_author(
full_name=author['full_name'],
affiliations=_filter_affiliation(author['affiliations']),
affiliations=_filter_affiliation(author.get('affiliations', [])),
))

for title in crawler_record.get('titles', []):
Expand Down
175 changes: 175 additions & 0 deletions tests/functional/iop/fixtures/test_iop.json
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,175 @@
[
{
"refereed": true,
"documents": [
{
"fulltext": true,
"description": "IOP",
"url": "file:///code/tests/functional/iop/fixtures/test_iop.xml",
"source": "iop",
"key": "test_iop.xml",
"hidden": true
}
],
"curated": false,
"_collections": [
"Literature"
],
"dois": [
{
"source": "iop",
"value": "10.1088/1742-6596/851/1/012001"
}
],
"acquisition_source": {
"source": "iop",
"method": "hepcrawl",
"submission_number": "None",
"datetime": "2018-01-12T13:06:39.088639"
},
"titles": [
{
"source": "iop",
"title": "Response of optically stimulated luminescence dosimeters subjected to X-rays in diagnostic energy range"
}
],
"copyright": [
{
"holder": "Institute of Physics"
}
],
"authors": [
{
"affiliations": [
{
"value": "Department of Physics, Universiti Teknologi Malaysia, Johor, Bahru Johor, 81300, Malaysia. Centre for Energy Research and Training, Ahmadu Bello University, Zaria, Kaduna State, P.M.B. 1014, Nigeria."
}
],
"full_name": "Musa, Y."
},
{
"full_name": "Hashim, S."
},
{
"full_name": "Karim, M. K A"
},
{
"full_name": "Bakar, K.A."
},
{
"full_name": "Ang, W.C."
},
{
"full_name": "Salehhon, N."
}
],
"publication_info": [
{
"journal_volume": "851",
"page_start": "012001",
"year": 2017,
"journal_issue": "1",
"journal_title": "J. Phys.: Conf. Ser."
}
],
"document_type": [
"article"
],
"abstracts": [
{
"source": "iop",
"value": "The use of optically stimulated luminescence (OSL) for dosimetry applications has recently increased considerably due to availability of commercial OSL dosimeters (nanoDots) for clinical use. The OSL dosimeter has a great potential to be used in clinical dosimetry because of its prevailing advantages in both handling and application. However, utilising nanoDot OSLDs for dose measurement in diagnostic radiology can only be guaranteed when the performance and characteristics of the dosimeters are apposite. In the present work, we examined the response of commercially available nanoDot OSLD (Al$_{2}$O$_{3}$:C) subjected to X-rays in general radiography. The nanoDots response with respect to reproducibility, dose linearity and signal depletion were analysed using microStar reader (Landauer, Inc., Glenwood, IL). Irradiations were performed free-in-air using 70, 80 and 120 kV tube voltages and tube currents ranging from 10 \u2013 100 mAs. The results showed that the nanoDots exhibit good linearity and reproducibility when subjected to diagnostic X-rays, with coefficient of variations (CV) ranging between 2.3% to 3.5% representing a good reproducibility. The results also indicated average of 1% signal reduction per readout. Hence, the nanoDots showed a promising potential for dose measurement in general X-ray procedure."
}
],
"imprints": [
{
"date": "2017"
}
],
"citeable": true
},
{
"refereed": true,
"documents": [
{
"fulltext": true,
"description": "IOP",
"url": "file:///code/tests/functional/iop/fixtures/test_iop.xml",
"source": "iop",
"key": "test_iop.xml",
"hidden": true
}
],
"curated": false,
"_collections": [
"Literature"
],
"dois": [
{
"source": "iop",
"value": "10.1088/1361-6560/aa6be8"
}
],
"acquisition_source": {
"source": "iop",
"method": "hepcrawl",
"submission_number": "None",
"datetime": "2018-01-12T13:06:39.192671"
},
"titles": [
{
"source": "iop",
"title": "Magnetic resonance imaging with hyperpolarized agents: methods and applications"
}
],
"copyright": [
{
"holder": "Institute of Physics"
}
],
"authors": [
{
"affiliations": [
{
"value": "Department of Medical Physics, University of Wisconsin\u2013Madison, Madison, WI, United States of America."
}
],
"full_name": "Adamson, Erin B."
},
{
"full_name": "Ludwig, Kai D."
},
{
"full_name": "Mummy, David G."
},
{
"full_name": "Fain, Sean B."
}
],
"publication_info": [
{
"page_end": "R123",
"journal_title": "Phys. Med. Biol.",
"journal_volume": "62",
"year": 2017,
"page_start": "R81",
"journal_issue": "13"
}
],
"document_type": [
"article"
],
"abstracts": [
{
"source": "iop",
"value": "In the past decade, hyperpolarized (HP) contrast agents have been under active development for MRI applications to address the twin challenges of functional and quantitative imaging. Both HP helium ($^{3}$He) and xenon ($^{129}$Xe) gases have reached the stage where they are under study in clinical research. HP $^{129}$Xe, in particular, is poised for larger scale clinical research to investigate asthma, chronic obstructive pulmonary disease, and fibrotic lung diseases. With advances in polarizer technology and unique capabilities for imaging of $^{129}$Xe gas exchange into lung tissue and blood, HP $^{129}$Xe MRI is attracting new attention. In parallel, HP $^{13}$C and $^{15}$N MRI methods have steadily advanced in a wide range of pre-clinical research applications for imaging metabolism in various cancers and cardiac disease. The HP [1-$^{13}$C] pyruvate MRI technique, in particular, has undergone phase I trials in prostate cancer and is poised for investigational new drug trials at multiple institutions in cancer and cardiac applications. This review treats the methodology behind both HP gases and HP $^{13}$C and $^{15}$N liquid state agents. Gas and liquid phase HP agents share similar technologies for achieving non-equilibrium polarization outside the field of the MRI scanner, strategies for image data acquisition, and translational challenges in moving from pre-clinical to clinical research. To cover the wide array of methods and applications, this review is organized by numerical section into (1) a brief introduction, (2) the physical and biological properties of the most common polarized agents with a brief summary of applications and methods of polarization, (3) methods for image acquisition and reconstruction specific to improving data acquisition efficiency for HP MRI, (4) the main physical properties that enable unique measures of physiology or metabolic pathways, followed by a more detailed review of the literature describing the use of HP agents to study: (5) metabolic pathways in cancer and cardiac disease and (6) lung function in both pre-clinical and clinical research studies, concluding with (7) some future directions and challenges, and (8) an overall summary."
}
],
"imprints": [
{
"date": "2017"
}
],
"citeable": true
}
]
106 changes: 106 additions & 0 deletions tests/functional/iop/fixtures/test_iop.xml
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,106 @@
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE ArticleSet PUBLIC "-//NLM//DTD PubMed 2.0//EN" "http://www.ncbi.nlm.nih.gov:80/entrez/query/static/PubMed.dtd">
<ArticleSet>
<Article>
<Journal>
<PublisherName>Institute of Physics</PublisherName>
<JournalTitle>J. Phys.: Conf. Ser.</JournalTitle>
<Issn>1742-6588</Issn>
<Volume>851</Volume>
<Issue>1</Issue>
<PubDate PubStatus="epublish">
<Year>2017</Year>
<Month>May</Month>
<Day>31</Day>
</PubDate>
</Journal>
<ArticleTitle>Response of optically stimulated luminescence dosimeters subjected to X-rays in diagnostic energy range</ArticleTitle>
<FirstPage LZero="save">012001</FirstPage>
<LastPage/>
<Language>EN</Language>
<AuthorList>
<Author>
<FirstName>Y</FirstName>
<LastName>Musa</LastName>
<Affiliation>Department of Physics, Universiti Teknologi Malaysia, Johor, Bahru Johor, 81300, Malaysia.
Centre for Energy Research and Training, Ahmadu Bello University, Zaria, Kaduna State, P.M.B. 1014, Nigeria.</Affiliation>
</Author>
<Author>
<FirstName>S</FirstName>
<LastName>Hashim</LastName>
</Author>
<Author>
<FirstName>M</FirstName>
<MiddleName>K A</MiddleName>
<LastName>Karim</LastName>
</Author>
<Author>
<FirstName>K</FirstName>
<MiddleName>A</MiddleName>
<LastName>Bakar</LastName>
</Author>
<Author>
<FirstName>W</FirstName>
<MiddleName>C</MiddleName>
<LastName>Ang</LastName>
</Author>
<Author>
<FirstName>N</FirstName>
<LastName>Salehhon</LastName>
</Author>
</AuthorList>
<ArticleIdList>
<ArticleId IdType="doi">10.1088/1742-6596/851/1/012001</ArticleId>
</ArticleIdList>
<Abstract>
The use of optically stimulated luminescence (OSL) for dosimetry applications has recently increased considerably due to availability of commercial OSL dosimeters (nanoDots) for clinical use. The OSL dosimeter has a great potential to be used in clinical dosimetry because of its prevailing advantages in both handling and application. However, utilising nanoDot OSLDs for dose measurement in diagnostic radiology can only be guaranteed when the performance and characteristics of the dosimeters are apposite. In the present work, we examined the response of commercially available nanoDot OSLD (Al<inf>2</inf>O<inf>3</inf>:C) subjected to X-rays in general radiography. The nanoDots response with respect to reproducibility, dose linearity and signal depletion were analysed using microStar reader (Landauer, Inc., Glenwood, IL). Irradiations were performed free-in-air using 70, 80 and 120 kV tube voltages and tube currents ranging from 10 – 100 mAs. The results showed that the nanoDots exhibit good linearity and reproducibility when subjected to diagnostic X-rays, with coefficient of variations (CV) ranging between 2.3% to 3.5% representing a good reproducibility. The results also indicated average of 1% signal reduction per readout. Hence, the nanoDots showed a promising potential for dose measurement in general X-ray procedure.
</Abstract>
</Article>
<Article>
<Journal>
<PublisherName>Institute of Physics</PublisherName>
<JournalTitle>Phys. Med. Biol.</JournalTitle>
<Issn>0031-9155</Issn>
<Volume>62</Volume>
<Issue>13</Issue>
<PubDate PubStatus="epublish">
<Year>2017</Year>
<Month>May</Month>
<Day>31</Day>
</PubDate>
</Journal>
<ArticleTitle>Magnetic resonance imaging with hyperpolarized agents: methods and applications</ArticleTitle>
<FirstPage LZero="save">R81</FirstPage>
<LastPage>R123</LastPage>
<Language>EN</Language>
<AuthorList>
<Author>
<FirstName>Erin</FirstName>
<MiddleName>B</MiddleName>
<LastName>Adamson</LastName>
<Affiliation>Department of Medical Physics, University of Wisconsin–Madison, Madison, WI, United States of America.</Affiliation>
</Author>
<Author>
<FirstName>Kai</FirstName>
<MiddleName>D</MiddleName>
<LastName>Ludwig</LastName>
</Author>
<Author>
<FirstName>David</FirstName>
<MiddleName>G</MiddleName>
<LastName>Mummy</LastName>
</Author>
<Author>
<FirstName>Sean</FirstName>
<MiddleName>B</MiddleName>
<LastName>Fain</LastName>
</Author>
</AuthorList>
<ArticleIdList>
<ArticleId IdType="doi">10.1088/1361-6560/aa6be8</ArticleId>
</ArticleIdList>
<Abstract>
In the past decade, hyperpolarized (HP) contrast agents have been under active development for MRI applications to address the twin challenges of functional and quantitative imaging. Both HP helium (<sup>3</sup>He) and xenon (<sup>129</sup>Xe) gases have reached the stage where they are under study in clinical research. HP <sup>129</sup>Xe, in particular, is poised for larger scale clinical research to investigate asthma, chronic obstructive pulmonary disease, and fibrotic lung diseases. With advances in polarizer technology and unique capabilities for imaging of <sup>129</sup>Xe gas exchange into lung tissue and blood, HP <sup>129</sup>Xe MRI is attracting new attention. In parallel, HP <sup>13</sup>C and <sup>15</sup>N MRI methods have steadily advanced in a wide range of pre-clinical research applications for imaging metabolism in various cancers and cardiac disease. The HP [1-<sup>13</sup>C] pyruvate MRI technique, in particular, has undergone phase I trials in prostate cancer and is poised for investigational new drug trials at multiple institutions in cancer and cardiac applications. This review treats the methodology behind both HP gases and HP <sup>13</sup>C and <sup>15</sup>N liquid state agents. Gas and liquid phase HP agents share similar technologies for achieving non-equilibrium polarization outside the field of the MRI scanner, strategies for image data acquisition, and translational challenges in moving from pre-clinical to clinical research. To cover the wide array of methods and applications, this review is organized by numerical section into (1) a brief introduction, (2) the physical and biological properties of the most common polarized agents with a brief summary of applications and methods of polarization, (3) methods for image acquisition and reconstruction specific to improving data acquisition efficiency for HP MRI, (4) the main physical properties that enable unique measures of physiology or metabolic pathways, followed by a more detailed review of the literature describing the use of HP agents to study: (5) metabolic pathways in cancer and cardiac disease and (6) lung function in both pre-clinical and clinical research studies, concluding with (7) some future directions and challenges, and (8) an overall summary.
</Abstract>
</Article>
</ArticleSet>
Loading

0 comments on commit 2173f2c

Please sign in to comment.