-
Notifications
You must be signed in to change notification settings - Fork 7
Deprecated Pages
Mason Watson edited this page Aug 16, 2023
·
29 revisions
CircleCI
PR #302 Remove use of CircleCI
Followed LHDI instructions and used lighthouse-di-circleci-java17-image. CircleCI was enabled in PR #16.
The configuration runs similar but not all operations as Github Actions.
-
config.yml uses
$GITHUB_USERNAMEand$GITHUB_ACCESS_TOKEN(to pull a Docker image), which are set in CircleCI's project settings (This needs to be done in the internal repo as well to get CircleCI running there.)
CircleCI does not push container images ("packages") to the Github Container Registry. We don't want to pollute it with development packages. See Docker-containers#Packages.
Commit c7b786c limits CircleCI runs for only the main and develop branches to reduce the time for PR checks.
MAS api spec (IBM hosted api)
{
"openapi": "3.0.3",
"info": {
"version": "1.0.0",
"title": "Mail Automation System - VRO Integration (Automated Benefits Delivery)",
"description": "Integration with VRO via MAS-hosted APIs",
"termsOfService": "",
"contact": {
"name": "IBM Dev Team",
"email": "[email protected]",
"url": ""
}
},
"servers": [{
"url": "https://viccs-api-dev.ibm-intelligent-automation.com/pca/api/dev",
"description": "(IBM - VICCS API)"
}
],
"paths": {
"/pcCheckCollectionStatus": {
"get": {
"tags": [
"pcCheckCollectionStatus"
],
"summary": "Get the status of the collection",
"description": "Get the status of the collection .i.e. whether it is OCRed, Indexed and ready to call the annotations API",
"operationId": "getCheckCollectionStatus",
"parameters": [{
"name": "Collection Identifiers",
"in": "query",
"description": "Collection Status Request",
"schema": {
"$ref": "#/components/schemas/collectionStatusReq"
}
}
],
"responses": {
"200": {
"description": "Successful operation",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/collectionStatusResp"
}
}
}
}
},
"400": {
"description": "Invalid input value"
}
},
"security": [{
"bearerAuth": []
}
]
}
},
"/pcQueryCollectionAnnots": {
"get": {
"tags": [
"pcQueryCollectionAnnots"
],
"summary": "Get the claim details",
"description": "Get the claim details",
"operationId": "pcQueryCollectionAnnots",
"parameters": [{
"name": "Collection Identifier",
"in": "query",
"description": "Get claim details Request",
"schema": {
"$ref": "#/components/schemas/collectionAnnotsReq"
}
}
],
"responses": {
"200": {
"description": "Success",
"content": {
"application/json": {
"schema": {
"type": "array",
"items": {
"$ref": "#/components/schemas/collectionAnnotsResp"
}
}
}
}
},
"422": {
"description": "Invalid input value",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
},
"default": {
"description": "unexpected error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
}
},
"security": [{
"bearerAuth": []
}
]
}
},
"/pcOrderExam": {
"post": {
"description": "Request a medical exam",
"operationId": "pcOrderExam",
"requestBody": {
"description": "Request a medical exam due to insufficient medical evidence for the condition specified in the claim",
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/orderExamReq"
}
}
}
},
"responses": {
"200": {
"description": "success",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/orderExamResp"
}
}
}
},
"422": {
"description": "Invalid input value",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
},
"default": {
"description": "Unexpected error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
}
},
"security": [{
"bearerAuth": []
}
]
}
}
},
"components": {
"schemas": {
"collectionStatusReq": {
"required": ["collectionId"],
"type": "object",
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "integer"
},
"collectionIds": {
"description": "List of unique identifiers for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "array",
"items": {
"type": "integer"
}
}
}
},
"collectionStatusResp": {
"type": "object",
"required": [
"collectionId",
"collectionStatus"
],
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "integer"
},
"collectionStatus": {
"description": "Status of the collection",
"type": "string",
"enum" : ["inProgress", "processed", "offramped", "vroNotified"]
}
}
},
"collectionAnnotsReq": {
"required": ["collectionId"],
"type": "object",
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "integer"
}
}
},
"documents": {
"required": ["eFolderVersionRefId ", "condition", "annotations"],
"type": "object",
"properties": {
"eFolderVersionRefId": {
"description": "eFolder version Reference ID",
"type": "integer"
},
"condition": {
"description": "Claims condition",
"type": "string"
},
"annotations": {
"description": "List of Annotations",
"type": "array",
"items": {
"$ref": "#/components/schemas/annotations"
}
}
}
},
"annotations": {
"type": "object",
"properties": {
"annotType": {
"description": "Annotation Type",
"type": "string"
},
"pageNum": {
"description": "Page Number",
"type": "string"
},
"annotName": {
"description": "Annotation Name",
"type": "string"
},
"annotVal": {
"description": "Annotation Value",
"type": "string"
},
"spellCheckVal": {
"description": "Spellcheck Value",
"type": "string"
},
"observationDate": {
"description": "Observation Date and Time (YYYY-MM-DDThh:mm:ss.sTZD)",
"type": "string",
"pattern": "(^\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(([+-]\\d\\d:\\d\\d)|Z)?$)"
},
"start": {
"description": "Start Value",
"type": "integer"
},
"end": {
"description": "End Value",
"type": "integer"
},
"acdPrefName": {
"description": "Acd Pref Name",
"type": "string"
},
"relevant": {
"description": "Is it relevant",
"type": "boolean"
}
}
},
"collectionAnnotsResp": {
"type": "object",
"required": [
"vtrnFileId",
"creationDate"
],
"properties": {
"vtrnFileId": {
"description": "Veteran File Identifier",
"type": "integer"
},
"creationDate": {
"description": "Claim creation date and Time (YYYY-MM-DDThh:mm:ss.sTZD)",
"type": "string",
"pattern": "(^\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(([+-]\\d\\d:\\d\\d)|Z)?$)"
},
"documents": {
"description": "List of documents",
"type": "array",
"items": {
"$ref": "#/components/schemas/documents"
}
}
}
},
"orderExamReq": {
"required": ["collectionId"],
"type": "object",
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "string"
}
}
},
"orderExamResp": {
"type": "object",
"required": [
"status"
],
"properties": {
"status": {
"description": "Order Exam Status",
"type": "string"
}
}
},
"error": {
"type": "object",
"required": [
"code",
"message"
],
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
},
"securitySchemes": {
"bearerAuth": {
"type": "http",
"scheme": "bearer",
"bearerFormat": "JWT"
}
}
}
}
MAS api spec (VRO hosted api)
{
"openapi": "3.0.0",
"info": {
"version": "1.0.0",
"title": "Mail Automation System - VRO Integration (Automated Benefits Delivery)",
"description": "Integration with Mail Automation System via VRO-hosted APIs",
"termsOfService": "",
"contact": {
"name": "ABD-VRO Maintenance Team",
"email": "[email protected]",
"url": ""
}
},
"servers": [{
"url": "http://localhost/abd-vro/v1",
"description": "(ABD-VRO API)"
}
],
"paths": {
"/automatedClaim": {
"post": {
"description": "Notify VRO of a new claim that has been forwarded to OCR and evidence gathering",
"operationId": "addclaimsNotification",
"requestBody": {
"description": "Claims Notification Request",
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/claimsNotification"
}
}
}
},
"responses": {
"200": {
"description": "Claims Notification Response",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/claimsNotificationResp"
}
}
}
},
"default": {
"description": "Unexpected error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
}
}
}
},
"/examOrderingStatus": {
"post": {
"description": "Notify health exam ordering status",
"operationId": "examOrderStatus",
"requestBody": {
"description": "Notify health exam ordering status",
"required": true,
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/examOrderStatus"
}
}
}
},
"responses": {
"200": {
"description": "Acknowledge Notify health exam ordering status",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/examOrderStatusResp"
}
}
}
},
"default": {
"description": "Unexpected error",
"content": {
"application/json": {
"schema": {
"$ref": "#/components/schemas/error"
}
}
}
}
}
}
}
},
"components": {
"schemas": {
"claimsNotification": {
"required": ["collectionId", "veteranIdentifiers", "dob", "firstName", "lastName", "claimDetail"],
"type": "object",
"properties": {
"veteranIdentifiers": {
"$ref": "#/components/schemas/veteranIdentifiers"
},
"dob": {
"description": "Date of Birth (yyyy-mm-dd format)",
"type": "string",
"pattern": "([12]\\d{3}-(0[1-9]|1[0-2])-(0[1-9]|[12]\\d|3[01]))"
},
"firstName": {
"description": "First Name",
"type": "string"
},
"lastName": {
"description": "Last Name",
"type": "string"
},
"gender": {
"description": "Gender",
"type": "string"
},
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "string"
},
"veteranFlashIds": {
"description": "Veteran Flash IDs",
"type": "array",
"items": {
"type": "string"
}
},
"claimDetail": {
"$ref": "#/components/schemas/claimDetail"
}
}
},
"veteranIdentifiers": {
"required": ["icn", "ssn", "edipn", "veteranFileId", "participantId"],
"type": "object",
"properties": {
"icn": {
"$ref": "#/components/schemas/icn"
},
"ssn": {
"$ref": "#/components/schemas/ssn"
},
"veteranFileId": {
"$ref": "#/components/schemas/veteranFileId"
},
"edipn": {
"$ref": "#/components/schemas/edipn"
},
"participantId": {
"$ref": "#/components/schemas/participantId"
}
}
},
"ssn": {
"description": "Veteran's Social Security number (Note: pass n/a in the absence of this field)",
"type": "string",
"default" : "N/A"
},
"icn": {
"description": "Veteran's Integration Control number (Note: pass n/a in the absence of this field)",
"type": "string",
"default" : "N/A"
},
"edipn": {
"description": "Veteran's DOD EDIPN ID (Electronic Data Interchange-Personal Identifier) (Note: pass n/a in the absence of this field)",
"type": "string",
"default" : "N/A"
},
"veteranFileId": {
"description": "Veteran File ID (a.k.a. BIRLS ID or CorpDB filenumber or VBMS filenumber)\n\nBIRLS : Beneficiary Identification Records Locator Subsystem\nVBMS: Veteran Benefits Management System\nCorpDB: VA Corporate Database (Note: pass n/a in the absence of this field)",
"type": "string",
"default" : "N/A"
},
"participantId": {
"description": "Veteran's participant id",
"type": "string",
"default" : "N/A"
},
"claimDetail": {
"required": ["benefitClaimId", "claimSubmissionDateTime", "claimSubmissionSource", "veteranFileId", "conditions"],
"type": "object",
"properties": {
"benefitClaimId": {
"description": "Benefit Claim Identifier",
"type": "string"
},
"claimSubmissionDateTime": {
"description": "Claims Submission Date and Time (YYYY-MM-DDThh:mm:ss.sTZD)",
"type": "string",
"pattern": "(^\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(([+-]\\d\\d:\\d\\d)|Z)?$)"
},
"claimSubmissionSource": {
"description": "Claims Submission Source VA.gov or MAS",
"type": "string",
"enum" : ["VA.GOV", "MAS", "OTHER"]
},
"conditions": {
"$ref": "#/components/schemas/claimCondition"
}
}
},
"claimCondition": {
"required": ["diagnosticCode"],
"type": "object",
"properties": {
"name": {
"description": "Claim Condition Name",
"type": "string"
},
"diagnosticCode": {
"description": "Claim Diagnostic Code",
"type": "string",
"enum" : ["7101"]
},
"disabilityActionType": {
"description": "Claim Disability Action Type",
"type": "string",
"enum" : ["INCREASE", "NEW"]
},
"disabilityClassificationCode": {
"description": "Claim Disability Classification Code",
"type": "string",
"enum" : ["3460", "3370"]
},
"ratedDisabilityId": {
"description": "Claim Rated Disability ID",
"type": "string"
}
}
},
"claimsNotificationResp": {
"type": "object",
"required": [
"id",
"message"
],
"properties": {
"id": {
"description": "Unique ID to identify the transaction (for audit and debug purpose)",
"type": "string"
},
"message": {
"type": "string"
}
}
},
"examOrderStatus": {
"required": ["collectionId", "collectionStatus"],
"type": "object",
"properties": {
"collectionId": {
"description": "Unique identifier for the collection of annotations resulting from OCR and NLP processing of relevant documents",
"type": "string"
},
"collectionStatus": {
"description": "Claim Collection Status",
"type": "string",
"enum" : [ "DRAFT", "FINAL", "ERROR"]
},
"examOrderDateTime": {
"description": "Exam order Date and Time (YYYY-MM-DDThh:mm:ss.sTZD)",
"type": "string",
"pattern": "(^\\d{4}-\\d\\d-\\d\\dT\\d\\d:\\d\\d:\\d\\d(\\.\\d+)?(([+-]\\d\\d:\\d\\d)|Z)?$)"
}
}
},
"examOrderStatusResp": {
"type": "object",
"required": [
"id",
"message"
],
"properties": {
"id": {
"description": "Unique ID to identify the transaction (for audit and debug purpose)",
"type": "string"
},
"message": {
"type": "string"
}
}
},
"error": {
"type": "object",
"required": [
"code",
"message"
],
"properties": {
"code": {
"type": "string"
},
"message": {
"type": "string"
}
}
}
},
"securitySchemes": {
"ApiKeyAuth": {
"type": "apiKey",
"description": "X-API-KEY:valid-api-key",
"name": "X-API-KEY",
"in": "header"
}
}
},
"security": [{
"ApiKeyAuth": []
}
],
"tags": []
}```
</details>
<details>
<summary>PDF Generator</summary>
## Introduction
The PDF Generator contains all the different templates used to generate documents by providing the appropriate data in the generation requests. The following libraries are available for rendering:
- WKHTMLTOPDF: https://wkhtmltopdf.org/
- WeasyPrint: https://weasyprint.org/
The PDF Generator routes allow you to specify which library you wish to use and if it is not provided in the request, it will default to `WKHTMLTOPDF`
## Queues and Endpoints
On `pdfgenerator` startup, the consumer will attempt to create a `generate-pdf`, `fetch-pdf`, and `generate-fetch-pdf` queue on a `pdf-generator` exchange.
- [Request PDF Generation - POST /evidence-pdf](http://localhost:8080/v1/evidence-pdf)
- [Fetch Generated PDF - GET /evidence-pdf/{claimSubmissionID}](http://localhost:8080/v1/evidence-pdf)
Both functions work off the same endpoint. The main difference being that the `POST` has a JSON request body while the `GET` uses the URL(`claimSubmissionID`) to find the corresponding PDF.
- [Immediate PDF - POST /immediate-pdf](http://localhost:8080/v1/immediate-pdf)
This endpoint is a combined version of the `generate-pdf` and `fetch-pdf`. Pass it the `POST` request body and it will respond with the PDF associated with the specified `claimSubmissionID`
## Generation Process
### Generating the PDF (1/2)
Any messages passed to the `generate-pdf` queue will use the `diagnosticCode`, `pdfTemplate` and `pdfLibrary`(optional) to select the appropriate template and generate the PDF. `diagnosticCode` is used to translate this into a human readable diagnostic type based on the mapping in `config/settings.py` in the `codes` variable.
Using `pdfTemplate` like `v1` along with a diagnostic type like `hypertension` for example, it will pull up the appropriate template and template variables in the `templates` and `template_variables` folder respectively. For example, `"diagnosticCode"=7101` and `"pdfTemplate"="v1"`will fetch `templates/hypertension-v1.html` and `template_variables/hypertension-v1.json`.
The generator will first load the `hypertension-v1.json` file which is prefilled with default values for the available variables within the template and attempt to replace them with what is provided in the message. If it cannot replace the data based on what's provided in the request, it will keep the default that is defined in the JSON file.
After the HTML template is generated with the replaced variables, it uses `WKHTMLTOPDF` by default or the library selected by `pdfLibrary` to create a PDF based on the HTML file.
### Saving the PDF (2/2)
Once the PDF has been generated, the consumer will create a key value pair in Redis to store the data due to it containing PII information. The `key` being `claimSubmissionId` while the `value` is a base64 encoded string representation of the PDF.
The consumer will return a response similar to:{ "claimSubmissionId: "1", "status": "COMPLETE" }
### Fetching the PDF
When the consumer receives a message in the `fetch-pdf` queue, it will use the provided `claimSubmissionId` to look it up on Redis.
If the PDF still hasn't been generated, you will receive a response similar to:
{ "claimSubmissionId: "1", "status": "IN_PROGRESS" }
but if the PDF is available then the response will be a downloadable file
## Libraries
### WKHTMLTOPDF
#### Table of Contents
ToCs are not part of the normal HTML template that gets generated for the PDF. They need to be created through a different process and merged with the main PD template
The PDF generator will check if there is a ToC file already created for the `diagnosticCode` that gets passed. If not found, it will generate the PDF without a ToC so you don't have to worry about having a empty section or page
#### Add a ToC for a new code:
1. Create a directory in `templates` where the name will be the human readable diagnosis name used in the `codes` variable in `settings.py`
2. Within this folder, create a `toc.xsl` file. Most ToCs will follow the same format so you can just copy one from any other diagnosis if available. If you needed to create one from scratch, in the command line run the following: `wkhtmltopdf --dump-default-toc-xsl` and copy the contents of the output to a new `toc.xsl` file as stated above
3. By default a ToC is generated by finding all `<h?>` related tags(`<h1>, <h2>, etc`) so you need to modify them if you want them ignored.
1. To ignore a heading, it must start with `‌` like this example: `<h3 class="text-center">‌Test PDF Heading</h3>`. The `toc.xsl` file has logic in place to skip over any headings that start with this special character. This character was used since it's an invisible character so it won't render on the PDF
4. The ToC page is fully customizable just like any HTML page
#### Notes
1. The library was built using Webkit 2.2 (~2012) and QT4 (2015) so many newer HTML features are unavailable or need to be added through other ways for them to render properly
2. This library renders at 96DPI but the value can be altered through the meta tags. We need to verify that the Figma design or other design software matches the proper DPI settings by making sure the resolution matches the paper size. Use the following links for proper conversions: https://a-size.com/legal-size-paper/ and https://www.papersizes.org/a-sizes-in-pixels.htm as well as https://pixelsconverter.com/inches-to-pixels
### WeasyPrint
#### Table of Contents
Work in Progress
#### Notes
1. This library does not accept Javascript. At the moment, we would need to come up with a workaround by prerendering in a secondary library or just using `WKHTMLTOPDF` for Javascript specific portions but this solution has yet to be implemented.
2. This library renders at 96DPI and the value cannot be changed. We need to verify that the Figma design or other design software matches the proper DPI settings by making sure the resolution matches the paper size. Use the following links for proper conversions: https://a-size.com/legal-size-paper/ and https://www.papersizes.org/a-sizes-in-pixels.htm as well as https://pixelsconverter.com/inches-to-pixels
## Development and Testing
## Development
### Setting up a Design
Before you can start working on the content of the PDF design, you must first match your DPI to the size dimensions of the design tool. If you skip this step, then the measurements will be all wrong and the design won't be a perfect match. This in turn will cause the developer to modify/test random values or mess with the `zoom` setting to get it to fit the design.
For example, if a user wants to make a new PDF, they must:
1. Get the size/dimension that the design team wants to use. Not just whether its A4, Legal, etc. but the pixel dimensions of the blank design page. We will use this number to set or match the DPI accordingly
2. Setup the new template to match based on the library:
- `WKHTMLTOPDF`: DPI is customizable so you can use the following for proper conversions:
- [Legal](https://a-size.com/legal-size-paper/) and [other sizes like A4, etc.](https://www.papersizes.org/a-sizes-in-pixels.htm): See what DPI setting the pixel dimensions fall under and set the meta tag to that DPI once you make the template file in the next step
- `WeasyPrint`: The DPI value is set to 96 and cannot be changed. Due to this, the process is somewhat backwards. The developer needs to use the above links to get the dimensions based on the requested page size and 96 DPI and send the dimensions back to the design team so they can adjust their document to match.
### Adding a new Diagnostic Code and Template
1. Edit the `codes` dictionary in `config/settings.py` by adding a new key, value pair for your code.
- Example:
```
codes = {
"0000": "cancer",
"0001": "diabetes" //new code with human readable diagnosis
}
```
2. Create a HTML version of the PDF you want to generate and save it in the `templates` folder along with a version number
- Take a look at [Jinja 2 Template Documentation](https://jinja.palletsprojects.com/en/3.1.x/templates/) for a better idea of what you can do within the template files
- Every template file needs a version number. By default, the system looks for `v1` if one is not specified in the request
- The file name should match the name you used in Step 1. Following that example, it should be called `diabetes-v1.html`
3. Create a JSON file in `template_variables` that will contain default values for the HTML file in case they are not provided in the RabbitMQ message
- The file name should match the name you used in Step 1 and 2. Following that example, it should be called `diabetes-v1.json`
### Helper Functions
Some diagnostic codes might need specific changes that don't need to affect other templates and instead of adding it to the assessment logic, we can use a helper function.
When generating a PDF, it will look for a helper function following this naming convention `pdf_helper_0000` where `0000` would be the code we want to use. If it does not find it, it move on and then applies a `pdf_helper_all` that gets applied to every single template. Usually these are edits like turning date string into proper date time objects, etc that would benefit all the templates.
## Testing
Currently there are 2 ways to develop/test the PDF service:
1. Run `./gradlew build check docker` to build all containers and run a full test. This can be used for the testing any updates that are made to the endpoints through Swagger but it takes longer due to having to load all the containers. After the containers are built, you can take it a step further and run the containers themselves using `./gradlew app:dockerComposeDown app:dcPrune app:dockerComposeUp` and then heading to the Swagger page to view and run the available endpoints.
2. Run `python pdfgenerator/src/lib/local_pdf_test.py` from the `service-python` directory. This file calls the PDF generator while bypassing all the related RabbitMQ and Redis code. You can alter the `diagnosis_name` and `message` to simulate an endpoint request and to quickly debug any template or PDF issues. The `diagnosis_name` should be the full name including version number like `hypertension-v1`
</details>