-
Notifications
You must be signed in to change notification settings - Fork 5
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[api] Add cost surface piece exporter and importer [MRXN23-111] [MRXN23-19] [MRXN23-112] [MRXN23-20] #1503
Merged
hotzevzl
merged 17 commits into
develop
from
chore/api/MRXN23-111-add-cost-surface-piece-exporter
Oct 6, 2023
Merged
[api] Add cost surface piece exporter and importer [MRXN23-111] [MRXN23-19] [MRXN23-112] [MRXN23-20] #1503
Changes from all commits
Commits
Show all changes
17 commits
Select commit
Hold shift + click to select a range
6b307bb
Add basic project cost surfaces piece exporter
yulia-bel 904de73
Add project cost surface piece exporter to ExportResourcePiecesAdapter
yulia-bel a9c0774
Fix format in api
yulia-bel 102cb50
Add migration file to update clone_piece_enum to include 'project-cos…
yulia-bel 9ab5a6f
Update export piece adapter unit test
yulia-bel c55b13f
Remove unused code
yulia-bel 52a1a13
Add ProjectCostSurfacesPieceExporter to pieces-exporter module
yulia-bel 0c82d5c
Add ProjectCostSurfacesPieceImporter for project cloning
yulia-bel 0fea4e6
Update project cost surfaces exporters and importers
yulia-bel 69b33a4
Fix api formatting
yulia-bel 93dded8
Add geo test for Cost Surface Piece Exporter
yulia-bel 1d14ab4
Update geo test for Cost Surface Piece Exporter
yulia-bel 792f42a
Add geo test for Cost Surface Piece Importer
yulia-bel d596d71
Update geo test for Cost Surface Piece Importer
yulia-bel 29e3b93
Update piece importers order
yulia-bel cb1c0c2
Update cost surfaces cleanup test
yulia-bel 45bff80
Update cost surfaces piece exporter + format fixes
yulia-bel File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
52 changes: 52 additions & 0 deletions
52
api/apps/api/src/migrations/api/1695040870835-AddClonePieceForProjectCostSurfaces.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,52 @@ | ||
| import { MigrationInterface, QueryRunner } from 'typeorm'; | ||
|
|
||
| export class AddClonePieceForProjectCostSurfaces1695040870835 | ||
| implements MigrationInterface { | ||
| public async up(queryRunner: QueryRunner): Promise<void> { | ||
| await queryRunner.query( | ||
| `ALTER TYPE clone_piece_enum ADD VALUE 'project-cost-surfaces'`, | ||
| ); | ||
| } | ||
|
|
||
| public async down(queryRunner: QueryRunner): Promise<void> { | ||
| await queryRunner.query(` | ||
| CREATE TYPE "clone_piece_enum_tmp" AS ENUM( | ||
| 'export-config', | ||
| 'project-metadata', | ||
| 'project-custom-protected-areas', | ||
| 'planning-area-gadm', | ||
| 'planning-area-custom', | ||
| 'planning-area-custom-geojson', | ||
| 'planning-units-grid', | ||
| 'planning-units-grid-geojson', | ||
| 'scenario-metadata', | ||
| 'scenario-planning-units-data', | ||
| 'scenario-run-results', | ||
| 'scenario-protected-areas', | ||
| 'project-custom-features', | ||
| 'features-specification', | ||
| 'scenario-features-data', | ||
| 'scenario-input-folder', | ||
| 'scenario-output-folder', | ||
| 'marxan-execution-metadata', | ||
| 'project-puvspr-calculations' | ||
| ); | ||
| `); | ||
|
|
||
| await queryRunner.query(` | ||
| ALTER TABLE export_components | ||
| ALTER COLUMN piece TYPE clone_piece_enum_tmp; | ||
| `); | ||
| await queryRunner.query(` | ||
| ALTER TABLE import_components | ||
| ALTER COLUMN piece TYPE clone_piece_enum_tmp; | ||
| `); | ||
|
|
||
| await queryRunner.query(` | ||
| DROP TYPE clone_piece_enum; | ||
| `); | ||
| await queryRunner.query(` | ||
| ALTER TYPE clone_piece_enum_tmp RENAME TO clone_piece_enum; | ||
| `); | ||
| } | ||
| } |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
144 changes: 144 additions & 0 deletions
144
api/apps/geoprocessing/src/export/pieces-exporters/project-cost-surfaces.piece-exporter.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,144 @@ | ||
| import { geoprocessingConnections } from '@marxan-geoprocessing/ormconfig'; | ||
| import { ClonePiece, ExportJobInput, ExportJobOutput } from '@marxan/cloning'; | ||
| import { ComponentLocation, ResourceKind } from '@marxan/cloning/domain'; | ||
| import { ClonePieceRelativePathResolver } from '@marxan/cloning/infrastructure/clone-piece-data'; | ||
| import { | ||
| ProjectCustomFeature, | ||
| ProjectCustomFeaturesContent, | ||
| } from '@marxan/cloning/infrastructure/clone-piece-data/project-custom-features'; | ||
| import { CloningFilesRepository } from '@marxan/cloning-files-repository'; | ||
| import { GeometrySource } from '@marxan/geofeatures'; | ||
| import { Injectable, Logger } from '@nestjs/common'; | ||
| import { InjectEntityManager } from '@nestjs/typeorm'; | ||
| import { isLeft } from 'fp-ts/lib/Either'; | ||
| import { Readable } from 'stream'; | ||
| import { EntityManager } from 'typeorm'; | ||
| import { | ||
| ExportPieceProcessor, | ||
| PieceExportProvider, | ||
| } from '../pieces/export-piece-processor'; | ||
| import { ProjectsPuEntity } from '@marxan-jobs/planning-unit-geometry'; | ||
| import { ProjectCostSurfacesContent } from '@marxan/cloning/infrastructure/clone-piece-data/project-cost-surfaces'; | ||
|
|
||
| type CreationStatus = ProjectCustomFeature['creation_status']; | ||
|
|
||
| type ProjectCostSurfacesSelectResult = { | ||
| id: string; | ||
| name: string; | ||
| min: number; | ||
| max: number; | ||
| is_default: boolean; | ||
| }; | ||
|
|
||
| type CostSurfaceDataSelectResult = { | ||
| cost_surface_id: string; | ||
| cost: number; | ||
| projects_pu_id: number; | ||
| }; | ||
|
|
||
| @Injectable() | ||
| @PieceExportProvider() | ||
| export class ProjectCostSurfacesPieceExporter implements ExportPieceProcessor { | ||
| private readonly logger: Logger = new Logger( | ||
| ProjectCostSurfacesPieceExporter.name, | ||
| ); | ||
|
|
||
| constructor( | ||
| private readonly fileRepository: CloningFilesRepository, | ||
| @InjectEntityManager(geoprocessingConnections.apiDB) | ||
| private readonly apiEntityManager: EntityManager, | ||
| @InjectEntityManager(geoprocessingConnections.default) | ||
| private readonly geoprocessingEntityManager: EntityManager, | ||
| ) {} | ||
|
|
||
| isSupported(piece: ClonePiece, kind: ResourceKind): boolean { | ||
| return ( | ||
| piece === ClonePiece.ProjectCostSurfaces && kind === ResourceKind.Project | ||
| ); | ||
| } | ||
|
|
||
| async run(input: ExportJobInput): Promise<ExportJobOutput> { | ||
| const costSurfaces: ProjectCostSurfacesSelectResult[] = await this.apiEntityManager | ||
| .createQueryBuilder() | ||
| .select(['cs.id', 'cs.name', 'cs.min', 'cs.max', 'cs.is_default']) | ||
| .from('cost_surfaces', 'cs') | ||
| .where('cs.project_id = :projectId', { projectId: input.resourceId }) | ||
| .execute(); | ||
|
|
||
| const costSurfacesIds = costSurfaces.map((costSurface) => costSurface.id); | ||
| let costSurfaceData: CostSurfaceDataSelectResult[] = []; | ||
| let projectPusMap: Record<string, number> = {}; | ||
|
|
||
| if (!costSurfacesIds) { | ||
| const errorMessage = `${ProjectCostSurfacesPieceExporter.name} - Project Cost Surfaces - couldn't find cost surfaces for project ${input.resourceId}`; | ||
| this.logger.error(errorMessage); | ||
| throw new Error(errorMessage); | ||
| } | ||
| costSurfaceData = await this.geoprocessingEntityManager | ||
| .createQueryBuilder() | ||
| .select(['cost_surface_id', 'cost', 'projects_pu_id']) | ||
| .from('cost_surface_pu_data', 'scpd') | ||
| .where('cost_surface_id IN (:...costSurfacesIds)', { | ||
| costSurfacesIds, | ||
| }) | ||
| .execute(); | ||
|
|
||
| projectPusMap = await this.getProjectPusMap(input.resourceId); | ||
|
|
||
| const fileContent: ProjectCostSurfacesContent = { | ||
| costSurfaces: costSurfaces.map(({ id, ...costSurface }) => ({ | ||
| ...costSurface, | ||
| data: costSurfaceData | ||
| .filter( | ||
| (data: CostSurfaceDataSelectResult) => data.cost_surface_id === id, | ||
| ) | ||
| .map(({ cost_surface_id, projects_pu_id, ...data }) => { | ||
| const puid = projectPusMap[projects_pu_id]; | ||
| return { puid, ...data }; | ||
| }), | ||
| })), | ||
| }; | ||
|
|
||
| const relativePath = ClonePieceRelativePathResolver.resolveFor( | ||
| ClonePiece.ProjectCostSurfaces, | ||
| ); | ||
|
|
||
| const outputFile = await this.fileRepository.saveCloningFile( | ||
| input.exportId, | ||
| Readable.from(JSON.stringify(fileContent)), | ||
| relativePath, | ||
| ); | ||
|
|
||
| if (isLeft(outputFile)) { | ||
| const errorMessage = `${ProjectCostSurfacesPieceExporter.name} - Project Cost Surfaces - couldn't save file - ${outputFile.left.description}`; | ||
| this.logger.error(errorMessage); | ||
| throw new Error(errorMessage); | ||
| } | ||
|
|
||
| return { | ||
| ...input, | ||
| uris: [new ComponentLocation(outputFile.right, relativePath)], | ||
| }; | ||
| } | ||
|
|
||
| private async getProjectPusMap( | ||
| projectId: string, | ||
| ): Promise<Record<string, number>> { | ||
| const projectPus: { | ||
| id: string; | ||
| puid: number; | ||
| }[] = await this.geoprocessingEntityManager | ||
| .createQueryBuilder() | ||
| .select(['id', 'puid']) | ||
| .from(ProjectsPuEntity, 'ppus') | ||
| .where('ppus.project_id = :projectId', { projectId }) | ||
| .execute(); | ||
|
|
||
| const projectPuIdByPuid: Record<string, number> = {}; | ||
| projectPus.forEach(({ puid, id }) => { | ||
| projectPuIdByPuid[id] = puid; | ||
| }); | ||
|
|
||
| return projectPuIdByPuid; | ||
| } | ||
| } | ||
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
just a note on style - my recommendation for these kinds of "augmentations" is to use named functions to make what we're doing via array operations (filter, map, etc.), spreads and so on clearer.
In practice here we're combining a lookup of cost surface costs with the setter part of a lens, using lower-level primitives: which is ok and works, but makes even a tiny snippet of code like this less legible than something like
const fileContent: ProjectCostSurfacesContent = addCostDataToCostSurfaces(costSurfaces, costSurfaceCostData)but no need to do anything here! (and as always, these kinds of style comments can be highly opinable)