csv exporting with value_sets

[aaroncramsey]

Hello, I'm playing around with custom modules and trying to use value_set, and I'm occasionally running into ConcurrentModificationException's when exporting to CSV (I haven't seen it happen when exporting to fhir). I first ran into it with my own custom module, then I tried it with a module I found at https://github.com/synthetichealth/synthea/blob/dac026690cb744e209396f3b2ed3ceadc14c8e84/src/test/resources/generic/imaging_study_with_valueset.json and ran into the same thing. It seems to mostly work but these pop up every now and then.

Wondering if there's an option I'm missing or if I'm doing something wrong?.

java.util.ConcurrentModificationException at java.util.ArrayList$ArrayListSpliterator.tryAdvance(ArrayList.java:1363) at java.util.stream.ReferencePipeline.forEachWithCancel(ReferencePipeline.java:126) at java.util.stream.AbstractPipeline.copyIntoWithCancel(AbstractPipeline.java:499) at java.util.stream.AbstractPipeline.copyInto(AbstractPipeline.java:486) at java.util.stream.AbstractPipeline.wrapAndCopyInto(AbstractPipeline.java:472) at java.util.stream.FindOps$FindOp.evaluateSequential(FindOps.java:152) at java.util.stream.AbstractPipeline.evaluate(AbstractPipeline.java:234) at java.util.stream.ReferencePipeline.findFirst(ReferencePipeline.java:464) at org.mitre.synthea.export.CSVExporter.export(CSVExporter.java:460) at org.mitre.synthea.export.Exporter.exportRecord(Exporter.java:257) at org.mitre.synthea.export.Exporter.export(Exporter.java:158) at org.mitre.synthea.engine.Generator.generatePerson(Generator.java:543) at org.mitre.synthea.engine.Generator.lambda$run$3(Generator.java:370) at java.util.concurrent.FutureTask.run(FutureTask.java:266) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) at java.lang.Thread.run(Thread.java:748)

I running something like java -jar synthea-with-dependencies.jar -p 1000 --exporter.csv.export true --exporter.fhir.export false -d my_modules/ --generate.terminology_service_url https://r4.ontoserver.csiro.au/fhir

[jawalonoski]

It looks like the CSV exporter has some logic to only export diagnosed conditions if they were mapped by the terminology service, and execution of that check uses stream processing that results in the ConcurrentModificationException.

Perhaps RandomCodeGenerator.selectedCodes.stream()... should become RandomCodeGenerator.selectedCodes.parallelStream() since multiple patients can be exported simultaneously.

synthea/src/main/java/org/mitre/synthea/export/CSVExporter.java

Lines 456 to 462 in 842ca4b

	if (!StringUtils.isEmpty(Config.get("generate.terminology_service_url"))
	&& !RandomCodeGenerator.selectedCodes.isEmpty()) {
	if (RandomCodeGenerator.selectedCodes.stream()
	.filter(code -> code.code.equals(condition.codes.get(0).code))
	.findFirst().isPresent()) {
	condition(personID, encounterID, condition);
	}