New API: Allow for full-fledged processing of protection profiles

[adamjanovsky]

Currently, the static dataset of protection profiles is loaded from web storage. This dataset was created using old API.

The new tool should allow for full processing of protection profiles without downloading the static dataset.

[adamjanovsky]

@J08nY I did some brief requirement analysis on this:

• I think this deserves a stand-alone dataset class, I would no longer represent it as a auxiliary_dataset
• We should think of primary key (name + link) used to link both datasets together?
• There are two types of protection profiles: regular PPs and collaborative PPs.
• Collaborative PPs have bunch of very diverse references: https://www.commoncriteriaportal.org/pps/collaborativePP.cfm?cpp=1, not sure if we should try to process them.
• As a minimum, I would download and convert PP and the corresponding certification report.
• AFAIK we don't use them at all now (except for some linking to the static dataset), so there should be no to little refactoring needed.
• Obviously, this warrants many new tests.

I'm wondering what are the benefits of implementing this?

• We could link to up-to-date PPs.
• We could process the PPs with our generic list of regexes, that could lead to some interesting analysis
• We could maybe invent new regexes to drive the analysis?
• We would have another thing to display at the web, but that would require additional work by @J08nY.

@mukrop all in all, I think this should fit 40 hours of work...

[J08nY]

• We would have another thing to display at the web, but that would require additional work by @J08nY.

This is already done, see for example this. If the format would change it would need changes though.

I'm wondering what are the benefits of implementing this?

Well the up-to-date part is quite important I think. Without considering PPs I believe our analysis is incomplete, also see #157 for example. In general the current PP dataset is not of great quality, many PPs are missing IDs and thus many certificates that have PPs do not display them properly because we will not match them (even after my matching fix goes through).

[adamjanovsky]

@J08nY so, this has a green light from @mukrop, meaning I'll implement it over the course of November (hopefully). Design-wise, I think that the primary key must be pp name + pp link, as that's the only information that's available from the certificate list page.

Do you have any other specific requirements on this, or do I proceed as I deem fitting and consult when needed?

[J08nY]

Please make the primary key only the PP filename as is done for the new dgst building in CC certificate, there is a function for that in sanitization helpers. This way we won't get screwed if they change the paths again.

[J08nY]

Please make the primary key only the PP filename as is done for the new dgst building in CC certificate, there is a function for that in sanitization helpers. This way we won't get screwed if they change the paths again.