Track LPNF vs. OPNF performance

[AlexLandau]

The goal of these is to determine if we can get rid of the LegacyPropNetFactory entirely in favor of the OptimizingPropNetFactory.

Results from a recent round of testing that focused on time taken:

The LPNF is faster on smaller games, where the exact time for computation doesn't really matter. The OPNF consistently takes at least 200 ms to run, while the LPNF can finish in under 20 ms. Instead, we look for games where the OPNF takes several seconds to run (at least) or fails while the LPNF is significantly faster.

Games where the OPNF takes at least 5 seconds to build a propnet and the LPNF is significantly faster:

Dresden/ggp-course2013_jordi [256] (6 vs. 1 seconds)
GGP.org/pancakes6 [70] (10 vs. <1 seconds)
GGP.org/pancakes [215] (10 vs. <1 seconds)
Dresden/knightstour [260] (40 vs. <1 seconds)
Dresden/ggp-course2013_a1 [396] (73 vs. 4 seconds)

Games where the OPNF times out (120 second limit) or fails and the LPNF does not:

GGP-Base/test_case_5a [483]
Stanford/tictictoe [429] (fails StaticValidator)
GGP.org/knightsTourLarge [21]
GGP.org/factoringImpossibleTurtleBrain [18]
Dresden/ggp-course2013_michal13 [373]
GGP.org/knightwar [33]
GGP.org/pancakes88 [171]
Dresden/ggp-course2013_nicolai [338]
GGP.org/god [66]

Note that games listed above may be earlier versions of games that have since been modified. (For some of these games, at least, adding bases and inputs makes the OPNF faster.) I need to go through and weed these out. Numbers in square brackets are game IDs from a personal database, for my reference (and so my output code doesn't conflate different versions of a game).

[AlexLandau]

For propnet size as measured by the number of components, even small propnets are better coming out of the OPNF. (These measurements are before any post-optimizations are done.)

Games where the number of components from the OPNF is greater than the number of components from the LPNF:

GGP-Base/test_case_3a [476] (fails StaticValidator; 39 vs. 36)
Dresden/rendezvous_asteroids [422] (fails StaticValidator; 1500 vs. 1473)
Dresden/asteroidsparallel [350] (fails StaticValidator; 1524 vs. 1056)
Dresden/SC_TestOnly_enabled [378] (2567 vs. 2069)
GGP.org/catcha_mouse [136] (4919 vs. 4443)
GGP.org/wallmaze [230] (fails StaticValidator; 34571 vs. 11264)
GGP.org/wallmaze [568] (34571 vs. 11264)
GGP.org/cubicup [151] (37251 vs. 4429)
GGP.org/cubicup_3player [9] (77477 vs. 15058)
Dresden/ggp-course2013_jordi [256] (83000 vs. 12962)
GGP.org/pancakes6 [70] (466732 vs. 22309)
GGP.org/pancakes [215] (466738 vs. 22309)
Dresden/ggp-course2013_a1 [396] (729008 vs. 264848)
Dresden/knightstour [260] (932424 vs. 2471)

And then there are the eight games listed above where OPNF creation failed and LPNF creation did not. Again, some of these games have been superseded.

[AlexLandau]

And measuring link counts instead of component counts:

GGP-Base/test_case_3a [476] (fails StaticValidator; 37 vs. 32)
GGP.org/asteroids [12] (fails StaticValidator; 1517 vs. 711)
Dresden/asteroidsparallel [350] (fails StaticValidator; 2986 vs. 990)
Dresden/rendezvous_asteroids [422] (fails StaticValidator; 3062 vs. 1827)
GGP.org/gt_coordination [178] (3322 vs. 3013)
Dresden/SC_TestOnly_enabled [378] (4098 vs. 3380)
GGP.org/asteroidsSerial [108] (4578 vs. 3053)
Dresden/asteroidsserial [372] (fails StaticValidator; 4578 vs. 3053)
GGP.org/catcha_mouse [136] (8015 vs. 7786)
GGP.org/cubicup [151] (62481 vs. 6422)
GGP.org/wallmaze [230] (fails StaticValidator; 94309 vs. 23219)
GGP.org/wallmaze [568] (94309 vs. 23219)
GGP.org/cubicup_3player [9] (132959 vs. 21784)
Dresden/ggp-course2013_jordi [256] (197160 vs. 18927)
GGP.org/hitori [133] (640091 vs. 527142)
GGP.org/pancakes6 [70] (979982 vs. 31135)
GGP.org/pancakes [215] (979988 vs. 31124)
Dresden/knightstour [260] (1856904 vs. 2883)
Dresden/ggp-course2013_a1 [396] (3717343 vs. 333361)
Dresden/javastrike [285] (4147988 vs. 3816158)