Test failures not in octave's __run_test_suite__ #72

apjanke · 2019-04-19T21:14:28Z

Testify is failing more tests than core Octave is.

Octave 4.4.1:

Core __run_test_suite__:

Summary:

  PASS                            14933
  FAIL                                7
  XFAIL (reported bug)               54
  SKIP (missing feature)             50
  SKIP (run-time condition)          54


Testify:

Summary:

  GNU Octave Version: 4.4.1 (hg id: 1f46d371968c + patches)
  Tests run on angharad.local (macOS) at 19-Apr-2019 17:13:18
  Test execution time: 00:10:33

  PASS                            18090
  FAIL                               34
  XFAIL (reported bug)              222
  SKIP (missing feature)             23
  SKIP (run-time condition)          60

  Failed tests:
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/pr-output.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/miscellaneous/inputname.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isocaps.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isosurface.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/reducevolume.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/copyobj.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/hgsave.m

And why is it reporting more PASSes, too? 18090 vs 14933.

Files with failed tests

And doesn't that seem like too few files to have 34 failures?

The text was updated successfully, but these errors were encountered:

apjanke · 2019-04-19T21:15:28Z

inputname:

Aw, crap. At least some of the BISTs are depending on the internal implementation details of test.

Here's some test code from inputname.m:

## Warning: heap big magic in the following tests!!!
## The test function builds a private context for each test, with only the
## specified values shared between them.  It does this using the following
## template:
##
##     function [<shared>] = testfn (<shared>)
##        <test>
##     endfunction
##
## To test inputname, I need a function context invoked with known parameter
## names.  So define a couple of shared parameters, et voila!, the test is
## trivial.

%!shared hello, worldly
%!assert (inputname (1), "hello")
%!assert (inputname (2), "worldly")
%!assert (inputname (3), "")

## Clear parameter list so that testfn is created with zero inputs/outputs
%!shared
%!assert (inputname (-1), "")
%!assert (inputname (1), "")

That seems to rely on the fact that Octave's existing test implements shared variables by passing them as parameters to a testing function call that it generates dynamically, and that that testing function (with shared variables passed as named parameters) is the immediately enclosing stack frame in which statements in the test block are executed. That's incompatible with the object-managed workspace approach that Testify is using.

apjanke · 2019-04-19T22:50:16Z

isocaps:

One of the "error" tests is getting the wrong error message.

Octave's:

>> test isocaps verbose
>>>>> /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isocaps.m
[...]
***** error <X must match the size of V>
 x = 1:2:24;
 fvc = isocaps (x, y, z, val, iso);
***** error <Y must match the size of V>
 y = -14:6:11;
 fvc = isocaps (x, y, z, val, iso);
***** error <Z must match the size of V>
[...]

Testify's:

*****  testify.internal.BistBlock: error (index=24)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/V must be a non-singleton 3-dimensional matrix/
Code:

 v3 = reshape (1:6*8, [6 1 8]);
 fvc = isocaps (v3, iso);
  -> success=1, msg=
*****  testify.internal.BistBlock: error (index=25)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/X must match the size of V/
Code:

 x = 1:2:24;
 fvc = isocaps (x, y, z, val, iso);
  -> success=1, msg=
*****  testify.internal.BistBlock: error (index=26)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/Y must match the size of V/
Code:

 y = -14:6:11;
 fvc = isocaps (x, y, z, val, iso);
  -> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isocaps: X must match thesize of V'
PASSES 22 out of 23 tests

Now, why would that happen? Is it because we're persisting our shared variables differently?

apjanke · 2019-04-19T22:57:12Z

getframe:

This failure is intermittent. Seems to be related to test ordering. When I get a failure, it complains about something not being implemented for the gnuplot toolkit. Normally, I'm running with the qt toolkit; looks like the toolkit got switched during test execution somewhere?

Yeah: In 5.1.0, after I do a full __run_test_suite2__, my graphics_toolkit is left as gnuplot:

>> have_window_system, graphics_toolkit
ans = 1
ans = gnuplot
>> test2 getframe
PASSES 2 out of 2 tests
Skipped 2 tests due to run-time conditions
>>

But in 4.4.1, the tests aren't properly skipped:

>> version
ans = 4.4.1
>> have_window_system, graphics_toolkit
ans = 1
ans = gnuplot
>> test2 getframe
PASSES 0 out of 1 test
>> test2 getframe -verbose

>>>>> /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
[...]
*****  testify.internal.BistBlock: testif (index=5)
  is_test=1  is_xtest=0
  feature_line=' HAVE_QT_OFFSCREEN; have_window_system ()'
  feature='{QT_OFFSCREEN}'
  runtime_feature_test=' have_window_system ()'
Code:

 hf = figure ("visible", "off");
 unwind_protect
   pos = get (hf, "position");
   assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
  -> success=0, msg=test failed: raised error: getframe: not implemented for gnuplot graphics toolkit
PASSES 0 out of 1 test

Back on 5.1.0, it's showing a different testif condition for that same block:

*****  testify.internal.BistBlock: testif (index=5)
  is_test=1  is_xtest=0
  feature_line=' HAVE_QT_OFFSCREEN; have_window_system () && strcmp ("qt", graphics_toolkit ())'
  feature='{QT_OFFSCREEN}'
  runtime_feature_test=' have_window_system () && strcmp ("qt", graphics_toolkit ())'
Code:

 hf = figure ("visible", "off");
 unwind_protect
   pos = get (hf, "position");
   assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
  -> success=1, msg=----- skipped test (runtime test)

Ah, yes: the test condition actually did change between 4.4.1 and 5.1.0. On 4.4.1, plain test is failing, too:

>> test getframe
***** testif HAVE_QT_OFFSCREEN; have_window_system ()
 hf = figure ("visible", "off");
 unwind_protect
   pos = get (hf, "position");
   assert (size (getframe (hf).cdata)(1:2), pos(4:-1:3));
 unwind_protect_cleanup
   close (hf);
 end_unwind_protect
!!!!! test failed
getframe: not implemented for gnuplot graphics toolkit
>>

apjanke · 2019-04-19T23:02:37Z

isosurface:

Similar test failure to isocaps, with an incorrect size-difference error message:

>> test2 isosurface -verbose
[...]
*****  testify.internal.BistBlock: error (index=25)
  is_test=1  is_xtest=0
  error_id=''
  pattern=/Y must match the size of V/
Code:

 y = -14:6:11;
 fvc = isosurface (x, y, z, val, iso);
  -> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of V/, but got 'isosurface: X must match the size of V'
PASSES 20 out of 21 tests
>>

apjanke · 2019-04-19T23:36:23Z

reducevolume:

Yet another problem with incorrect error messages for input size comparisons:

>> test reducevolume
PASSES 22 out of 22 tests
>> test2 reducevolume -no-fail-fast
----- Failure details: block 19 -----
*****  test block 19: error /X must match the size of data V/
  is_test=1 is_xtest=0
Code:
 x = 1:2:24;
 [nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /X must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 20 -----
*****  test block 20: error /Y must match the size of data V/
  is_test=1 is_xtest=0
Code:
 y = -14:6:11;
 [nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Y must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 21 -----
*****  test block 21: error /Z must match the size of data V/
  is_test=1 is_xtest=0
Code:
 z = linspace (16, 18, 5);
 [nx, ny, nz, nv] = reducevolume (x, y, z, v, [4 3 2]);
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /Z must match the size of data V/, but got 'reducevolume: data mustbe a non-singleton 3-dimensional matrix'
----- Failure details: block 22 -----
*****  test block 22: error /reduction value R is too high/
  is_test=1 is_xtest=0
Code:
 [nv] = reducevolume (v, 5)
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /reduction value R is too high/, but got 'reducevolume: data must be a non-singleton 3-dimensional matrix'
----- Failure details: block 23 -----
*****  test block 23: error /reduction value R is too high/
  is_test=1 is_xtest=0
Code:
 [nv] = reducevolume (v, [4 7 2])
--> success=0, msg=!!!!! Incorrect error raised: expected error message matching /reduction value R is too high/, but got 'reducevolume: data must be a non-singleton 3-dimensional matrix'
PASSES 17 out of 22 tests
>>

apjanke · 2019-04-19T23:37:04Z

copyobj and hgsave are failing under regular Octave test too. Marking them off.

apjanke · 2019-04-20T00:04:53Z

Results running under CLI vanilla Homebrewed Octave 5.1.0:


Summary:

  GNU Octave Version: 5.1.0 (hg id: d05d6eebde10)
  Tests run on angharad.local (macOS) at 19-Apr-2019 20:02:45
  Test execution time: 00:12:21

  PASS                            18810
  FAIL                               35
  XFAIL (reported bug)              247
  SKIP (missing feature)            176
  SKIP (run-time condition)          78

  Failed tests:
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/etc/tests/libinterp/corefcn/cellfun.cc-tst
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/etc/tests/libinterp/corefcn/mappers.cc-tst
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/general/logspace.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/miscellaneous/inputname.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/miscellaneous/tar.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/isocaps.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/isosurface.m
     /usr/local/Cellar/octave/5.1.0/share/octave/5.1.0/m/plot/draw/reducevolume.m

Those failures are more what I'm used to.

Full test results:
testsuite-angharad.local-macOS-20190419_194818.zip

apjanke · 2019-04-20T00:09:15Z

I manually extracted all the failures from the progress output, and it matches the summary results... except all the M-file tests are being run twice!

  <Octave/tests>/libinterp/corefcn/cellfun.cc-tst ............. PASS    122/130 
                                                                FAIL      4
  <Octave/tests>/libinterp/corefcn/mappers.cc-tst ............. PASS    414/421 
                                                                FAIL      1
  <Octave/m>/general/logspace.m ............................... PASS     14/16  
                                                                FAIL      2
  <Octave/m>/miscellaneous/inputname.m ........................ PASS      9/11  
                                                                FAIL      2
  <Octave/m>/miscellaneous/tar.m .............................. PASS      5/6   
                                                                FAIL      1
  <Octave/m>/plot/draw/isocaps.m .............................. PASS     27/29  
                                                                FAIL      2
  <Octave/m>/plot/draw/isosurface.m ........................... PASS     26/30  
                                                                FAIL      3
  <Octave/m>/plot/draw/reducevolume.m ......................... PASS     17/22  
                                                                FAIL      5
  <Octave/m>/general/logspace.m ............................... PASS     14/16  
                                                                FAIL      2
  <Octave/m>/miscellaneous/inputname.m ........................ PASS      9/11  
                                                                FAIL      2
  <Octave/m>/miscellaneous/tar.m .............................. PASS      5/6   
                                                                FAIL      1
  <Octave/m>/plot/draw/isosurface.m ........................... PASS     26/30  
                                                                FAIL      3
  <Octave/m>/plot/draw/reducevolume.m ......................... PASS     17/22  
                                                                FAIL      5

apjanke · 2019-04-20T00:42:54Z

After fixing #78, I'm down to this:

Summary:

  GNU Octave Version: 4.4.1 (hg id: 1f46d371968c + patches)
  Tests run on angharad.local (macOS) at 19-Apr-2019 20:41:12
  Test execution time: 00:05:29

  PASS                            11260
  FAIL                               18
  XFAIL (reported bug)              157
  SKIP (missing feature)             14
  SKIP (run-time condition)          51

  Failed tests:
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/etc/tests/libinterp/corefcn/pr-output.cc-tst
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/image/getframe.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/miscellaneous/inputname.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isocaps.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/isosurface.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/draw/reducevolume.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/copyobj.m
     /Applications/Octave-4.4.1.app/Contents/Resources/usr/Cellar/octave-octave-app_4.4.1_0/4.4.1/share/octave/4.4.1/m/plot/util/hgsave.m

Better. Now I think I'm down to legit individual test failures.

apjanke mentioned this issue Apr 19, 2019

Way too many failures in test suite #60

Closed

apjanke self-assigned this Apr 19, 2019

apjanke added the bug Something isn't working label Apr 19, 2019

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Test failures not in octave's __run_test_suite__ #72

Test failures not in octave's __run_test_suite__ #72

apjanke commented Apr 19, 2019 •

edited

Loading

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 20, 2019

apjanke commented Apr 20, 2019

apjanke commented Apr 20, 2019

Test failures not in octave's __run_test_suite__ #72

Test failures not in octave's __run_test_suite__ #72

Comments

apjanke commented Apr 19, 2019 • edited Loading

Files with failed tests

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 19, 2019

apjanke commented Apr 20, 2019

apjanke commented Apr 20, 2019

apjanke commented Apr 20, 2019

apjanke commented Apr 19, 2019 •

edited

Loading