Intermittent System.TypeLoadExceptions and System.Security.VerificationExceptions

[mrkaarel]

After upgrading our solution from .NET Core 2.1 to .NET 5 we have seen intermittent (about one failed run in every 6-7 build server test runs) System.TypeLoadExceptions and System.Security.VerificationExceptions with the following messages:

System.Security.VerificationException example:

System.Security.VerificationException : Method Castle.Proxies.ICustomerFinderProxy.Find: type argument 'Domain.Entities.Customers.Customer' violates the constraint of type parameter 'T'.
   at Web.Models.Orders.AddOrUpdateOrderValidator.ValidateOrder(Int32 customerId, OrderViewModelBase model, String customerType) in /src/Web/Models/Orders/AddOrUpdateOrderValidator.cs:line 75
   at Web.Tests.Models.Orders.AddOrUpdateOrderValidatorTests.Validate_returns_error_when_package_is_currently_not_sold() in /src/Web.Tests/Models/Orders/AddOrUpdateOrderValidatorTests.cs:line 361

System.TypeLoadException example:

System.TypeLoadException : GenericArguments[0], 'T', on 'Castle.Proxies.Invocations.IUserAccountFinder_Find_2[T]' violates the constraint of type parameter 'T'.
   at Castle.Proxies.IUserAccountFinderProxy.Find[T](Guid userAccountId, Int32 customerId)
   at Web.Areas.Account.Models.Relatives.StudentsRelativesValidator.CanChangeRelatives(Guid studentAccountId, IPrincipal currentPrincipal) in /src/Web/Areas/Account/Models/Relatives/StudentsRelativesValidator.cs:line 27
   at Web.Tests.Areas.Account.Models.Relatives.StudentsRelativesValidatorTests.Validate_Returns_False_When_User_Has_UserRelativesCanChange_Privilege_But_No_User_Account_With_Given_Id_Exists_In_Current_Users_School() in /src/Web.Tests/Areas/Account/Models/Relatives/StudentsRelativesValidatorTests.cs:line 75

The common part about all these exceptions is the error about violating the constraint of type parameter and that they are all related to mocks.

There are around 350 tests in total in which these exceptions have occurred. In a typical build there is either none of those exceptions or a relatively large amount (100-200), almost never just a few.

I have no reason to suspect a bug in Moq and I am just posting it here because I have run out of ideas what might be causing it and was hoping to find out if anyone else has seen such behaviour in their projects or based on these symptoms has a hunch what could be wrong.

[stakx]

Hi @mrkaarel, thanks for reporting. We do need some kind of minimal repro code that exhibits this problem. Can you at the very least provide a full definition of one of the type/method definitions that the error message is about?

Also, what platform do your CI builds run on (.NET TFM / OS / CPU architecture)?

[mrkaarel]

Thank you for a quick reply @stakx.

So far we've been unsucessful isolating the issue to provide a minimal repro, the main reason being that it has only very rarely occurred on development machines.

Here is a type declaration for ICustomerFinder, mentioned in Security.VerificationException example above.

public interface ICustomerFinder
{
    Customer Find(int id);
    T Find<T>(int id) where T : Customer;
}

We are running our builds inside Docker container based on mcr.microsoft.com/dotnet/sdk:5.0-focal. The build agents are running 64-bit Windows Enterprise and 64-bit Ubuntu (20.04.2), we are getting these errors on both. We are targeting netcoreapp5.0. And maybe also worth noting, we are running tests in parallel.

I understand if it is too vague, feel free to close this issue. I just put it out there in hope someone somewhere had already seen the same behaviour and found a solution.

[stakx]

And you're saying the problem only started appearing after you upgraded to net5.0? Makes me wonder if it would vanish if you downgraded to netcoreapp3.x without changing any NuGet package versions. If so this could indicate a problem in the runtime — especially if there is absolutely nothing exotic about your types, e.g. them being internal and you using [InternalsVisibleTo], them being records (we haven't really covered them yet in our unit test suite, and neither has DynamicProxy), them being readonly/ref structs, using those fancy new generic constraints like unmanaged, etc. If it's just plain public classes, there's no reason why you should see such low-level errors.

If you do manage to create a small repro solution — even if the error doesn't occur on every single run – then please post it.

Let's leave this open for a while to see if anyone else is experiencing similar problems.

[stakx]

I've searched for this issue and it seems you're not the only one who experienced this. Possibly related: dotnet/runtime#45600.

[mrkaarel]

Thank you for looking into it.

I've searched for this issue and it seems you're not the only one who experienced this. Possibly related: dotnet/runtime#45600.

I have looked at the referenced dotnet/runtime issue before, but it seems different in a way that it has a specific repro that always fails whereas our problem only occurs sometimes (some race condition, or something else related to order things are executed?).

And you're saying the problem only started appearing after you upgraded to net5.0?

Yes, for us the issue started after the upgrade. Although during that upgrade there were many other notable changes as well – moving our builds + test runs to Docker, numerous NuGet package upgrades etc. As to exotic types being used – not really. Most of the code changes during that upgrade were related to infrastructure, "business" code, including the types causing the exceptions were in general not changed.

I'll give it another try to isolate the issue.

[stakx]

@mrkaarel, any news on this?

[mrkaarel]

@stakx Unfortunately no.

After digging some more we have found that:

• these exceptions are not caused by running multiple test assemblies in parallel – they can also occur when just running tests from one test project
• we did not get these exceptions when running just a handful of those failing tests (about 100 tests out of the total of ~5500 in the test project)
• we also did not get the exceptions when running the tests from this project non-parallel

As we cannot get these exceptions to occur in IDE, I think one approach might be to log out some more details at the moment we catch one of those exceptions – I just can't think where to look.

I guess, for now we will just start running this project's tests non-parallel until we get some new ideas what to try.

[BrunoJuchli]

This might be related to: #246
Which itself is an issue in Castle.DynamicProxy, the corresponding issue: castleproject/Core#193

[stakx]

@BrunoJuchli, I cannot exclude that as a possibility; however, given that the issue reported here only surfaced after migrating to net5.0, it seems unlikely. Seems more probable to me that it's a new problem in .NET Core ≥ 5.

(Or at least I hope so. That other DynamicProxy issue was a beast. I'd hate to admit how much time I wasted trying to find its cause. 😆)

[stakx]

@mrkaarel shall we close this issue? There's nothing we can do without a repro, really. That other runtime issue I linked to above has apparently been fixed, I'm wondering if the reported problem might go away after upgrading to the latest version of the .NET runtime?

[mrkaarel]

@stakx Yes, thank you for the patience, let's close this.

For us the workaround was just to start running this project's tests non-parallel. I'll give it a quick try if anything has changed with the latest runtime.