Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

ucar.nc2.filter.UnknownFilterException: Unknown filter - v5.6.0 #1382

Open
1 task done
jacobwentzib opened this issue Aug 26, 2024 · 10 comments
Open
1 task done

ucar.nc2.filter.UnknownFilterException: Unknown filter - v5.6.0 #1382

jacobwentzib opened this issue Aug 26, 2024 · 10 comments
Labels
bug Something isn't working

Comments

@jacobwentzib
Copy link

jacobwentzib commented Aug 26, 2024

Versions impacted by the bug

5.6.0 - Latest

Environment:
UCAR Library Version: cdm-core 5.6.0
Java Version: 21

What went wrong?

Recently upgraded to 5.6.0 and see the following exception being thrown where the ServiceLoader is unable to load Filters properly.

throw new UnknownFilterException(id);

Once we downgrade back to 5.5.3 everything works properly.

Has there been any changes to classpath or anything else that could be causes this issue? We are seeing this isolated to jar executions but IDE development with target classes runs properly.

We have created a custom ServiceLoader within our application and can see the resources being found.

Relevant stack trace

Being invoked from:
Variable.class:

    public Array read(Section section) throws IOException, InvalidRangeException {
        return section == null ? this._read() : this._read(Section.fill(section, this.shape));
    }

Error throw from above link:
java.io.IOException: ucar.nc2.filter.UnknownFilterException: Unknown filter: no filter found with id 2

Relevant log messages

No response

If you have an example file that you can share, please attach it to this issue.

If so, may we include it in our test datasets to help ensure the bug does not return once fixed?
Note: the test datasets are publicly accessible without restriction.

Yes

Code of Conduct

  • I agree to follow the UCAR/Unidata Code of Conduct
@jacobwentzib jacobwentzib added the bug Something isn't working label Aug 26, 2024
@tdrwenski
Copy link
Contributor

  • Can you add the full error stack trace?
  • Are any FilterProviders getting loaded (yours or NetCDF-java's)? It looks like the error is about "id 2" which should be the Shuffle filter.

@jacobwentzib
Copy link
Author

NetCDF-java's FilterProviders.

Here is the full trace:
java.io.IOException: ucar.nc2.filter.UnknownFilterException: Unknown filter: no filter found with id 2
at ucar.nc2.internal.iosp.hdf5.H5tiledLayoutBB.(H5tiledLayoutBB.java:90) ~[cdm-core-5.6.0.jar!/:5.6.0]
at ucar.nc2.internal.iosp.hdf5.H5iospNew.readData(H5iospNew.java:226) ~[cdm-core-5.6.0.jar!/:5.6.0]
at ucar.nc2.internal.iosp.hdf5.H5iospNew.readData(H5iospNew.java:204) ~[cdm-core-5.6.0.jar!/:5.6.0]
at ucar.nc2.NetcdfFile.readData(NetcdfFile.java:2122) ~[cdm-core-5.6.0.jar!/:5.6.0]
at ucar.nc2.Variable.reallyRead(Variable.java:797) ~[cdm-core-5.6.0.jar!/:5.6.0]
at ucar.nc2.Variable._read(Variable.java:736) ~[cdm-core-5.6.0.jar!/:5.6.0]
at ucar.nc2.Variable._read(Variable.java:754) ~[cdm-core-5.6.0.jar!/:5.6.0]
at ucar.nc2.Variable.read(Variable.java:600) ~[cdm-core-5.6.0.jar!/:5.6.0]
at .GoesrDecoder.extractDataFromVariable(GoesrDecoder.java:376) ~[goesr-0.0.0.jar!/:0.0.0]
at .GoesrDecoder.extractDataAsync(GoesrDecoder.java:320) ~[goesr-0.0.0.jar!/:0.0.0]
at .GoesrDecoder.decode(GoesrDecoder.java:145) ~[goesr-0.0.0.jar!/:0.0.0]
at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103) ~[na:na]
at java.base/java.lang.reflect.Method.invoke(Method.java:580) ~[na:na]
at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:354) ~[spring-aop-6.1.11.jar!/:6.1.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:196) ~[spring-aop-6.1.11.jar!/:6.1.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163) ~[spring-aop-6.1.11.jar!/:6.1.11]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.11.jar!/:6.1.11]
at org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:379) ~[spring-tx-6.1.11.jar!/:6.1.11]
at org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:119) ~[spring-tx-6.1.11.jar!/:6.1.11]
at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:184) ~[spring-aop-6.1.11.jar!/:6.1.11]
at org.springframework.aop.framework.CglibAopProxy$CglibMethodInvocation.proceed(CglibAopProxy.java:768) ~[spring-aop-6.1.11.jar!/:6.1.11]
at org.springframework.aop.interceptor.AsyncExecutionInterceptor.lambda$invoke$0(AsyncExecutionInterceptor.java:113) ~[spring-aop-6.1.11.jar!/:6.1.11]
at org.springframework.util.concurrent.FutureUtils.lambda$toSupplier$0(FutureUtils.java:74) ~[spring-core-6.1.11.jar!/:6.1.11]
at java.base/java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1768) ~[na:na]
at java.base/java.lang.VirtualThread.run(VirtualThread.java:311) ~[na:na]
Caused by: ucar.nc2.filter.UnknownFilterException: Unknown filter: no filter found with id 2
at ucar.nc2.filter.Filters.getFilter(Filters.java:80) ~[cdm-core-5.6.0.jar!/:5.6.0]
at ucar.nc2.internal.iosp.hdf5.H5tiledLayoutBB.(H5tiledLayoutBB.java:88) ~[cdm-core-5.6.0.jar!/:5.6.0]
... 24 common frames omitted


@tdrwenski
Copy link
Contributor

Thanks for sending that. I am not aware of any changes between version 5.5.3 and 5.6.0 that may have caused this issue. And I am not yet able to reproduce this behavior. Just to be sure I am testing the same situation:

  • is it correct you are using the artifact cdm-core-5.6.0.jar?
  • Can you check that you have the netcdf-java filters in that jar? By doing e.g. jar tf cdm-core-5.6.0.jar | grep META-INF/services/ucar.nc2.filter.FilterProvider
  • is your custom Filter implemented similar to the example here and registered via a META-INF/services/ucar.nc2.filter.FilterProvider file?

@jacobwentzib
Copy link
Author

  • Correct we are utilizing: cdm-core-5.6.0.jar
  • Directory is Empty in META-INF/services for the dependency.
  • We do not have any custom Filter we are just trying to leverage existing.

@tdrwenski
Copy link
Contributor

Thanks for checking that. I don't know how it's possible that your cdm-core-5.6.0.jar is missing those filters. Are you getting your artifacts from our artifacts server: https://artifacts.unidata.ucar.edu/repository/unidata-all/? Can you try replacing that dependency? For instance, by downloading it manually from here?

@jacobwentzib
Copy link
Author

the pom.xml:

  <properties>
    <edu.ucar.version>5.6.0</edu.ucar.version>
  </properties>
  
  ...
    <dependency>
      <groupId>edu.ucar</groupId>
      <artifactId>cdm-core</artifactId>
      <version>${edu.ucar.version}</version>
      <exclusions>
        <exclusion>
          <groupId>commons-logging</groupId>
          <artifactId>commons-logging</artifactId>
        </exclusion>
      </exclusions>
    </dependency>
    ... 
      <repositories>
        <repository>
          <id>unidata-all</id>
          <name>Unidata All</name>
          <url>https://artifacts.unidata.ucar.edu/repository/unidata-all/</url>
        </repository>
        <repository>
          <id>ucar-unidata</id>
          <url>https://repo.osgeo.org/repository/release/</url>
        </repository>
      <repositories>

Expanding the jar I see the services in the META-INF of the cdm-core jar.

It seems like the SPI is not working when bundled. As mentioned previously this works when ran in IDE with all jars on class path but when we create our jar it gets these issues where it cannot find the Filters.

We do not want to shade our jar to navigate around this.

@jacobwentzib
Copy link
Author

Bump on this.

@jacobwentzib
Copy link
Author

Following up on this? Is there anything else we can do help get around this issue?

@tdrwenski
Copy link
Contributor

Hi @jacobwentzib, sorry for the delay getting back to this.

I have looked at the cdm-core-5.5.3.jar and cdm-core-5.6.0.jar and see no differences with respect to the filter providers or class path. I have also tried using a jar execution of cdm-core-5.6.0.jar and am unable to reproduce your issue, the filters do get loaded.

Can you confirm there are no other environmental differences between your testing of 5.5.3 and 5.6.0? Can you also test with our 5.6.1-SNAPSHOT version?

@tdrwenski
Copy link
Contributor

Not ideal, but maybe as a workaround, you can try copying this into your own src/main/resources/META-INF/services/

@github-staff github-staff deleted a comment from Lxx-c Oct 23, 2024
@github-staff github-staff deleted a comment from Lxx-c Oct 23, 2024
@github-staff github-staff deleted a comment from Lxx-c Oct 23, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

3 participants
@tdrwenski @jacobwentzib and others