Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ADAM-536] Add support for scala 2.11 #547

Closed
wants to merge 3 commits into from
Closed

[ADAM-536] Add support for scala 2.11 #547

wants to merge 3 commits into from

Conversation

fnothaft
Copy link
Member

Resolves #536. Doesn't quite work yet; still working to resolve this issue with the scala 2.11 build:

[ERROR] /Users/fnothaft/IdeaProjects/adam/adam-core/src/main/scala/org/bdgenomics/adam/rdd/ShuffleRegionJoin.scala:94: diverging implicit expansion for type ((org.bdgenomics.adam.models.ReferenceRegion, Int)) => Comparable[(org.bdgenomics.adam.models.ReferenceRegion, Int)]
starting with method orderingToOrdered in object Ordered
[ERROR]       keyedLeft.repartitionAndSortWithinPartitions(ManualRegionPartitioner(bins.value.numBins))
[ERROR]       ^
[ERROR] /Users/fnothaft/IdeaProjects/adam/adam-core/src/main/scala/org/bdgenomics/adam/rdd/ShuffleRegionJoin.scala:94: value repartitionAndSortWithinPartitions is not a member of org.apache.spark.rdd.RDD[((org.bdgenomics.adam.models.ReferenceRegion, Int), T)]
[ERROR]       keyedLeft.repartitionAndSortWithinPartitions(ManualRegionPartitioner(bins.value.numBins))
[ERROR]                 ^
[ERROR] /Users/fnothaft/IdeaProjects/adam/adam-core/src/main/scala/org/bdgenomics/adam/rdd/ShuffleRegionJoin.scala:96: diverging implicit expansion for type ((org.bdgenomics.adam.models.ReferenceRegion, Int)) => Comparable[(org.bdgenomics.adam.models.ReferenceRegion, Int)]
starting with method orderingToOrdered in object Ordered
[ERROR]       keyedRight.repartitionAndSortWithinPartitions(ManualRegionPartitioner(bins.value.numBins))
[ERROR]       ^
[ERROR] /Users/fnothaft/IdeaProjects/adam/adam-core/src/main/scala/org/bdgenomics/adam/rdd/ShuffleRegionJoin.scala:96: value repartitionAndSortWithinPartitions is not a member of org.apache.spark.rdd.RDD[((org.bdgenomics.adam.models.ReferenceRegion, Int), U)]
[ERROR]       keyedRight.repartitionAndSortWithinPartitions(ManualRegionPartitioner(bins.value.numBins))

@laserson I was trying to figure out where the ordering for (ReferenceRegion, Int) was defined/is coming in from, but I wasn't able to grok that from the ShuffleRegionJoin code. Do you know where that is coming from/would you be able to take a look?

@fnothaft
Copy link
Member Author

Also, it looks like this problem is related to a regression in Scala 2.11, but I believe we should be able to WAR the regression.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/534/

Build result: FAILURE

GitHub pull request #547 of commit 8964be2 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/547/merge^{commit} # timeout=10Checking out Revision 4e79348 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 4e79348 > git rev-list b409bd0004a00c2dac712b5c567963b3e013c8e3 # timeout=10Triggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILUREADAM-prb ? 2.3.0,centos completed with result FAILURE
Test FAILed.

@laserson
Copy link
Contributor

Just submitted a PR against your branch that fixes the Ordering problem. Mo' problems to solve, though...

@laserson
Copy link
Contributor

(Also, I'm not sure where the Ordering was coming from to begin with, at it appears Tuples don't have default Orderings.)

@fnothaft
Copy link
Member Author

Thanks @laserson! I was confused as well...

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/536/

Build result: ABORTED

GitHub pull request #547 of commit 0fc67a6 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/547/merge^{commit} # timeout=10Checking out Revision 39b5236 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 39b5236 > git rev-list b409bd0004a00c2dac712b5c567963b3e013c8e3 # timeout=10Triggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosADAM-prb ? 2.2.0,centos completed with result ABORTEDADAM-prb ? 1.0.4,centos completed with result FAILUREADAM-prb ? 2.3.0,centos completed with result FAILURE
Test FAILed.

@fnothaft
Copy link
Member Author

Jenkins, retest this please.

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/541/

Build result: FAILURE

GitHub pull request #547 of commit 0fc67a6 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/547/merge^{commit} # timeout=10Checking out Revision 39b5236 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 39b5236 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILUREADAM-prb ? 2.3.0,centos completed with result FAILURE
Test FAILed.

@massie
Copy link
Member

massie commented Jan 20, 2015

Jenkins, test this please.

@massie
Copy link
Member

massie commented Jan 20, 2015

This approach looks good to me. I'd like to hear from @heuermh too since he's strong with the maven fu.

massie
massie reviewed Jan 20, 2015
View reviewed changes
pom.xml
@@ -483,6 +487,13 @@

<profiles>
<profile>
<id>scala-2.11</id>
<properties>
<scala.version>2.11.2</scala.version>
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We are up to 2.11.5 now and brew has 2.11.4. We might want to bump this up before we commit it?

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/542/

Build result: FAILURE

GitHub pull request #547 of commit 0fc67a6 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/547/merge^{commit} # timeout=10Checking out Revision 39b5236 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 39b5236 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILUREADAM-prb ? 2.3.0,centos completed with result FAILURE
Test FAILed.

@massie
Copy link
Member

massie commented Jan 20, 2015

It looks like there are a few places where 2.10 is still hard-coded, e.g.

$ find . -name "pom.xml" | xargs grep "2.10"
./adam-cli/pom.xml:      <artifactId>scalate-core_2.10</artifactId>
./adam-cli/pom.xml:      <artifactId>scalatra-json_2.10</artifactId>
./adam-cli/pom.xml:      <artifactId>scalatra_2.10</artifactId>
./pom.xml:    <scala.version>2.10.4</scala.version>
./pom.xml:    <scala.artifact.suffix>2.10</scala.artifact.suffix>
./pom.xml:        <artifactId>scalate-core_2.10</artifactId>
./pom.xml:        <artifactId>scalatra-json_2.10</artifactId>
./pom.xml:        <artifactId>scalatra_2.10</artifactId>

we should probably update this to use $scala.artifact.suffix

@massie
Copy link
Member

massie commented Jan 20, 2015

This doesn't build on my Mac, e.g.

$ mvn -P scala-2.11 clean package

Throws the following error:

[ERROR] /Users/matt/workspace/berkeley/adam/adam-core/src/main/scala/org/bdgenomics/adam/models/VariantContext.scala:84: in class VariantContext, multiple overloaded alternatives of constructor VariantContext define default arguments.
[ERROR] class VariantContext(
[ERROR]       ^
[WARNING] 5 warnings found
[ERROR] one error found

Things build fine when I remove the scala-2.11 profile.

Are others seeing the same issue?

@fnothaft
Copy link
Member Author

Let me take a look; this built on my end when I opened the PR, not sure if there's been a change since then that'd break it...

@fnothaft
Copy link
Member Author

Er, duh! The 2.10 would build; 2.11 wouldn't build because the error in the original PR post wasn't fixed yet. I'll take a look at this locally now.

@fnothaft
Copy link
Member Author

This should work for Scala 2.11 now. Can someone test on their own machine?

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/546/

Build result: FAILURE

GitHub pull request #547 of commit 5cc2b32 automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/547/merge^{commit} # timeout=10Checking out Revision 0fe3c16 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 0fe3c16 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILUREADAM-prb ? 2.3.0,centos completed with result FAILURE
Test FAILed.

@heuermh
Copy link
Member

heuermh commented Jan 22, 2015

Also fails for me, see https://gist.github.com/heuermh/14f7b33def990f4eb7a2

@fnothaft
Copy link
Member Author

Ah! Looking at this again...

@AmplabJenkins
Copy link

Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/ADAM-prb/550/

Build result: FAILURE

GitHub pull request #547 of commit feedc7c automatically merged.[EnvInject] - Loading node environment variables.Building remotely on amp-jenkins-worker-05 (centos) in workspace /home/jenkins/workspace/ADAM-prb > git rev-parse --is-inside-work-tree # timeout=10Fetching changes from the remote Git repository > git config remote.origin.url https://github.com/bigdatagenomics/adam.git # timeout=10Fetching upstream changes from https://github.com/bigdatagenomics/adam.git > git --version # timeout=10 > git fetch --tags --progress https://github.com/bigdatagenomics/adam.git +refs/pull/:refs/remotes/origin/pr/ > git rev-parse origin/pr/547/merge^{commit} # timeout=10Checking out Revision 2e33854 (detached) > git config core.sparsecheckout # timeout=10 > git checkout -f 2e33854 > git rev-list d8e02d9 # timeout=10Triggering ADAM-prb ? 2.2.0,centosTriggering ADAM-prb ? 1.0.4,centosTriggering ADAM-prb ? 2.3.0,centosADAM-prb ? 2.2.0,centos completed with result FAILUREADAM-prb ? 1.0.4,centos completed with result FAILUREADAM-prb ? 2.3.0,centos completed with result FAILURE
Test FAILed.

@fnothaft
Copy link
Member Author

Closing this. Will open a new PR tomorrow that builds on bigdatagenomics/utils#36.

@fnothaft fnothaft closed this Apr 30, 2015
@fnothaft fnothaft deleted the split-compile branch April 30, 2015 01:10
@fnothaft fnothaft mentioned this pull request Mar 30, 2016
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

scala 2.11
5 participants
@fnothaft @AmplabJenkins @laserson @massie @heuermh