Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use cabal-helper 1.0 #26

Merged
merged 33 commits into from
Nov 5, 2019
Merged

Conversation

fendor
Copy link
Collaborator

@fendor fendor commented Oct 3, 2019

Introduces cabal-helper cradle.

  • Cradle discovers projects instead of just assuming current working directory as root
  • Does produce diagnostics
  • HIE uses ghc --version to discover the projec ghc version. However, this is wrong for stack-based projects. Ghc Version is correctly discovered based on project type.

cc, @bubba

@fendor fendor requested a review from lukel97 October 3, 2019 18:38
@fendor fendor force-pushed the cabal-helper-helper branch from ea2f97b to cf23e26 Compare October 3, 2019 20:55
@lukel97
Copy link
Collaborator

lukel97 commented Oct 4, 2019

Were there any changes needed inside the cabal-helper submodule after updating it to 1.0?

@fendor
Copy link
Collaborator Author

fendor commented Oct 4, 2019

@bubba no, this is just the submodules and the cradle type of cabal-helper-helper is taken from https://github.com/wz1000/cabal-helper-helper/blob/master/Main.hs

@jneira
Copy link

jneira commented Oct 8, 2019

I had to add

- clock-0.7.2@sha256:c5c2514aead260101c7528d10bf0f41d9d1d7b919136e2dff77a57ec16740bd2,3820

to stack.yaml to build it with stack

@jneira
Copy link

jneira commented Oct 8, 2019

Maybe you are already aware but just in case: stack build (after adding the previour dep) fails when building hare:

D:\ws\haskell\haskell-ide-engine>stack build
HaRe              > build (lib + exe)
HaRe              > Preprocessing library for HaRe-0.8.4.1..
HaRe              > Building library for HaRe-0.8.4.1..
HaRe              > Preprocessing executable 'ghc-hare' for HaRe-0.8.4.1..
HaRe              > Building executable 'ghc-hare' for HaRe-0.8.4.1..
HaRe              >
HaRe              > <no location info>: warning: [-Wmissing-home-modules]
HaRe              >     These modules are needed for compilation but not listed
in your .cabal file's other-modules:
HaRe              >         Paths_HaRe
HaRe              > [2 of 2] Compiling Main [Haskell.Ide.Engine.PluginApi change
d]
HaRe              >
HaRe              > D:\ws\haskell\haskell-ide-engine\submodules\HaRe\app\MainHaR
e.hs:369:17: error:
HaRe              >     Not in scope: data constructor `HIE.BiosOptions'
HaRe              >     Module `Haskell.Ide.Engine.PluginApi' does not export `B
iosOptions'.
HaRe              >     |
HaRe              > 369 | globalArgSpec = HIE.BiosOptions
HaRe              >     |                 ^^^^^^^^^^^^^^^
HaRe              >
HaRe              > D:\ws\haskell\haskell-ide-engine\submodules\HaRe\app\MainHaR
e.hs:376:16: error:
HaRe              >     Not in scope: data constructor `HIE.BlWarning'
HaRe              >     Module `Haskell.Ide.Engine.PluginApi' does not export `B
lWarning'.
HaRe              >     |
HaRe              > 376 |       <*> pure HIE.BlWarning
HaRe              >     |                ^^^^^^^^^^^^^

--  While building package HaRe-0.8.4.1 using:
      D:\sr\setup-exe-cache\x86_64-windows\Cabal-simple_Z6RU0evB_2.4.0.1_ghc-8.6
.5.exe --builddir=.stack-work\dist\e626a42b build lib:HaRe exe:ghc-hare --ghc-op
tions ""
    Process exited with code: ExitFailure 1
Progress 1/2

Install it with cabal install works... maybe it is using its own hie submodule instead the main one?

@fendor fendor force-pushed the cabal-helper-helper branch 5 times, most recently from b7c6e2f to ebb5b44 Compare October 14, 2019 16:20
@jneira
Copy link

jneira commented Oct 15, 2019

hi! trying to build it with cabal or stack in windows i've got an error in the solver due to a too restrictive lower bound on Win32

cabal.exe: Could not resolve dependencies:
[__0] trying: HaRe-0.8.4.1 (user goal)
[__1] trying: ghc-8.6.5/installed-8.6... (dependency of HaRe)
[__2] trying: Win32-2.6.1.0/installed-2.6... (dependency of ghc)
[__3] next goal: cabal-helper (user goal)
[__3] rejecting: cabal-helper-1.0.0.0 (conflict: ghc =>
Win32==2.6.1.0/installed-2.6..., cabal-helper => Win32<2.9 && >=2.8.3.0)
[__3] rejecting: cabal-helper-0.8.2.0, cabal-helper-0.8.1.2,
cabal-helper-0.8.0.2, cabal-helper-0.8.0.1, cabal-helper-0.8.0.0,
cabal-helper-0.7.3.0, cabal-helper-0.7.2.0, cabal-helper-0.7.1.0,
cabal-helper-0.6.3.1, cabal-helper-0.6.3.0, cabal-helper-0.6.2.0,
cabal-helper-0.6.1.0, cabal-helper-0.6.0.0, cabal-helper-0.5.3.0,
cabal-helper-0.5.1.0, cabal-helper-0.5.0.0, cabal-helper-0.4.0.0,
cabal-helper-0.3.9.0, cabal-helper-0.3.8.0, cabal-helper-0.3.7.0,
cabal-helper-0.3.6.0, cabal-helper-0.3.5.0, cabal-helper-0.3.4.0,
cabal-helper-0.3.3.0, cabal-helper-0.3.2.1, cabal-helper-0.3.2.0,
cabal-helper-0.3.1.0, cabal-helper-0.3.0.0, cabal-helper-0.2.0.0,
cabal-helper-0.1.0.1, cabal-helper-0.1.0.0, cabal-helper-0.8.1.1,
cabal-helper-0.8.1.0, cabal-helper-0.7.0.1, cabal-helper-0.5.2.0 (constraint
[__3] fail (backjumping, conflict set: cabal-helper, ghc)
After searching the rest of the dependency tree exhaustively, these were the
goals I've had most trouble fulfilling: cabal-helper, Win32, HaRe, ghc
Try running with --minimize-conflict-set to improve the error message.

i had to relax the lower bound of cabal-helper to the version used by the older ghc (8.2.2) used by hie:

if os(windows)
    build-depends:     Win32            < 2.9  && >= 2.5.4.1

@jneira
Copy link

jneira commented Oct 15, 2019

Opened a pr to relax the lower bound upstream DanielG/cabal-helper#81

@jneira
Copy link

jneira commented Oct 15, 2019

When testing with a simple cabal project, cabal-helper is not able to install a private copy of Cabal-3.0.0.0 (my cabal version in $PATH), opened an issue upstream: DanielG/cabal-helper#82

NOTE: It happened due a cabal 3.0.0.0 executable version built from source without the patch that fixed that behaviour

@jneira
Copy link

jneira commented Oct 15, 2019

Changin my cabal in $PATH to 2.4.1.0 i've got hoogle docs on hover in vscode but:

  • It shows an error when open a file: lintCmd: no access to the persisted file.
  • It doesn't show quick fixes for unknown variables on hover (even if the function is in a qualified module and the error message is aware of that)
    EDIT: i have the same issue with master so it should be caused by another reason

@fendor fendor changed the title WIP: Use cabal-helper 1.0 Use cabal-helper 1.0 Oct 15, 2019
@fendor
Copy link
Collaborator Author

fendor commented Oct 16, 2019

Needs review and a bit more Documentation.
Also, the builds need to be fixed.

@fendor fendor force-pushed the cabal-helper-helper branch from 2279cf2 to 8c3ff68 Compare October 16, 2019 11:32
-- | Given a FilePath, find the Cradle the FilePath belongs to.
--
-- TODO: document how and why this works.
Copy link

@jneira jneira Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah, i am trying to know where the project build is triggered with no luck 😄
See #26 (comment)

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The line packages <- runQuery projectPackages env initialises the project contexts, e.g. configures projects enough such that we know the different packages in a project.
It produces the output one can see from HieWrapper.hs. However, afaik, this does not actually compile anything, I think it is only the configure step.
Actual compilation of the different units, e.g. library, executable happens on cradle basis: unitInfos_ <- mapM (\unit -> runQuery (unitInfo unit) env) units
Which should not compile the unit but download all dependencies and so, such that we know the targets in that unit.

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is possible that cabal-helper is running the cabal build --dry-run shown in the logs?

Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But hie is passing the unitId to cabal-helper and it can be the cause of errors with components with the same name, maybe we should prefix it with the package name (not sure what field would give us it), to avoid ambiguities

Copy link
Collaborator Author

@fendor fendor Oct 18, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is possible that cabal-helper is running the cabal build --dry-run shown in the logs?

It is, but I dont know to be honest. I am just using the High-Level API.

But hie is passing the unitId to cabal-helper and it can be the cause of errors with components with the same name, maybe we should prefix it with the package name (not sure what field would give us it), to avoid ambiguities

We are simply passing Units that have been obtained from a Package. Unless I am overlooking something, I feel like this is rather a minor bug of c-h.

Thank you for thorough analysis, investigative questions and constructive feedback!

@jneira
Copy link

jneira commented Oct 18, 2019

Ghc Version is correctly discovered based on project type.

I think the actual version could work in most cases, but for cabal based projects, could we get the compiler configured for build the project from cabal configuration (with-compiler field in cabal.project or in the setup-config(via cabal-helper compiler-id?) when cabal configure -w whatever) ?

@jneira
Copy link

jneira commented Oct 18, 2019

@fendor it seems the projects builds are triggered by cabal-helper (as a lib):

https://github.com/DanielG/cabal-helper/blob/447814db7ecda25afa13a7a699a72c5223649d98/lib/Distribution/Helper.hs#L446-L457

before calling cabal-helper (the executable)

So this seems to be the origen of the errors when components from different subpackages have the same name (like dhall)
Not sure 100% though, i'll try to reproduce it using cabal-helper directly and open a issue upstream if necessary

@jneira
Copy link

jneira commented Oct 21, 2019

Hi! haskell-lsp-0.17 is already in master, and it has the fix to get code actions in vscode again.
I've merged it with this branch: https://github.com/fendor/haskell-ide-engine/compare/cabal-helper-helper...jneira:chh-haskell-lsp-0.17?expand=1
@fendor are you interested in merge or did you already did that?

@fendor
Copy link
Collaborator Author

fendor commented Oct 21, 2019

@jneira I think you can open a PR straight to hie-bios. This pr will just be rebased.

@jneira
Copy link

jneira commented Oct 30, 2019

Not sure if it is expected but if i open vscode in a simple project with a cabal.project within a file with compile errors, the file is not loaded and there is no response on hover or other actions.

The vscode output shows:

cd d:\ws\haskell\stack-test; cabal v2-build --with-ghc=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\ghc.exe --with-ghc-pkg=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\ghc-pkg.exe --with-haddock=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\haddock.exe --project-file=d:\ws\haskell\stack-test\cabal.project --builddir=d:\ws\haskell\stack-test\dist-newstyle --dry-run all
Build profile: -w ghc-8.6.5 -O1
In order, the following would be built (use -v for more details):
 - stack-test-0.1.0.0 (lib) (configuration changed)
 - stack-test-0.1.0.0 (exe:stack-test-exe) (configuration changed)
Using hie version: Version 0.13.0.0, Git revision 9ceec1ebad2733d4193c860e3e6f4fe3fc1b158a (dirty) (3292 commits) x86_64 ghc-8.6.5
cd d:\ws\haskell\stack-test; cabal v2-build --with-ghc=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\ghc.exe --with-ghc-pkg=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\ghc-pkg.exe --with-haddock=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\haddock.exe --project-file=d:\ws\haskell\stack-test\cabal.project --builddir=d:\ws\haskell\stack-test\dist-newstyle --dry-run all
Build profile: -w ghc-8.6.5 -O1
In order, the following would be built (use -v for more details):
 - stack-test-0.1.0.0 (lib) (configuration changed)
 - stack-test-0.1.0.0 (exe:stack-test-exe) (configuration changed)
Using hoogle db at: C:\Users\user\AppData\Roaming\hoogle\default-haskell-5.0.17.hoo
cd D:\ws\haskell\stack-test; cabal v2-build --with-ghc=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\ghc.exe --with-ghc-pkg=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\ghc-pkg.exe --with-haddock=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\haddock.exe --project-file=D:\ws\haskell\stack-test\cabal.project --builddir=D:\ws\haskell\stack-test\dist-newstyle --dry-run all
Build profile: -w ghc-8.6.5 -O1
In order, the following would be built (use -v for more details):
 - stack-test-0.1.0.0 (lib) (configuration changed)
 - stack-test-0.1.0.0 (exe:stack-test-exe) (configuration changed)
cd D:\ws\haskell\stack-test; cabal v2-build --with-ghc=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\ghc.exe --with-ghc-pkg=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\ghc-pkg.exe --with-haddock=D:\bin\Programs\stack\x86_64-windows\ghc-8.6.5\bin\haddock.exe --project-file=D:\ws\haskell\stack-test\cabal.project --builddir=D:\ws\haskell\stack-test\dist-newstyle --only-configure test:stack-test-test
Build profile: -w ghc-8.6.5 -O1
In order, the following will be built (use -v for more details):
 - stack-test-0.1.0.0 (lib) (configuration changed)
 - stack-test-0.1.0.0 (test:stack-test-test) (dependency rebuilt)
Configuring library for stack-test-0.1.0.0..
Preprocessing library for stack-test-0.1.0.0..
Building library for stack-test-0.1.0.0..
[2 of 2] Compiling Lib              ( src\Lib.hs, D:\ws\haskell\stack-test\dist-newstyle\build\x86_64-windows\ghc-8.6.5\stack-test-0.1.0.0\build\Lib.o )

src\Lib.hs:76:12: error: Variable not in scope: xxx :: Expr -> Expr
   |
76 | simplify = xxx
   |            ^^^
cabal: Failed to build stack-test-0.1.0.0 (which is required by
test:stack-test-test from stack-test-0.1.0.0).

and the hie.log: https://gist.github.com/jneira/94c016ea42cb0cc5c167703441966f70

Opening it with hie master behaves correctly. Deleting cabal.project and keep stack.yaml (making hie use stack instead cabal) also works.

@fendor
Copy link
Collaborator Author

fendor commented Oct 30, 2019

@jneira Very interesting! I can confirm this behaviour! I suspect, though, this is behaviour of cabal-helper that we have no control over. We should probably talk with @dxld what is happening and if we can patch it!

@fendor fendor requested a review from mpickering October 31, 2019 15:23
hie-plugin-api/Haskell/Ide/Engine/Cradle.hs Outdated Show resolved Hide resolved
hie-plugin-api/Haskell/Ide/Engine/Cradle.hs Outdated Show resolved Hide resolved
@DanielG
Copy link

DanielG commented Nov 1, 2019

@jneira the error seems to be coming from the stack-test component whereas c-h is trying to (correctly) --only-configure the stack-test-test component, which I assume depends on stack-test. What would you expect it to do in this case? If depedencies fail to compile there's just no way to get a cabal component loaded into ghc properly.

PS: @fendor you're lucky @dxld notifications also end up in my inbox :P

@fendor
Copy link
Collaborator Author

fendor commented Nov 1, 2019

Oh, now I understand, it is about the order the units are queried! If the test unit is queried before the library, it will fail since the dependencies won't compile! OK, great, we can handle that, actually, I think!

@DanielG Haha, I didn't really want to tag you, I planned asking you in IRC about it!
But if you are already here, I would really appreciate a code review :)

@jneira
Copy link

jneira commented Nov 2, 2019

Oh, now I understand, it is about the order the units are queried! If the test unit is queried before the library, it will fail since the dependencies won't compile! OK, great, we can handle that, actually, I think!

Great! in my case i had not opened in the editor any file of the exe or test component, only the lib file with the compile error. I thought it wouldn't even try to open it.


[submodule "submodules/ghc-mod"]
path = submodules/ghc-mod
# url = https://github.com/arbor/ghc-mod.git
url = https://github.com/alanz/ghc-mod.git
# url = https://github.com/bubba/ghc-mod.git
url = https://github.com/fendor/ghc-mod.git
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can eventually move ghc-project-types into another location if needed, possibly into hie itself.

Comment on lines +45 to +48
isStackCradle = (`elem` ["stack", "Cabal-Helper-Stack", "Cabal-Helper-Stack-None"])
. BIOS.actionName
. BIOS.cradleOptsProg
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This feels like a really hacky way to work out if it is a stack cradle. I know hie-bios is intended to be low level and unconstrained to allow easy extension, but perhaps we should ask for certain common flags to be set indicating whether it is stack, or make the actionName a sum type identifying the well-known built-in hie-bios variants, and then have a constructor explicitly identifying that it is not one of those.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We can not make actionName a sum type, because then we can not add a cradle that hie-bios doesnt know about, like in this case "Cabal-Helper-Stack", except by adding a custom constructor such as Other String, which in my opinion defeats the purpose of it.
I talked to @mpickering to add predicates upstream that hide this string check.

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The reason this check is to hack around two things.

  1. The options that the "stack" cradle from hie-bios doesn't really return the correct options. So there is a hack to add an additional target.
  2. The cradle doesn't specify the version of GHC which is expected to be used to compile a project. For a cabal project we could just configure the project with the correct version of GHC (the one which was used to build that specific version of HIE) but that wouldn't work for stack projects which are locked to a specific compiler version.

So if we fix both of these things we can also remove this check. That seems like a better long term solution.

--
-- Utility functions to manipulate FilePath's
--
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it would be wise to build these up from the primitives in https://hackage.haskell.org/package/filepath, which take cognizance of platform differences, and are battle tested.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I actually opened an issue upstream (haskell/filepath#73) to add these functions to filepath, because I did not find a way to express them with the given API easily.
They are based on filepath, though.

@fendor
Copy link
Collaborator Author

fendor commented Nov 2, 2019

Oh, now I understand, it is about the order the units are queried! If the test unit is queried before the library, it will fail since the dependencies won't compile! OK, great, we can handle that, actually, I think!

Great! in my case i had not opened in the editor any file of the exe or test component, only the lib file with the compile error. I thought it wouldn't even try to open it.

We have to find out to which unit a Module belongs to. To find it out, we have to configure each unit, which builds the dependencies, and see if it defines the given Module. If it does, we found our unit and can continue, however, if the test-unit, is queried first, for no particular reason, then it crashes.

@DanielG
Copy link

DanielG commented Nov 2, 2019

Oh, now I understand, it is about the order the units are queried! If the test unit is queried before the library, it will fail since the dependencies won't compile! OK, great, we can handle that, actually, I think!

Hang on there. The order shouldn't matter. Cabal will just compile dependencies as needed. Can you give me some more context on what the problem you're having is?

@fendor
Copy link
Collaborator Author

fendor commented Nov 3, 2019

Oh, now I understand, it is about the order the units are queried! If the test unit is queried before the library, it will fail since the dependencies won't compile! OK, great, we can handle that, actually, I think!

Hang on there. The order shouldn't matter. Cabal will just compile dependencies as needed. Can you give me some more context on what the problem you're having is?

I am referring to the code in https://github.com/mpickering/haskell-ide-engine/pull/26/files#diff-8134e86af784446400558bb3cbfe7dcfR387
The approach is that we are trying to find the correct Unit for a given Module. We lazily try to load a single unit via runQuery (unitInfo unit) env and check if one of its components exposes the Module.
This builds the dependencies of a unit.
In the case of a test unit, that depends on the library. If currently library does not build, then the test unit can not be loaded, e.g. there is an exception in runQuery (unitInfo unit) env.
This is what I meant with order of unit-loading, we get all units in a package and query one unit after another to see if the module is exposed by any of the units. In the case of stack, the unit of the library is loaded first.

-- Intended usage:
--
-- >>> isFilePathPrefixOf "./src/" "./src/File.hs"
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice. We should run doctests in CI.

Just yaml -> fixCradle <$> BIOS.loadCradle yaml
Nothing -> cabalHelperCradle fp
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does this then replace logic in hie-bios for implicitly finding a cradle with cabal-helper instead? Presumably cabal-helper handles this better then?

Copy link
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes it does and in theory, yes it does.

@mpickering
Copy link
Owner

The documentation is amazing for this patch 😲

I think we should merge it and deal with any fall out in subsequent PRs. Amazing work @fendor !

@fendor
Copy link
Collaborator Author

fendor commented Nov 5, 2019

I think we should merge it and deal with any fall out in subsequent PRs.

The issue described in #26 (comment) needs to be fixed, I would like to fix it before merge. The fix is simply catching exceptions in https://github.com/mpickering/haskell-ide-engine/pull/26/files#diff-8134e86af784446400558bb3cbfe7dcfR398 .

@mpickering
Copy link
Owner

Ok, once you are happy then just merge the branch in and then #40 as well.

@fendor
Copy link
Collaborator Author

fendor commented Nov 5, 2019

@jneira, the issue https://github.com/mpickering/haskell-ide-engine/pull/26/files#diff-8134e86af784446400558bb3cbfe7dcfR398 should be solved now!

@fendor fendor merged commit d79e033 into mpickering:hie-bios Nov 5, 2019
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants