Allow alire to access indexes using ssh

[rowan-walshe]

Currently to add a remote index alire expects the URL that starts git+. SSH urls will normally start git@ instead. By changing a few checks, alire is able to access an index via ssh.

This is useful for if you want to host a non public index, but don't want to enter a username and password to be able to access the index (which is not best practice security wise).

[Fabien-Chouteau]

clic-user_input.adb:265:31: error: literal out of range of type Standard.Character
clic-tty.ads:101:19: error: literal out of range of type Standard.Character
clic-tty.ads:106:17: error: literal out of range of type Standard.Character

-gnatW8 has made its first victim 😛

@reznikmm can you help us fix this? -> https://github.com/alire-project/clic/blob/c7aabf94e0946ae3a5b77c29cc45fd28f70e17f6/src/clic-tty.ads#L101

[mosteo]

I'm surprised because I thought this was enough:

$ file clic-tty.ads
clic-tty.ads: Unicode text, UTF-8 text

[reznikmm]

I'm surprised because I thought this was enough:

$ file clic-tty.ads
clic-tty.ads: Unicode text, UTF-8 text

That character isn't a Latin-1 character, but String contains only Latin-1 subset of Unicode. That's why compiler is complaining.

If you want to keep using String to handle UTF-8 values, then you can use a conversion function like Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Encode. It will encode Wide_Wide_String to UTF-8 and return String with UTF-8 bytes:

y : constant String :=
  Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Encode 
    ("✓");

The -gnatW8 also changes Ada.Text_IO.Put_Line behavior. It will convert Latin-1 string to UTF-8. If you pass already encoded UTF-8 value, then you will get a "broken" output:

with Ada.Strings.UTF_Encoding.Wide_Wide_Strings;
with Ada.Text_IO;

procedure Main is
y : constant String :=
  Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Encode 
    ("✓");

begin
   Ada.Text_IO.Put_Line (y);
end Main;

$ /tmp/def/obj/main
â

One way of fixing output is to decode UTF-8 to Wide_Wide_String and use Ada.Wide_Wide_Text_IO:

with Ada.Strings.UTF_Encoding.Wide_Wide_Strings;
with Ada.Wide_Wide_Text_IO;

procedure Main is
y : constant String :=
  Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Encode 
    ("✓");

begin
   Ada.Wide_Wide_Text_IO.Put_Line
     (Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Decode (y));
end Main;

If this is too intrusive to you, then you could revert Ada.Text_IO behavior (disable Latin-1->UTF-8 conversion during output) by specifying -Wb switch for gnatbind.

-Wxe
    Override default wide character encoding for standard Text_IO files.

(Values are described in GNAT RM).

with Ada.Strings.UTF_Encoding.Wide_Wide_Strings;
with Ada.Text_IO;

procedure Main is
y : constant String :=
  Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Encode 
    ("✓");

begin
   Ada.Text_IO.Put_Line (y);
end Main;

$ gprbuild -gnatW8 main.adb -bargs -Wb
$ ./main
✓

I hope my explanation help.

[godunko]

Is it time to start to separate stream elements and text strings and to migrate to VSS? ;)

[mosteo]

Yup, at the bare minimum we need clean type separation. Oh well, let's not derail this PR any more. I'm sorry, @rowan-walshe, it seems your PR will have to wait for a little while until this is fixed.

[rowan-walshe]

So I've run into a few more issues that aren't fixed by this change so far. Currently it appears that Alire is setup to work assuming everything is public (i.e. the index, and the things that crates in an index can link out to like git commits or tarballs).

Obviously downloading a tarball that is not publicly accessible is hard to handle, as how would Alire know how to authenticate with the download service. Fortunately with git this is not an issue as Alire just uses the system's git, and therefore, assuming the user has the relevant ssh key stuff setup, Alire theoretically shouldn't have do anything different to what it currently does when calling out to git.

However, as Alire assumes that git repositories are publicly accessible, there are a number of defensive checks scattered around to help out in cases where a git@ URL is provided instead of a http URL. In some cases (all?) it will also try to convert a git@ URL to a public http URL. Theoretically if these checks were removed, it should allow users to setup and run private indexes that contain private crates. I understand that for public indexes, like the community index, you would still want all crates to be publicly accessible, and CI checks could be added to make sure this is the case (if there isn't already something similar).

I didn't want dig deeper into making changes if people had objections to this, or think it should be done in a different way.

Maybe this question/suggestion is too big for this PR. Let me know if I should raise a ticket for it to be discussed

[mosteo]

Right, publish was implemented with the idea that the crate destination is the community index. Some checks have been already relaxed with --force, but we might be better served by having something like --private where we operate with the idea that the result is going to be be used privately.

[onox]

It might be useful to be able to specify something like private = true in alire.toml. alr could use this to prevent someone from accidentally submitting a crate to a public index (especially if in the future less manual steps are needed to publish a crate).

At least NPM and PyPI support this.

[dalybrown]

Has this feature been abandoned or implemented elsewhere? We also would like to use a private index with ssh keys. I'm happy to contribute back to finish this if there is anything left to get it merged!

[mosteo]

Not implemented yet, simply because of lack of time. If you want to help with it, you're certainly welcome.

From looking at the patch, it looks like this does the bare minimum to allow adding an index with git@ transport. That could be enough, the publishing changes and private flag in manifests could be done separately.

[dalybrown]

Should I create a new pull request or can @rowan-walshe maybe give me permissions to contribute to this branch? Thanks!

[mosteo]

It's OK if you pick up were Rowan left and open a new PR from your own fork.

[rowan-walshe]

@dalybrown Feel free to pick this up where I left off (not that I'd done much). Note that on playing around with Alire more, I found that while undocumented, it is possible to use Alire with indexes accessed using ssh already. For example, normally you might add an index like:

alr index --name=community --add=git+https://github.com/alire-project/alire-index.git

It is possible to add the same index using ssh:

alr index --name=community_ssh --add=git+ssh://[email protected]/alire-project/alire-index.git

Note that this is not a direct copy of the field from GitHub. GitHub gives you:

[email protected]:alire-project/alire-index.git

Then you need to add git+ssh:// to the start, and replace a : with a /. Maybe some work could be done to automate this conversion internally, or to update the documentation to document how it currently works?

[dalybrown]

Sounds good. I have my own fork of it and am now using it internally with ssh. Once we've used it a bit more and are confident it works are required I'll update this pull request.

[mosteo]

Merged via #1571