Replies: 10 comments 46 replies
-
|
I know everybody uses UTF-8 locales these days, but, at least on GNU/Linux, wouldn't it make sense to switch the behaviour depending on the value of the |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately, sources are tied to the |
Beta Was this translation helpful? Give feedback.
-
|
macOS (in the UK): |
Beta Was this translation helpful? Give feedback.
-
I don't follow you. What I meant is: default to UTF-8 when Alire crate is initialized if LANG contains UTF-8. Default to do nothing otherwise. What you are proposing is determined at crate initialization or at crate building? |
Beta Was this translation helpful? Give feedback.
-
|
Sorry, I misunderstood you. What you propose makes sense in isolation, but the problem for |
Beta Was this translation helpful? Give feedback.
-
|
Yes, you're right, I only gave a thought to an isolated development, but when you publish your sources, the value of your particular |
Beta Was this translation helpful? Give feedback.
-
|
Disclaimer: I do not know much about encoding and codepages and such. But the discussion rang a bell about databases; |
Beta Was this translation helpful? Give feedback.
-
|
If you could narrow down a bit the information to look at, I'd definitely will give it a read. More generally, I think the general computing trend has been for years to steadily move towards adopting Unicode, and UTF-8 as the default Unicode encoding and environment. Trouble is always the breakage along the way. Mind you, for Alire I don't think the situation is as dire as it sounds. It's only to agree on a couple of things, and for most code it won't be even a point to consider. The problem IMO is the huge inertia with using |
Beta Was this translation helpful? Give feedback.
-
|
A better way IMO would be something like ;and then perhaps YMMV |
Beta Was this translation helpful? Give feedback.
-
|
Hi, I don’t normally use Text_IO, so I haven’t had to deal with the issues. However, given the bullet item “ With -gnatW8, Ada.Text_IO applies a Latin1 --> UTF8 conversion which garbles UTF-8 strings” Does this mean Ada.Text_IO should no longer be used if -gnatW8 is applied, and instead Ada.Wide_Text_IO should be used instead? |
Beta Was this translation helpful? Give feedback.
-
|
That's it, @Micronian2. |
Beta Was this translation helpful? Give feedback.
-
Text_IO is trash and Wide_Wide_Text_IO is not much better. I think about borrowing TTextWriter from Delphi, able to operate on any stream, unlike Ada IO. Modula-2 has some useful stuff too, to some extent. I like ReadToken. Delphi's TTextReader/TTextWriter with Modula-2 addons would be nice. Also terminal handling stuff is useful: moving cursor over screen, and setting colors like Turbo Pascal was able to do since 1980s. And waiting for "single keypress". I almost find no use of Ada's paragraph stuff occuping so much space in Ada.(Wide_Wide_)Text_IO. |
Beta Was this translation helpful? Give feedback.
-
|
I voted for UTF8 support for all Ada sources in Alire. But it's not without pause. Personally, I stick to ASCII only in my personal projects. It is my impression that one of the original design goals of the Ada language is for the code to look as much as possible like normal English prose. Just look at all the reserved keywords, they are all English words. Thus, introducing the possibility of using UTF8 in the source code with the purpose of the code looking like English prose doesn't seem like a natural course of action. Being able to use identifiers in the source code expressed in Spanish, Japanese or Farsi in one and the same code base won't help maintainability. Well, it is possible to put Farsi in the source code even when the code is ASCII only, for example "function Payam return String;". From the compilers perspective, in general, the identifiers only need to be unique no matter what language is used. Let's say it is possible to use a static code analysis tool to check that an Ada code base only uses identifiers from English regardless of the Ada source code is in ASCII or UTF8. Why else would one want to use UTF8 in the code base? Perhaps one would like to put Emojis in the comments of the source code. Perhaps that is the main motivation for allowing UTF8 in one's source code? No, thinking about it, I would say the main motivation for UTF8 in the source code is to be able to embed UTF8 strings. When sticking to ASCII only, all the UTF8 strings need to be put in separate files from the Ada source code. It is more convenient to be able to embed UTF8 strings directly in the source code. So there is a use-case where it is more convenient, but not necessary. Another reason I avoid UTF8 in the source code are the existence of different code-points in UTF8 which look the same (https://util.unicode.org/UnicodeJsps/confusables.jsp?a=test&r=None) . I don't want the possibility in my personal projects to define a function Hello and then having the compiler tell me it is not possible to call it because there is no function Hello, and the difference would be perhaps for the "e" in the definition of the function and the "e" used in the function call to be different code points but look the same to the naked eye. It would not take me long to figure out what's wrong, but it is a source of error that I am happy to be without. Yet another reason against UTF8 Ada source code is security. Allowing UTF8 in the source code enables dangerous UTF8 character sequences (https://krebsonsecurity.com/2021/11/trojan-source-bug-threatens-the-security-of-all-code/). When for example writing Go code there is a static source analysis tool for detecting these in Go source code: https://github.com/breml/bidichk The main motivation for allowing UTF8 is to make the transition for newcomers to the Ada language easier. If people have the culture and tradition of putting UTF8 in the source code of their Go, Rust, Java, C# applications etc. it may be off-putting if the Ada source code would be ASCII only. |
Beta Was this translation helpful? Give feedback.
-
|
I thought those string literals outside the Latin-1 set would raise some of the compilation errors I saw in aep-0004.md#impacted-crates, like |
Beta Was this translation helpful? Give feedback.
-
|
String is NOT a sequence of bytes, it's a sequence of Latin-1 characters. See ARM 3.5.2 2/3:
For sure Ada Stemmer contains incorrect Ada code. For example Among_String : constant String := "вшись" &Without
|
Beta Was this translation helpful? Give feedback.
-
Thank you Maxime for the clarification. I've always considered String as a sequence of bytes due to the |
Beta Was this translation helpful? Give feedback.
-
|
The consequence of Ada String being Latin1 means that with Ada.Strings.UTF_Encoding.Wide_Wide_Strings;
with Ada.Text_IO;
procedure Print is
S : String := Ada.Strings.UTF_Encoding.Wide_Wide_Strings.Encode ("вшись");
begin
Ada.Text_IO.Put_Line (S);
end Print;will never print the expected value despite the fact that at some point Does this mean that if we want to support UTF-8 in some program, we must not use |
Beta Was this translation helpful? Give feedback.
-
|
Your projects that you are putting into Alire, are then going to be linked with the main program that is not supposed to be uploaded to Alire, but this main program can contain a lot of native strings because the program only makes sense in a specific locale. For instance, EAEU customs formats handling only makes sense in one country only due to different local laws. Also if the whole developer team are not native English, but program is internationalized, then developers write strings in the language they know best, and then dedicated translators provide English counterparts on demand. I worked in several such projects. What is the point in requiring all developers to have good written English knowledge if we don't sell much to English speaking countries? |
Beta Was this translation helpful? Give feedback.
-
|
The Consider the following simple Ada program that reads the current directory and print file names. This Ada program only uses the ASCII character set, no UTF-8 in it: with Ada.Directories;
with Ada.Text_IO;
procedure Readdir is
Search_Filter : constant Ada.Directories.Filter_Type
:= (Ada.Directories.Ordinary_File => True,
Ada.Directories.Directory => True,
Ada.Directories.Special_File => False);
Search : Ada.Directories.Search_Type;
Ent : Ada.Directories.Directory_Entry_Type;
begin
Ada.Directories.Start_Search (Search, Directory => ".",
Pattern => "*", Filter => Search_Filter);
while Ada.Directories.More_Entries (Search) loop
Ada.Directories.Get_Next_Entry (Search, Ent);
Ada.Text_Io.Put_Line (Ada.Directories.Simple_Name (Ent));
end loop;
end Readdir;Create a file that shows the issue: If you compile it with If you compile it with The same issue will arise if you get your UTF-8 string from a database or from an i18N database such as Please, stay away from that broken compiler option until it is really fixed. |
Beta Was this translation helpful? Give feedback.
-
|
Propert Some Linux APIs like libidn2 can accept UTF-32 directly. |
Beta Was this translation helpful? Give feedback.
-
The GtkAda library uses |
Beta Was this translation helpful? Give feedback.
-
Then GtkAda binding is wrong. They should have used Wide_Wide_String. I have faced GPS crashes on cyrillic letters because of this. GPS handled Ada fine, but crashed on non-Ada source. Ada application crashing. Pathetic. Stop this madness already. |
Beta Was this translation helpful? Give feedback.
-
I have had not crashes but bizarre behavior with Spanish-heavy UTF-8 sources. On-the-fly casing is broken (it capitalizes after a non-ASCII letter), which causes style warnings and with warnings-as-errors it's a pain. Tabulation is also broken, wild guessing because there are some mixups of bytes/character counts. |
Beta Was this translation helpful? Give feedback.
-
No such bad behaviour if you use |
Beta Was this translation helpful? Give feedback.
-
|
ARM 2.1 16 says:
Without |
Beta Was this translation helpful? Give feedback.
-
|
Looking at ACATS test C250002, judiciously edited for reasons documented in GCC PR81114 to remove the use of library unit names using extended characters (e.g. The fact that the source has a BOM (which In any case, whether you need |
Beta Was this translation helpful? Give feedback.
-
|
Here is my cont-example. Save it in UTF-8 and compile: procedure Main is
C : Character := '©';
begin
null;
end Main;So, my conclusion is that the GNAT doesn't accept source in UTF-8 without
PS |
Beta Was this translation helpful? Give feedback.
-
|
Just looked at AI05-0286 and you are of course right. From that document,
I feel quite strongly that this is a case of the tail wagging the dog! |
Beta Was this translation helpful? Give feedback.
-
|
As @reznikmm pointed out, the Ada String is a sequence of Latin-1 which to my understanding means that:
Any other advises? |
Beta Was this translation helpful? Give feedback.
-
I could agree with that, but then why
Would that work on Windows? I guess that, without As an example of what I mean. This simple program works mostly right in default mode (except for the length problem in UTF-8 terminal; it works perfect if you configure the terminal in Latin-1, as in the old days). On the other hand, it is miserably broken when using with Ada.Text_IO;
procedure Text_Io_Test is
Test : constant String := Ada.Text_Io.Get_Line;
begin
Ada.Text_Io.Put_Line (Test);
Ada.Text_Io.Put_Line ("Length: " & Test'Length'Image);
end Text_Io_Test;
In default compiler mode + UTF-8 terminal: In default compiler mode + Latin-1 terminal: |
Beta Was this translation helpful? Give feedback.
-
|
Discussion in comp.lang.ada related to what I'm saying: https://groups.google.com/g/comp.lang.ada/c/01RzkosroZs |
Beta Was this translation helpful? Give feedback.
-
|
The issue comes from |
Beta Was this translation helpful? Give feedback.
-
That's the point. It should be everywhere or nowhere, and it should be independent on how the source code text encoding is selected. That's why I think there should be two switches. |
Beta Was this translation helpful? Give feedback.
-
|
The thing is that for files there's also the And then there's the |
Beta Was this translation helpful? Give feedback.
-
|
I guess program text encoding is best stored as program text meta data in the alire crate. |
Beta Was this translation helpful? Give feedback.
-
|
Unfortunately this won't work with current gnat compiler, because it reads
all sources in a single encoding. For instance if you specify -gnatW8 for
your library then compiler will read all dependency as utf8.
|
Beta Was this translation helpful? Give feedback.
-
|
чт, 9 бер. 2023, 05:38 користувач Jesper Quorning @.***> I guess program encoding is best stored as program text meta data in the —
|
Beta Was this translation helpful? Give feedback.
-
|
Lots of interesting discussion. It reinforces my impressions though that without mandated The hard sad truth seems to be that without I don't see how moving forward without Of course you can use String to store UTF-8... as long as you deal with the IO in some raw form. EDIT: and let's not forget that even without |
Beta Was this translation helpful? Give feedback.
-
|
Is there an 'easy way' to cofigure which I read somewhere about a person (from Finland, Hakkinen? perhaps) who(m?) wrote his own standard with - among other things - another definition of Update: |
Beta Was this translation helpful? Give feedback.
-
|
What you say reminds me of Drake, but no mention of embedded and not from Finland, but from Japan. Drake is a different runtime for GNAT, which assumes UTF-8 for String instead of Latin-1: https://github.com/ytomino/drake/wiki/Incompatibility#string-is-utf-8-not-latin-1 |
Beta Was this translation helpful? Give feedback.
-
We are again discussing what (or if) Alire should do something about Unicode. My last attempt at documenting the situation is at #1332 (see unicode.md and AEP-0004.md for full details). In a way I see this as the same pain that arose in the Python2 -> Python3 transition.
I want to get more feedback from experts and the community feeling. There's a poll below.
The gist of the conumdrum is:
-gnatW8GNAT presumes UTF-8 sources and changesText_IOandWide_Wide_Text_IOdefault behavior globally.Ada.Text_IOwill behave differently depending on-gnatW8being active.-gnatW8,Ada.Text_IOprints bytes as-is.-gnatW8,Ada.Text_IOapplies a Latin1 --> UTF8 conversion which garbles UTF-8 strings.Ada.Text_IOworks "as expected" for UTF-8 sources without-gnatW8.Ada.Wide_Wide_Text_IOdoesn't work by default without-gnatW8for non-Latin1 characters (prints brackets encoding).There's no good default that will work for all cases for both unaware and aware users, which is the main point of contention.
Reading past debates at
comp.lang.ada(mainly this one) it seems this is beyond repair from the ARG side, and there are proponents of both extremes, even of avoiding it completely by staying in ASCII in sources.In the poll below you'll find the following options:
-gnatW8).-gnatW8) are more likely to find trouble from non-gnatW8 crates.Text_IOdon't really behave as they should, even if this counterintuitively works.-gnatW8the default (this is currently enabled in themasterbranch)Ada.Text_IOis unclear until runtime.Ada.Text_IOgarbling UTF-8 strings.27 votes ·
Beta Was this translation helpful? Give feedback.
All reactions