IDE 1.6.2 platform.txt uses new undefined platform variable runtime.tools.avr-gcc.path

[jantje]

The arduino eclipse plugin does not work with arduino IDE 1.6.2 and 1.6.3
The reason is the definition of compiler.path in platform.txt
In Arduino IDE 1.6.1 in avr it is defined as
compiler.path={runtime.ide.path}/hardware/tools/avr/bin/
In Arduino IDE 1.6.2 and 1.6.3 in avr it is defined as
compiler.path={runtime.tools.avr-gcc.path}/bin/

The problem is that runtime.tools.avr-gcc.path is not defined in the platform.txt so it adds another unneeded and probably undocumented environment variable.

There is a very similar situation with sam

Can we revert back to the old way?

[ffissore]

We can't revert because the toolchain is no more installed into {runtime.ide.path}/hardware/tools/avr/bin/ (only the bundled avr one and only if you haven't upgraded the IDE yet)
This is where the IDE is creating those runtime keys https://github.com/arduino/Arduino/blob/master/arduino-core/src/processing/app/BaseNoGui.java#L761-L771

[ffissore]

With "upgrading the IDE" I meant upgrading core/tools using boards manager

[jantje]

I just upgraded AVR on my system and the {runtime.ide.path}/hardware/tools/avr/bin/ is still valid.
Anyway my main concern is not that compiler.path should be defined as {runtime.ide.path}/hardware/tools/avr/bin/

my concern is that when compiler.path is defined as {runtime.tools.avr-gcc.path}/bin/
runtime.tools.avr-gcc.path should be defined in platform.txt.
I really can't understand why code is needed to define runtime.tools.avr-gcc.path.

And just in case code is actually needed.... why is it not part of the boards manager adding it to the platform.txt when installing?

[matthijskooijman]

AFAICS, the reason that runtime.tools.avr-gcc.path is not defined in platform.txt, is that in general, a core/platform doesn't actually ship the gcc binaries and doesn't know where these will be installed. The IDE (code) downloads and installs the toolchain, and sets that variable to let the platforms know where the avr-gcc toolchain lives.

In the old situation, you had to define {runtime.ide.path} as well in code, right? In the new situation, the platform.txt makes fewer assumptions about where the toolchain is installed, which is a good thing IMHO.

[jantje]

In the old situation I defined a default for runtime.ide.path. If the platform contains one the default gets overwritten.
In the new situation I still need to define a default for runtime.ide.path (for users using arduino IDE 1.5.X to 1.6.1)
And I need to define a default for runtime.tools.avr-gcc.path (for users using arduino IDE 1.6.2 and later) to get a correct runtime.ide.path

In the new situation, the platform.txt makes fewer assumptions about where the toolchain is installed, which is a good thing IMHO.

I'm not sure whether I agree that making fewer assumptions about the toolchain location is a good thing.
But lets first assume it is
In the old situation
pre 1.5.6 compiler.path was not defined
in 1.5.6 compiler.path was defined
a89f5e6
in 1.6.2 compiler.path is defined as {runtime.tools.avr-gcc.path}/bin/
in other words platform.txt still makes more assumptions about where the compiler is than pre 1.5.6

Now what is the benefit of platform.txt of not making assumptions where the compiler is?
I always assumed that arduino bundled a compiler avr dude and IDE that works. I saw this bundling in the platform.txt and boards.txt. In the new way the folder even assumes a version for these. packages/arduino/hardware/[architecture]/[version]
I would assume that at this level you know (and want to know/enforce) which version of the compiler you want to use.

If you stick to the current solution please update https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5---3rd-party-Hardware-specification the section global predefined properties

[jantje]

Any input for me?

[razerraz]

I've just spended couple hours trying to understand why my projects won't compile anymore
This issue was particuraly vicious for me because I've AVR standalone gcc & headers installed on my system (arch linux), most of my projects was ok compiling, just some weird issues with librairies (software serial for instance) on others
Just forgot I had done an upgrade few days before, with 1.6.3 included :/

Just upset not having found this report before spending so much time, not having clear view of what was happening on that mess

[jantje]

I'd still appreciate some input here.

[matthijskooijman]

in other words platform.txt still makes more assumptions about where the compiler is than pre 1.5.6

@jantje, You're right about that. But in pre-1.5.6, the path to avr-gcc was passed to the platform.txt using an avr-specific setting of the compiler.path variable, which is inconsistent wrt sam and other platforms. In the latest code, the runtime.tools.avr-gcc.path variable is a lot better named. There is no need for any platform hard-coding (i.e. it can be defined for all platforms, so even the sam platform.txt could decide to call avr-gcc if it wanted). For sam, there is the corresponding runtime.tools.arm-none-eabi-gcc.path variable now, making things consistent again.

Now what is the benefit of platform.txt of not making assumptions where the compiler is?

The benefit is that (third-party) platform.txt files no longer have to hardcode where, inside the IDE, the toolchain lives. The toolchain can be freely moved to another location, without needing to update all platform.txt files. Since, AFAIK, the toolchains are now downloaded on-demand and installed outside of runtime.ide.path, I think the previous approach wouldn't even work.

I always assumed that arduino bundled a compiler avr dude and IDE that works. I saw this bundling in the platform.txt and boards.txt. In the new way the folder even assumes a version for these. packages/arduino/hardware/[architecture]/[version]
I would assume that at this level you know (and want to know/enforce) which version of the compiler you want to use.

Not sure what you're saying exactly here. In what way does the new way assume that stuff?

This issue was particuraly vicious for me because I've AVR standalone gcc & headers installed on my system (arch linux), most of my projects was ok compiling, just some weird issues with librairies (software serial for instance) on others

@razerraz But then you should have had local modifications to your Arduino install before? The IDE doesn't normally support a non-bundled toolchain out of the box?

[jantje]

@matthijskooijman
Thanks for the feedback.

The benefit is that (third-party) platform.txt files no longer have to hardcode where, inside the IDE, the toolchain lives.

I think this is a really good argument. It also explains the need for a sam and avr "environment variable".
However -from my point of view- it moves the problem of finding the right toolchain location from the platform (commercial) provider to the tool provider (non commercial) :-(.

Thinking about how to handle this.
I'm wondering: will 1 arduino IDE installation support 1 or more toolchain(s) for each architecture?
If more are supported (and there are already as you can install the avr toolchain with the library) how will the arduino IDE (and the toolprovider) know which version to use?

Jantje

PS A update of https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5---3rd-party-Hardware-specification is needed.

[matthijskooijman]

However -from my point of view- it moves the problem of finding the right toolchain location from the platform (commercial) provider to the tool provider (non commercial) :-(.

I don't think this commercial / non-commercial distinction always applies. Also, doing it like this simply makes sense from a technical perspective - it puts the burden of locating the toolchain at the "tool provider", which is really the only one that can know where it is installed. In your case, I think, it's a bit more complicated because you use the Arduino IDE installation to provide the toolchain, so your tool has to hardcode the path within the IDE installation, since there is not any way to query the IDE for toolchain locations. That might actually be worth considering - having a commandline option that prints toolchain locations and other automatic variables like these. I'm not sure off-hand how easy these would be to implement - IIRC the preferences/recipe variables are a bit intermixed and scattered throughout the code.

I'm wondering: will 1 arduino IDE installation support 1 or more toolchain(s) for each architecture?
If more are supported (and there are already as you can install the avr toolchain with the library) how will the arduino IDE (and the toolprovider) know which version to use?

I'm not sure if any support for this is present now (I think not). However, I think it could be implemented reasonably simply, just have the IDE track multiple versions of the toolchain, let the user select the one to use and then set the runtime.tools.avr-gcc.path to the selected one?

[damellis]

BTW, is it possible for the Eclipse plugin (or other third party tools) to
just call the command-line version of the Arduino IDE to do the
compilation?

On Tue, Apr 28, 2015 at 3:38 AM, Matthijs Kooijman <[email protected]

wrote:

However -from my point of view- it moves the problem of finding the right
toolchain location from the platform (commercial) provider to the tool
provider (non commercial) :-(.

I don't think this commercial / non-commercial distinction always applies.
Also, doing it like this simply makes sense from a technical perspective -
it puts the burden of locating the toolchain at the "tool provider", which
is really the only one that can know where it is installed. In your case, I
think, it's a bit more complicated because you use the Arduino IDE
installation to provide the toolchain, so your tool has to hardcode the
path within the IDE installation, since there is not any way to query the
IDE for toolchain locations. That might actually be worth considering -
having a commandline option that prints toolchain locations and other
automatic variables like these. I'm not sure off-hand how easy these would
be to implement - IIRC the preferences/recipe variables are a bit
intermixed and scattered throughout the code.

I'm wondering: will 1 arduino IDE installation support 1 or more
toolchain(s) for each architecture?
If more are supported (and there are already as you can install the avr
toolchain with the library) how will the arduino IDE (and the toolprovider)
know which version to use?

I'm not sure if any support for this is present now (I think not).
However, I think it could be implemented reasonably simply, just have the
IDE track multiple versions of the toolchain, let the user select the one
to use and then set the runtime.tools.avr-gcc.path to the selected one?

—
Reply to this email directly or view it on GitHub
#2982 (comment).

[matthijskooijman]

BTW, is it possible for the Eclipse plugin (or other third party tools) to just call the command-line version of the Arduino IDE to do the compilation?

Yup, that's possible. I was planning to suggest that in my previous comment, but it seems I forgot :-)

[jantje]

BTW, is it possible for the Eclipse plugin (or other third party tools) to just call the command-line version of the Arduino IDE to do the compilation?

Yup, that's possible. I was planning to suggest that in my previous comment, but it seems I forgot :-)

It is possible but it would take away most of the added functionality offered by the Eclipse plugin.
Basically CDT has lots of functionality that the Arduino IDE does not do. To be able to do so CDT needs the recipes. This extra functionality is enhancing the end user experience. I typically call it the indexer. I'm talking about "context sensetive colouring for defines", "the keywords.txt without keywords.txt", auto completion, code validation while typing....

So yes technically it is possible but practically it is not.
Well at least CDT needs to know the "recipes" and if it knows that .... why not use them directly?

The technically possible versus the practical advantage is the same thing I can't get my head around with the switching of the toolchain.
I explained this before but matthijs marked it as "Not sure what you're saying exactly here." So I'll try again.
I think one of the most important things that made Arduino big is: "Here you have a set of tools and it works".
Basically that means (in avr world) combining a avrdude version with a avr-gcc toolchain version with a IDE and finding/coding the right commands to make it work.
This means that a platform.txt in a arduino IDE version assumes avrdude VX.YZ and avr-gcc VX.YZ.
All these are delivered in 1 package and the package is as such self contained.

Now with the boards manager one can "technically" add a toolchain but does that have a practical advantage? Will it not break what made arduino Big?

If Arduino core team decides there is only 1 toolchain per architecture the environment variable {runtime.tools.[architecture].path} can be defined in the platform.txt of that architecture. Or in a platform.txt of the ide.
If Arduino core team decides there can be multiple toolchains per architecture I'd like to know how they will handle that. And personally I don't see added value of having the user selecting it over having multiple instances of the arduino IDE.

[matthijskooijman]

If Arduino core team decides there is only 1 toolchain per architecture the environment variable {runtime.tools.[architecture].path} can be defined in the platform.txt of that architecture. Or in a platform.txt of the ide.
If Arduino core team decides there can be multiple toolchains per architecture I'd like to know how they will handle that. And personally I don't see added value of having the user selecting it over having multiple instances of the arduino IDE.

I actually think it's the reverse: There can be multiple platforms per architecture, sharing a single toolchain. One of these platforms is currently the official Arduino one, the others are third-party platforms. If a toolchain has a 1-to-1 connection with a platform, each third-party platform has to supply its own toolchain, which you don't really want.

Not sure if this answers your concerns? :-)

[bunnis]

Why dont the arduino core team support this plugin? damn

[jantje]

Not sure if this answers your concerns? :-)

It doesn't. I'm talkig about 1 toolchain per architecture. Not 1 toolchain per platform.

[matthijskooijman]

It doesn't. I'm talkig about 1 toolchain per architecture. Not 1 toolchain per platform.

I can't speak for the Arduino team (perhaps @ffissore or @cmaglie can clarify), but I do not think there are plans to support multiple toolchains per architecture.

Before, you said:

If Arduino core team decides there is only 1 toolchain per architecture the environment variable {runtime.tools.[architecture].path} can be defined in the platform.txt of that architecture.

As mentioned before, a (third-party) platform.txt doesn't know (and can't be expected to know) in advance where the toolchain lives, except if it actually provides the toolchain itself (this is why I talked about a 1-on-1 connection between toolchain and platform before).

IIUC, the avr toolchain is currently included with the IDE (and stored inside the Arduino install directory), while the sam toolchain is downloaded on-demand (and stored inside some documents or settings directory I think). This means that there is no way for a platform.txt to reach both, given just the runtime.ide.path. Even if it could, it would need to hardcode where to find this toolchain. If a later version of the IDE decides to ship the sam toolchain again, or decides to download the avr toolchain on-demand, their location would change, breaking all platforms again. I'm not 100% sure I got this exactly right, but the general gist should be ok.

Taking a step back, why is this change a problem for you exactly? In your eclipse plugin, you have to expose a few variables while evaluating the recipes. This just adds a few more variables there, for the toolchain locations. If you think that (third-party) platform.txt files could hard-code the toolchain path, why can't you do the same in your plugin? In fact, I'd say your plugin is more suited to do so, since I assume it can run some code to try multiple locations and use the first one that actually exists, or apply some other smartness to find the toolchains (as opposed to platform.txt, which can only hardcode a single value).

[jantje]

Before, you said:

If Arduino core team decides there is only 1 toolchain per architecture the environment variable {runtime.tools.[architecture].path} can be defined in the platform.txt of that architecture.

As mentioned before, a (third-party) platform.txt doesn't know (and can't be expected to know) in advance where the toolchain lives, except if it actually provides the toolchain itself (this is why I talked about a 1-on-1 connection between toolchain and platform before).

I see where the communication went wrong. Let me rephrase it.
If Arduino core team decides there is only 1 toolchain per architecture the environment variable {runtime.tools.[architecture].path} can be defined in the arduino delivered platform.txt of that architecture.

This would allow the plugin to parse the platform.txt and get the value (and use those as defaults). As it is a fixed value (due to: there is only 1 toolchain per arduino IDE version per architecture) these can be "hard coded" in the platform.txt files of arduino. In other words: This solution would have very little impact to the arduino core team (1 line per platform.txt file) with 100% result and it would enable "tool providers" to find the correct version.
I think a better idea (but with more impact on the arduino core team) would be to have a environment.txt file in the arduino IDE that allows to configure environment variables like arduino IDE version and the values we discuss now. The order of processing should be environment->platform->board.
Even if the arduino core team doesn't want to "hard code" the value; it could be part of the installation instructions of the plugin; or the plugin could do it; but I'd like to get the confirmation 1 arduino IDE version + 1 hardware = 1toolchain

[jantje]

As 1.6.4 is about to be released.
Will I have to declare the plugin incompatible with 1.6.4?
Can I get some feedback?

[ffissore]

sorry @jantje I think you may keep flagging 1.6.4 as incompatible with eclipse plugin. I hope we'll have a workaround for 1.6.5

[matthijskooijman]

Putting a harcoded path in the bundled platform.txt does not seem like a useful solution:

• In the most recent IDE, the cores (including platform.txt) can be updated separately from the IDE (and I think toolchains are again downloaded separately from cores, right?).
• How would a third-party platform.txt obtain the right compiler path? It would still need the current variable defined by the IDE, which would end up with two different ways of defining the path.
• Having the path in the official platform.txt means people will be copying that platform.txt, and we'll again have all third-party platform.txt files with hardcoded paths.

Your suggestion of an environment.txt could be a good way to handle this. Since toolchains can be downloaded on-demand, this means that the environment.txt must be generated dynamically by the IDE, whenever a toolchain is downloaded. Alternatively, having a arduino --toolchain-info that generates such info (or perhaps a more generic name, depends on what it outputs exactly) could work.

I'm not familiar with the new modular downloading code at all, so I can't really tell what's the best way to do this, but it seems that having some way to let Arduino export a bit of info useful for third party tools is a good idea. Perhaps @ffissore can comment on what he thinks is the best way to approach this? And, as always, patches welcome :-)

[jantje]

Perhaps @ffissore can comment on what he thinks is the best way to approach this? And, as always, patches welcome :-)

@ffissore
Any update?
My preference goes to a arduino ide conf file (in platform.txt format) containing platform dependent info like compiler paths and so on.
This would allow for default settings and overruling in platform.txt and boards.txt and the boards manager can maintain the file in case of upates or new toolchains.

[ffissore]

Sorry for the delay, @jantje I'll look into this tomorrow

[jantje]

any update?

[ffissore]

Sorry for the late reply. I've just pushed a commit that should at least mitigate your issue
By calling the IDE from the command line with --get-pref and NO pref name, it will dump ALL prefs
Previously --get-pref required the pref name, now it's optional
If you can call the IDE something like once everytime your users open eclipse, or something like that, you'll be able to gather all installed tools locations.
Output is in .properties format: KEY=VALUE
What do you think?

[jantje]

First of all I think: Thanks for taking this seriously.
Secondly I think this is a good proposal. I'll download the latest build and give it a try next week.

[jantje]

I just ran the command and it contains the info I need :-)
Thinking about the future I think it is safe to say I can ignore entries that start with
console.
editor.
export.
last.
platform.
preproc.
proxy.
serial.
upload.

I'm not so sure about this one. @ffissore can you tell me whether you think this could be useful to understand the build in the future?
run.

that currently leaves
build.
runtime.
and some others.

[jantje]

I added all fields for now.
I need some beautifying but it works :-)
Sloeber/arduino-eclipse-plugin@f55d4c3
Seems 1.6.5 will be compatible again :-)
txs
Jantje

[jantje]

I'm sorry but I have to reopen this issue.
I tried this method and I failed because I do not only get "common" information but also the settings based on the "selected" hardware.
More precisely I bumped into

runtime.platform.path=/home/jan/programs/Arduino/arduino-nightly/hardware/arduino/avr
runtime.hardware.path=/home/jan/programs/Arduino/arduino-nightly/hardware/arduino

I get the info for avr (because the last time I used the ide it was with a avr) but now I'm using/testing due.
Would it be possible to filter all global variables? So only the info that is independent of user selections?

[matthijskooijman]

I think you should be able to obtain that by passing in the --board option when calling the IDE to select the right board (e.g. --board arduino.sam.due to s elect the due IIRC). This should only modify the selected board for this IDE run, not change the saved preference.

[jantje]

@matthijskooijman
I tried running

bash-4.1$ ./arduino --get-pref --board arduino.sam.due
Loading configuration...
Initializing packages...
Preparing boards...
bash-4.1$

As you see no result. I alo got a dialog box saying "error"; "can not specify any sketch files"

I tried

bash-4.1$ ./arduino --board arduino:sam:due
bash-4.1$ ./arduino --get-pref

I also tried

bash-4.1$ ./arduino --get-pref --board arduino:sam:due

all to no avail.
May I ask you to check your advice before giving it?

[jantje]

@ffissore @cmaglie @mbanzi
It is hard for me to understand why Arduino opened it's IDE by specifying everything in text files and now the arduino core team added 2 variables that can not be found easily by external parties.

I understand that the board manager is a unexpected and architecturally big change. These changes always come with pain. But why can't the Arduino core team provide a decent fix?
Is it so hard to make a boards managers config file?
Why do I feel like we are playing ping pong? I prefer a constructive discussion an the issue and a solution.
A constructive discussion starts from a set of agreed upon decisions called "architectural decisions".

I would propose the following as an architectural decision.
Based on a limited amount of the configuration files like platform.txt, boards.txt and others unknown at the time of writing; on must be capable of compiling and building a arduino sketch in exact the same way as the arduino IDE. In other words there is no need for: software logic based on folder structure, web pages, command line instructions and others to know the compile and build instructions the arduino IDE uses.
If this is a architectural decision following issues/env vars are non-compliant with this decision (I think this is a complete list):
#2982 (this issue)
#2039
compiler.path
build.system.path
build.architecture

I would put the following under a exception rule (so these need to be defined by the ide and not the files)
runtime.ide.path
runtime.hardware.path
runtime.platform.path
runtime.ide.version
build.path (The path you are using for building)
build.project_name (the base of the name of the output files)

I think it is clear from this list that the current situation is very close to the proposed architectural decision. This does not imply it was ever the intention of the arduino core team to support my proposed architectural decision.
From the point of view of a tool builder it would be great to see this accepted by the arduino core team.

Hoping for the best
Jantje

[ffissore]

@jantje try with arduino --board arduino:avr:sam --get-pref (--get-pref is the last argument)

[cmaglie]

The correct paramater is:

./arduino --board arduino:sam:arduino_due_x --get-pref

arduino is the package
sam is the architecture
arduino_due_x is the board identifier as it is in boards.txt

[ffissore]

yep sorry: I ran the test with arduino:avr:uno and changed it to sam to fit your example, but I changed it the wrong way

[jantje]

Thanks for that. I tested it and it seems to work.
Is there a way to know the last time the boards manager changed something?
This to allow me to be smart about updating "the --get-pref dumps".
A file which is guaranteed to be written to in case of change is OK (if I can use the file date).

[ffissore]

preferences.txt gets rewritten every time the IDE is closed. However, since dumps produced by --get-pref changes with the actual board selected, you'd better run it every time your users update the target board

[jantje]

preferences.txt looks good to me.
When the plugin starts I want to generate all the "dump files" for all the boards used in the workspace.
The dump files are processed (read converted to eclipse environment variables) each time the board gets selected/updated.
If the dump file of the selected board does not yet exist (or the preferences.txt has been updated) I can generate alll the dump files again on the fly.

PS: You may know this but just to be sure: with the plugin you can work on multiple sketches (called projects) each with their own board at the same time.

[ffissore]

I knew and I love it

[ffissore]

Can we then close this issue?

[jantje]

yes

[jantje]

Sorry again.
I'm not sure I need a new defect for this but when I run the command from my home folder it works but from my workspace folder it doesn't.

bash-4.1$ cd "/home/jan/workspaces/eclipse_plugin/runtime arduino.product"
bash-4.1$
bash-4.1$ "/home/jan/programs/Arduino/arduino-nightly/arduino" --board arduino:sam:arduino_due_x --get-pref
Loading configuration...
Initializing packages...
Preparing boards...
bash-4.1$

It may be related to the space in the foldername.

[ffissore]

That was an unrelated bug but I've just pushed a fix. Indeed it was because of the space in the folder name. Get next hourly build

[jantje]

txs :-)

[jantje]

I probably better create a new issue but I'll try this one first ;-)
When the parameter I provide to arduino IDE is incorrect (for instance because I have teensy stuff when upgrading to 1.6.5) I get a dialog box telling me it is wrong.
Can I add a flag so I get no dialog box?