Cannot compile: GitHub API rate limitation

[sylque]

This issue has started today:

Processing debug (platform: https://github.com/pioarduino/platform-espressif32/releases/download/stable/platform-espressif32.zip; board: lolin_d32; framework: arduino)
-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
Latest release Arduino core URL: None
Latest release Arduino core tag: None
TypeError: Traceback (most recent call last):
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\__main__.py", line 103, in main
    cli()  # pylint: disable=no-value-for-parameter
    ^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\click\core.py", line 1157, in __call__
    return self.main(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\click\core.py", line 1078, in main
    rv = self.invoke(ctx)
         ^^^^^^^^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\cli.py", line 85, in invoke
    return super().invoke(ctx)
           ^^^^^^^^^^^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\click\core.py", line 1688, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
                           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\click\core.py", line 1434, in invoke
    return ctx.invoke(self.callback, **ctx.params)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\click\core.py", line 783, in invoke
    return __callback(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\click\decorators.py", line 33, in new_func
    return f(get_current_context(), *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\run\cli.py", line 145, in cli
    process_env(
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\run\cli.py", line 208, in process_env
    ).process()
      ^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\run\processor.py", line 81, in process
    install_project_env_dependencies(
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\package\commands\install.py", line 132, in install_project_env_dependencies
    _install_project_env_platform(project_env, options),
    ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\package\commands\install.py", line 149, in _install_project_env_platform
    PlatformPackageManager().install(
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\package\manager\platform.py", line 60, in install
    p.configure_project_packages(project_env, project_targets)
  File "C:\Users\S\.platformio\penv\Lib\site-packages\platformio\platform\base.py", line 185, in configure_project_packages
    self.configure_default_packages(options, targets or [])
  File "C:\Users\S\.platformio\platforms\espressif32@src-88f34194123b4b05ff6ef6b8e68c7f60\platform.py", line 57, in configure_default_packages
    URL = "https://raw.githubusercontent.com/espressif/arduino-esp32/" + tag + "/package/package_esp32_index.template.json"
          ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~
TypeError: can only concatenate str (not "NoneType") to str

============================================================

An unexpected error occurred. Further steps:

* Verify that you have the latest version of PlatformIO using
  `python -m pip install -U platformio` command

* Try to find answer in FAQ Troubleshooting section
  https://docs.platformio.org/page/faq/index.html

* Report this problem to the developers
  https://github.com/platformio/platformio-core/issues

============================================================

The problem seems to be a rate limitation of the GitHub API. When I enter https://api.github.com/repos/espressif/Arduino-esp32/releases/latest/ in my browser, I get this:

{"message":"API rate limit exceeded for 88.186.41.204. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.)","documentation_url":"https://docs.github.com/rest/overview/resources-in-the-rest-api#rate-limiting"}

[sylque]

The issue doesn't occur with platform = platformio/espressif32.

[TD-er]

Hmm that rate limiting is potentially quite annoying as the rates are quite low when not logged in.
See: https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28

@sylque I think it is hard to tell whether it does or does not occur with the platformio/espressif32 platform as you may hit it 'suddenly' when you reach that limit.
Thus the order of trying and how quick you try is making a difference.

Not sure if PlatformIO does use some authentication token while fetching the data as you can login to PlatformIO using GH, so in theory there could be some token to overcome this rate limit.

[sylque]

Thanks @TD-er .

it is hard to tell whether it does or does not occur with the platformio/espressif32 platform

I know it does not occur with the platformio/espressif32 platform, because platformio/espressif32 compiles fine without an Internet access. Maybe the issue here is not the rate limitation itself, but the fact that pioarduino/platform-espressif32 accesses the Internet at each compilation. See #61.

[TD-er]

Ah check!

[Jason2866]

If you have an other solution to get latest version, a PR is very welcome.
The query is here

platform-espressif32/platform.py

Lines 49 to 59 in 9a07688

	ARDUINO_CORE_API_URL = "https://api.github.com/repos/espressif/Arduino-esp32/releases/latest"
	api_data = requests.get(ARDUINO_CORE_API_URL, timeout=10).json()
	zipball = api_data.get("zipball_url")
	tag = api_data.get("tag_name")
	# print("Latest release Arduino core URL:", zipball)
	self.packages["framework-arduinoespressif32"]["version"] = zipball
	# use corresponding espressif Arduino libs to release
	URL = "https://raw.githubusercontent.com/espressif/arduino-esp32/" + tag + "/package/package_esp32_index.template.json"
	packjdata = requests.get(URL, timeout=10).json()
	dyn_lib_url = packjdata['packages'][0]['tools'][0]['systems'][0]['url']
	self.packages["framework-arduinoespressif32-libs"]["version"] = dyn_lib_url

[Jason2866]

The issue doesn't occur with platform = platformio/espressif32.

Well pioarduino is a fork. See #61

[Jason2866]

With the change done to make offline compile work again this problem will not occur anymore. Still it would be nice to have a "clean" solution for avoiding rate limitations