diff --git a/CHANGELOG.md b/CHANGELOG.md index acf3d0eb..63ccb39f 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,14 +7,31 @@ and this project adheres to [Semantic Versioning](http://semver.org/). ## [Unreleased] ### Added +- Special handling of attempts to run the `arduino_ci.rb` CI script against the ruby library instead of an actual Arduino project +- Explicit checks for attemping to test `arduino_ci` itself as if it were a library, resolving a minor annoyance to this developer. +- Code coverage tooling +- Explicit check and warning for library directory names that do not match our guess of what the library should/would be called +- Symlink tests for `Host` ### Changed +- Arduino backend is now `arduino-cli` version `0.13.0` +- `ArduinoCmd` is now `ArduinoBackend` +- `CppLibrary` now relies largely on `ArduinoBackend` instead of making its own judgements about libraries (metadata, includes, and examples) +- `ArduinoBackend` functionality related to `CppLibrary` now lives in `CppLibrary` +- `CppLibrary` now works in an installation-first manner for exposure to `arduino-cli`'s logic -- without installation, there is no ability to reason about libraries +- `CppLibrary` forces just-in-time recursive dependency installation in order to work sensibly +- `ArduinoBackend` maintains the central "best guess" logic on what a library (on disk) might be named ### Deprecated +- `arduino_ci_remote.rb` CLI switch `--skip-compilation` +- Deprecated `arduino_ci_remote.rb` in favor of `arduino_ci.rb` ### Removed +- `ARDUINO_CI_SKIP_SPLASH_SCREEN_RSPEC_TESTS` no longer affects any tests because there are no longer splash screens since switching to `arduino-cli` ### Fixed +- Mismatches between library names in `library.properties` and the directory names, which can cause cryptic failures +- `LibraryProperties` skips over parse errors instead of crashing: only lines with non-empty keys and non-nil values are recorded ### Security diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index 63e4de20..e0a691e7 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -31,7 +31,6 @@ See `SampleProjects/TestSomething/test/*.cpp` for the existing tests (run by rsp To speed up testing by targeting only the files you're working on, you can set several environment variables that `bundle exec rspec` will respond to: -* `ARDUINO_CI_SKIP_SPLASH_SCREEN_RSPEC_TESTS`: if set, this will avoid any rspec test that calls the arduino executable (and as such, causes the splash screen to pop up). * `ARDUINO_CI_SKIP_RUBY_RSPEC_TESTS`: if set, this will skip all tests against ruby code (useful if you are not changing Ruby code). * `ARDUINO_CI_SKIP_CPP_RSPEC_TESTS`: if set, this will skip all tests against the `TestSomething` sample project (useful if you are not changing C++ code). * `ARDUINO_CI_SELECT_CPP_TESTS=`: if set, this will skip all C++ unit tests whose filenames don't match the provided glob (executed in the tests directory) diff --git a/Gemfile b/Gemfile index c88b26e7..995ed3f5 100644 --- a/Gemfile +++ b/Gemfile @@ -4,3 +4,10 @@ git_source(:github) { |repo_name| "https://github.com/#{repo_name}" } # Specify your gem's dependencies in arduino_ci.gemspec gemspec + +gem "bundler", "> 1.15", require: false, group: :test +gem "keepachangelog_manager", "~> 0.0.2", require: false, group: :test +gem "rspec", "~> 3.0", require: false, group: :test +gem 'rubocop', '~>0.59.0', require: false, group: :test +gem 'simplecov', require: false, group: :test +gem 'yard', '~>0.9.11', require: false, group: :test diff --git a/README.md b/README.md index 2a591590..c97ed550 100644 --- a/README.md +++ b/README.md @@ -1,6 +1,6 @@ -# ArduinoCI Ruby gem (`arduino_ci`) -[![Gem Version](https://badge.fury.io/rb/arduino_ci.svg)](https://rubygems.org/gems/arduino_ci) +# ArduinoCI Ruby gem (`arduino_ci`) +[![Gem Version](https://badge.fury.io/rb/arduino_ci.svg)](https://rubygems.org/gems/arduino_ci) [![Documentation](http://img.shields.io/badge/docs-rdoc.info-blue.svg)](http://www.rubydoc.info/gems/arduino_ci/0.4.0) [![Gitter](https://badges.gitter.im/Arduino-CI/arduino_ci.svg)](https://gitter.im/Arduino-CI/arduino_ci?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge) @@ -36,6 +36,9 @@ For a bare-bones example that you can copy from, see [SampleProjects/DoSomething The complete set of C++ unit tests for the `arduino_ci` library itself are in the [SampleProjects/TestSomething](SampleProjects/TestSomething) project. The [test files](SampleProjects/TestSomething/test/) are named after the type of feature being tested. +> Arduino expects all libraries to be in a specific `Arduino/libraries` directory on your system. If your library is elsewhere, `arduino_ci` will _automatically_ create a symbolic link in the `libraries` directory that points to the directory of the project being tested. This simplifieds working with project dependencies, but **it can have unintended consequences on Windows systems** because [in some cases deleting a folder that contains a symbolic link to another folder can cause the _entire linked folder_ to be removed instead of just the link itself](https://superuser.com/a/306618). +> +> If you use a Windows system **it is recommended that you only run `arduino_ci` from project directories that are already inside the `libraries` directory** ### You Need Ruby and Bundler diff --git a/SampleProjects/ExcludeSomething/library.properties b/SampleProjects/ExcludeSomething/library.properties index 1745537f..64065e1f 100644 --- a/SampleProjects/ExcludeSomething/library.properties +++ b/SampleProjects/ExcludeSomething/library.properties @@ -1,4 +1,4 @@ -name=TestSomething +name=ExcludeSomething version=0.1.0 author=Ian Katz maintainer=Ian Katz diff --git a/SampleProjects/NetworkLib/library.properties b/SampleProjects/NetworkLib/library.properties index 2efc89bd..61e2d980 100644 --- a/SampleProjects/NetworkLib/library.properties +++ b/SampleProjects/NetworkLib/library.properties @@ -1,4 +1,4 @@ -name=Ethernet +name=NetworkLib version=0.1.0 author=James Foster maintainer=James Foster diff --git a/arduino_ci.gemspec b/arduino_ci.gemspec index 9b27f138..3227efde 100644 --- a/arduino_ci.gemspec +++ b/arduino_ci.gemspec @@ -27,10 +27,4 @@ Gem::Specification.new do |spec| spec.add_dependency "os", "~> 1.0" spec.add_dependency "rubyzip", "~> 1.2" - - spec.add_development_dependency "bundler", "> 1.15" - spec.add_development_dependency "keepachangelog_manager", "~> 0.0.2" - spec.add_development_dependency "rspec", "~> 3.0" - spec.add_development_dependency 'rubocop', '~>0.59.0' - spec.add_development_dependency 'yard', '~>0.9.11' end diff --git a/exe/arduino_ci.rb b/exe/arduino_ci.rb old mode 100644 new mode 100755 index 4ea2d614..bb8dcfb1 --- a/exe/arduino_ci.rb +++ b/exe/arduino_ci.rb @@ -9,6 +9,7 @@ @failure_count = 0 @passfail = proc { |result| result ? "✓" : "✗" } +@backend = nil # Use some basic parsing to allow command-line overrides of config class Parser @@ -29,11 +30,6 @@ def self.parse(options) output_options[:skip_unittests] = p end - opts.on("--skip-compilation", "Don't compile example sketches (deprecated)") do |p| - puts "The option --skip-compilation has been deprecated in favor of --skip-examples-compilation" - output_options[:skip_compilation] = p - end - opts.on("--skip-examples-compilation", "Don't compile example sketches") do |p| output_options[:skip_compilation] = p end @@ -68,11 +64,11 @@ def self.parse(options) def terminate(final = nil) puts "Failures: #{@failure_count}" unless @failure_count.zero? || final - puts "Last message: #{@arduino_cmd.last_msg}" + puts "Last message: #{@backend.last_msg}" puts "========== Stdout:" - puts @arduino_cmd.last_out + puts @backend.last_out puts "========== Stderr:" - puts @arduino_cmd.last_err + puts @backend.last_err end retcode = @failure_count.zero? ? 0 : 1 exit(retcode) @@ -161,7 +157,7 @@ def file_is_hidden_somewhere?(path) # print out some files def display_files(pathname) # `find` doesn't follow symlinks, so we should instead - realpath = pathname.symlink? ? pathname.readlink : pathname + realpath = Host.symlink?(pathname) ? Host.readlink(pathname) : pathname # suppress directories and dotfile-based things all_files = realpath.find.select(&:file?) @@ -172,25 +168,33 @@ def display_files(pathname) non_hidden.each { |p| puts "#{margin}#{p}" } end -def install_arduino_library_dependencies(aux_libraries) - aux_libraries.each do |l| - if @arduino_cmd.library_present?(l) - inform("Using pre-existing library") { l.to_s } +# @return [Array] The list of installed libraries +def install_arduino_library_dependencies(library_names, on_behalf_of, already_installed = []) + installed = already_installed.clone + library_names.map { |n| @backend.library_of_name(n) }.each do |l| + if installed.include?(l) + # do nothing + elsif l.installed? + inform("Using pre-existing dependency of #{on_behalf_of}") { l.name } else - assure("Installing aux library '#{l}'") { @arduino_cmd.install_library(l) } + assure("Installing dependency of #{on_behalf_of}: '#{l.name}'") do + next nil unless l.install + + l.name + end end + installed << l.name + installed += install_arduino_library_dependencies(l.arduino_library_dependencies, l.name, installed) end + installed end -def perform_unit_tests(file_config) +def perform_unit_tests(cpp_library, file_config) if @cli_options[:skip_unittests] inform("Skipping unit tests") { "as requested via command line" } return end config = file_config.with_override_config(@cli_options[:ci_config]) - cpp_library = ArduinoCI::CppLibrary.new(Pathname.new("."), - @arduino_cmd.lib_dir, - config.exclude_dirs.map(&Pathname.method(:new))) # check GCC compilers = config.compilers_to_use @@ -214,10 +218,25 @@ def perform_unit_tests(file_config) # iterate boards / tests if !cpp_library.tests_dir.exist? - inform_multiline("Skipping unit tests; no tests dir at #{cpp_library.tests_dir}") do - puts " In case that's an error, this is what was found in the library:" - display_files(cpp_library.tests_dir.parent) - true + # alert future me about running the script from the wrong directory, instead of doing the huge file dump + # otherwise, assume that the user might be running the script on a library with no actual unit tests + if Pathname.new(__dir__).parent == Pathname.new(Dir.pwd) + inform_multiline("arduino_ci seems to be trying to test itself") do + [ + "arduino_ci (the ruby gem) isn't an arduino project itself, so running the CI test script against", + "the core library isn't really a valid thing to do... but it's easy for a developer (including the", + "owner) to mistakenly do just that. Hello future me, you probably meant to run this against one of", + "the sample projects in SampleProjects/ ... if not, please submit a bug report; what a wild case!" + ].each { |l| puts " #{l}" } + false + end + exit(1) + else + inform_multiline("Skipping unit tests; no tests dir at #{cpp_library.tests_dir}") do + puts " In case that's an error, this is what was found in the library:" + display_files(cpp_library.tests_dir.parent) + true + end end elsif cpp_library.test_files.empty? inform_multiline("Skipping unit tests; no test files were found in #{cpp_library.tests_dir}") do @@ -228,7 +247,7 @@ def perform_unit_tests(file_config) elsif config.platforms_to_unittest.empty? inform("Skipping unit tests") { "no platforms were requested" } else - install_arduino_library_dependencies(config.aux_libraries_for_unittest) + install_arduino_library_dependencies(config.aux_libraries_for_unittest, "") config.platforms_to_unittest.each do |p| config.allowable_unittest_files(cpp_library.test_files).each do |unittest_path| @@ -256,39 +275,12 @@ def perform_unit_tests(file_config) end end -def perform_compilation_tests(config) +def perform_example_compilation_tests(cpp_library, config) if @cli_options[:skip_compilation] inform("Skipping compilation of examples") { "as requested via command line" } return end - # index the existing libraries - attempt("Indexing libraries") { @arduino_cmd.index_libraries } unless @arduino_cmd.libraries_indexed - - # initialize library under test - installed_library_path = attempt("Installing library under test") do - @arduino_cmd.install_local_library(Pathname.new(".")) - end - - if !installed_library_path.nil? && installed_library_path.exist? - inform("Library installed at") { installed_library_path.to_s } - else - assure_multiline("Library installed successfully") do - if installed_library_path.nil? - puts @arduino_cmd.last_msg - else - # print out the contents of the deepest directory we actually find - @arduino_cmd.lib_dir.ascend do |path_part| - next unless path_part.exist? - - break display_files(path_part) - end - false - end - end - end - library_examples = @arduino_cmd.library_examples(installed_library_path) - # gather up all required boards for compilation so we can install them up front. # start with the "platforms to unittest" and add the examples # while we're doing that, get the aux libraries as well @@ -297,6 +289,7 @@ def perform_compilation_tests(config) aux_libraries = Set.new(config.aux_libraries_for_build) # while collecting the platforms, ensure they're defined + library_examples = cpp_library.example_sketches library_examples.each do |path| ovr_config = config.from_example(path) ovr_config.platforms_to_build.each do |platform| @@ -317,35 +310,36 @@ def perform_compilation_tests(config) # do that, set the URLs, and download the packages all_packages = example_platform_info.values.map { |v| v[:package] }.uniq.reject(&:nil?) + builtin_packages, external_packages = all_packages.partition { |p| config.package_builtin?(p) } + # inform about builtin packages - all_packages.select { |p| config.package_builtin?(p) }.each do |p| + builtin_packages.each do |p| inform("Using built-in board package") { p } end # make sure any non-builtin package has a URL defined - all_packages.reject { |p| config.package_builtin?(p) }.each do |p| + external_packages.each do |p| assure("Board package #{p} has a defined URL") { board_package_url[p] } end # set up all the board manager URLs. # we can safely reject nils now, they would be for the builtins - all_urls = all_packages.map { |p| board_package_url[p] }.uniq.reject(&:nil?) + all_urls = external_packages.map { |p| board_package_url[p] }.uniq.reject(&:nil?) unless all_urls.empty? assure("Setting board manager URLs") do - @arduino_cmd.board_manager_urls = all_urls + @backend.board_manager_urls = all_urls end end - all_packages.each do |p| + external_packages.each do |p| assure("Installing board package #{p}") do - @arduino_cmd.install_boards(p) + @backend.install_boards(p) end end - install_arduino_library_dependencies(aux_libraries) + install_arduino_library_dependencies(aux_libraries, "") - last_board = nil if config.platforms_to_build.empty? inform("Skipping builds") { "no platforms were requested" } return @@ -356,46 +350,58 @@ def perform_compilation_tests(config) return end - attempt("Setting compiler warning level") { @arduino_cmd.set_pref("compiler.warning_level", "all") } - - # switching boards takes time, so iterate board first - # _then_ whichever examples match it - examples_by_platform = library_examples.each_with_object({}) do |example_path, acc| + library_examples.each do |example_path| ovr_config = config.from_example(example_path) ovr_config.platforms_to_build.each do |p| - acc[p] = [] unless acc.key?(p) - acc[p] << example_path - end - end - - examples_by_platform.each do |platform, example_paths| - board = example_platform_info[platform][:board] - assure("Switching to board for #{platform} (#{board})") { @arduino_cmd.use_board(board) } unless last_board == board - last_board = board - - example_paths.each do |example_path| + board = example_platform_info[p][:board] example_name = File.basename(example_path) - attempt("Verifying #{example_name}") do - ret = @arduino_cmd.verify_sketch(example_path) + attempt("Compiling #{example_name} for #{board}") do + ret = @backend.compile_sketch(example_path, board) unless ret puts - puts "Last command: #{@arduino_cmd.last_msg}" - puts @arduino_cmd.last_err + puts "Last command: #{@backend.last_msg}" + puts @backend.last_err end ret end end end - end # initialize command and config config = ArduinoCI::CIConfig.default.from_project_library -@arduino_cmd = ArduinoCI::ArduinoInstallation.autolocate! -inform("Located Arduino binary") { @arduino_cmd.binary_path.to_s } +@backend = ArduinoCI::ArduinoInstallation.autolocate! +inform("Located arduino-cli binary") { @backend.binary_path.to_s } + +# initialize library under test +cpp_library_path = Pathname.new(".") +cpp_library = assure("Installing library under test") do + @backend.install_local_library(cpp_library_path) +end + +assumed_name = @backend.name_of_library(cpp_library_path) +ondisk_name = cpp_library_path.realpath.basename +if assumed_name != ondisk_name + inform("WARNING") { "Installed library named '#{assumed_name}' has directory name '#{ondisk_name}'" } +end + +if !cpp_library.nil? + inform("Library installed at") { cpp_library.path.to_s } +else + # this is a longwinded way of failing, we aren't really "assuring" anything at this point + assure_multiline("Library installed successfully") do + puts @backend.last_msg + false + end +end + +install_arduino_library_dependencies( + cpp_library.arduino_library_dependencies, + "<#{ArduinoCI::CppLibrary::LIBRARY_PROPERTIES_FILE}>" +) -perform_unit_tests(config) -perform_compilation_tests(config) +perform_unit_tests(cpp_library, config) +perform_example_compilation_tests(cpp_library, config) terminate(true) diff --git a/exe/arduino_library_location.rb b/exe/arduino_library_location.rb index c3c8c6de..f11e9fc6 100755 --- a/exe/arduino_library_location.rb +++ b/exe/arduino_library_location.rb @@ -2,6 +2,6 @@ require 'arduino_ci' # locate and/or forcibly install Arduino, keep stdout clean -@arduino_cmd = ArduinoCI::ArduinoInstallation.autolocate!($stderr) +@backend = ArduinoCI::ArduinoInstallation.autolocate!($stderr) -puts @arduino_cmd.lib_dir +puts @backend.lib_dir diff --git a/lib/arduino_ci/arduino_backend.rb b/lib/arduino_ci/arduino_backend.rb new file mode 100644 index 00000000..32d03fd8 --- /dev/null +++ b/lib/arduino_ci/arduino_backend.rb @@ -0,0 +1,218 @@ +require 'fileutils' +require 'pathname' +require 'json' + +# workaround for https://github.com/arduino/Arduino/issues/3535 +WORKAROUND_LIB = "USBHost".freeze + +module ArduinoCI + + # To report errors that we can't resolve or possibly even explain + class ArduinoExecutionError < StandardError; end + + # Wrap the Arduino executable. This requires, in some cases, a faked display. + class ArduinoBackend + + # We never even use this in code, it's just here for reference because the backend is picky about it. Used for testing + # @return [String] the only allowable name for the arduino-cli config file. + CONFIG_FILE_NAME = "arduino-cli.yaml".freeze + + # the actual path to the executable on this platform + # @return [Pathname] + attr_accessor :binary_path + + # If a custom config is deired (i.e. for testing), specify it here. + # Note https://github.com/arduino/arduino-cli/issues/753 : the --config-file option + # is really the director that contains the file + # @return [Pathname] + attr_accessor :config_dir + + # @return [String] STDOUT of the most recently-run command + attr_reader :last_out + + # @return [String] STDERR of the most recently-run command + attr_reader :last_err + + # @return [String] the most recently-run command + attr_reader :last_msg + + # @return [Array] Additional URLs for the boards manager + attr_reader :additional_urls + + def initialize(binary_path) + @binary_path = binary_path + @config_dir = nil + @additional_urls = [] + @last_out = "" + @last_err = "" + @last_msg = "" + end + + def _wrap_run(work_fn, *args, **kwargs) + # do some work to extract & merge environment variables if they exist + has_env = !args.empty? && args[0].class == Hash + env_vars = has_env ? args[0] : {} + actual_args = has_env ? args[1..-1] : args # need to shift over if we extracted args + custom_config = @config_dir.nil? ? [] : ["--config-file", @config_dir.to_s] + full_args = [binary_path.to_s, "--format", "json"] + custom_config + actual_args + full_cmd = env_vars.empty? ? full_args : [env_vars] + full_args + + shell_vars = env_vars.map { |k, v| "#{k}=#{v}" }.join(" ") + @last_msg = " $ #{shell_vars} #{full_args.join(' ')}" + work_fn.call(*full_cmd, **kwargs) + end + + # build and run the arduino command + def run_and_output(*args, **kwargs) + _wrap_run((proc { |*a, **k| Host.run_and_output(*a, **k) }), *args, **kwargs) + end + + # run a command and capture its output + # @return [Hash] {:out => String, :err => String, :success => bool} + def run_and_capture(*args, **kwargs) + ret = _wrap_run((proc { |*a, **k| Host.run_and_capture(*a, **k) }), *args, **kwargs) + @last_err = ret[:err] + @last_out = ret[:out] + ret + end + + def capture_json(*args, **kwargs) + ret = run_and_capture(*args, **kwargs) + ret[:json] = JSON.parse(ret[:out]) + ret + end + + # Get a dump of the entire config + # @return [Hash] The configuration + def config_dump + capture_json("config", "dump")[:json] + end + + # @return [String] the path to the Arduino libraries directory + def lib_dir + Pathname.new(config_dump["directories"]["user"]) + "libraries" + end + + # Board manager URLs + # @return [Array] The additional URLs used by the board manager + def board_manager_urls + config_dump["board_manager"]["additional_urls"] + @additional_urls + end + + # Set board manager URLs + # @return [Array] The additional URLs used by the board manager + def board_manager_urls=(all_urls) + raise ArgumentError("all_urls should be an array, got #{all_urls.class}") unless all_urls.is_a? Array + + @additional_urls = all_urls + end + + # check whether a board is installed + # we do this by just selecting a board. + # the arduino binary will error if unrecognized and do a successful no-op if it's installed + # @param boardname [String] The board to test + # @return [bool] Whether the board is installed + def board_installed?(boardname) + # capture_json("core", "list")[:json].find { |b| b["ID"] == boardname } # nope, this is for the family + run_and_capture("board", "details", "--fqbn", boardname)[:success] + end + + # install a board by name + # @param name [String] the board name + # @return [bool] whether the command succeeded + def install_boards(boardfamily) + result = run_and_capture("core", "install", boardfamily) + result[:success] + end + + # @return [Hash] information about installed libraries via the CLI + def installed_libraries + capture_json("lib", "list")[:json] + end + + # @param path [String] The sketch to compile + # @param boardname [String] The board to use + # @return [bool] whether the command succeeded + def compile_sketch(path, boardname) + ext = File.extname path + unless ext.casecmp(".ino").zero? + @last_msg = "Refusing to compile sketch with '#{ext}' extension -- rename it to '.ino'!" + return false + end + unless File.exist? path + @last_msg = "Can't compile Sketch at nonexistent path '#{path}'!" + return false + end + ret = run_and_capture("compile", "--fqbn", boardname, "--warnings", "all", "--dry-run", path.to_s) + ret[:success] + end + + # Guess the name of a library + # @param path [Pathname] The path to the library (installed or not) + # @return [String] the probable library name + def name_of_library(path) + src_path = path.realpath + properties_file = src_path + CppLibrary::LIBRARY_PROPERTIES_FILE + return src_path.basename.to_s unless properties_file.exist? + return src_path.basename.to_s if LibraryProperties.new(properties_file).name.nil? + + LibraryProperties.new(properties_file).name + end + + # Create a handle to an Arduino library by name + # @param name [String] The library "real name" + # @return [CppLibrary] The library object + def library_of_name(name) + raise ArgumentError, "name is not a String (got #{name.class})" unless name.is_a? String + + CppLibrary.new(name, self) + end + + # Create a handle to an Arduino library by path + # @param path [Pathname] The path to the library + # @return [CppLibrary] The library object + def library_of_path(path) + # the path must exist... and if it does, brute-force search the installed libs for it + realpath = path.realpath # should produce error if the path doesn't exist to begin with + entry = installed_libraries.find { |l| Pathname.new(l["library"]["install_dir"]).realpath == realpath } + probable_name = entry["real_name"].nil? ? realpath.basename.to_s : entry["real_name"] + CppLibrary.new(probable_name, self) + end + + # install a library from a path on the local machine (not via library manager), by symlink or no-op as appropriate + # @param path [Pathname] library to use + # @return [CppLibrary] the installed library, or nil + def install_local_library(path) + src_path = path.realpath + library_name = name_of_library(path) + cpp_library = library_of_name(library_name) + destination_path = cpp_library.path + + # things get weird if the sketchbook contains the library. + # check that first + if cpp_library.installed? + # maybe the project has always lived in the libraries directory, no need to symlink + return cpp_library if destination_path == src_path + + uhoh = "There is already a library '#{library_name}' in the library directory (#{destination_path})" + # maybe it's a symlink? that would be OK + if Host.symlink?(destination_path) + current_destination_target = Host.readlink(destination_path) + return cpp_library if current_destination_target == src_path + + @last_msg = "#{uhoh} and it's symlinked to #{current_destination_target} (expected #{src_path})" + return nil + end + + @last_msg = "#{uhoh}. It may need to be removed manually." + return nil + end + + # install the library + libraries_dir = destination_path.parent + libraries_dir.mkpath unless libraries_dir.exist? + Host.symlink(src_path, destination_path) + cpp_library + end + end +end diff --git a/lib/arduino_ci/arduino_cmd.rb b/lib/arduino_ci/arduino_cmd.rb deleted file mode 100644 index 577617ab..00000000 --- a/lib/arduino_ci/arduino_cmd.rb +++ /dev/null @@ -1,332 +0,0 @@ -require 'fileutils' -require 'pathname' - -# workaround for https://github.com/arduino/Arduino/issues/3535 -WORKAROUND_LIB = "USBHost".freeze - -module ArduinoCI - - # To report errors that we can't resolve or possibly even explain - class ArduinoExecutionError < StandardError; end - - # Wrap the Arduino executable. This requires, in some cases, a faked display. - class ArduinoCmd - - # Enable a shortcut syntax for command line flags - # @param name [String] What the flag will be called (prefixed with 'flag_') - # @return [void] - # @macro [attach] flag - # The text of the command line flag for $1 - # @!attribute [r] flag_$1 - # @return [String] the text of the command line flag (`$2` in this case) - def self.flag(name, text = nil) - text = "(flag #{name} not defined)" if text.nil? - self.class_eval("def flag_#{name};\"#{text}\";end", __FILE__, __LINE__) - end - - # the array of command components to launch the Arduino executable - # @return [Array] - attr_accessor :base_cmd - - # the actual path to the executable on this platform - # @return [Pathname] - attr_accessor :binary_path - - # part of a workaround for https://github.com/arduino/Arduino/issues/3535 - attr_reader :libraries_indexed - - # @return [String] STDOUT of the most recently-run command - attr_reader :last_out - - # @return [String] STDERR of the most recently-run command - attr_reader :last_err - - # @return [String] the most recently-run command - attr_reader :last_msg - - # set the command line flags (undefined for now). - # These vary between gui/cli. Inline comments added for greppability - flag :get_pref # flag_get_pref - flag :set_pref # flag_set_pref - flag :save_prefs # flag_save_prefs - flag :use_board # flag_use_board - flag :install_boards # flag_install_boards - flag :install_library # flag_install_library - flag :verify # flag_verify - - def initialize - @prefs_cache = {} - @prefs_fetched = false - @libraries_indexed = false - @last_out = "" - @last_err = "" - @last_msg = "" - end - - # Convert a preferences dump into a flat hash - # @param arduino_output [String] The raw Arduino executable output - # @return [Hash] preferences as a hash - def parse_pref_string(arduino_output) - lines = arduino_output.split("\n").select { |l| l.include? "=" } - ret = lines.each_with_object({}) do |e, acc| - parts = e.split("=", 2) - acc[parts[0]] = parts[1] - acc - end - ret - end - - # @return [String] the path to the Arduino libraries directory - def lib_dir - Pathname.new(get_pref("sketchbook.path")) + "libraries" - end - - # fetch preferences in their raw form - # @return [String] Preferences as a set of lines - def _prefs_raw - resp = run_and_capture(flag_get_pref) - fail_msg = "Arduino binary failed to operate as expected; you will have to troubleshoot it manually" - raise ArduinoExecutionError, "#{fail_msg}. The command was #{@last_msg}" unless resp[:success] - - @prefs_fetched = true - resp[:out] - end - - # Get the Arduino preferences, from cache if possible - # @return [Hash] The full set of preferences - def prefs - prefs_raw = _prefs_raw unless @prefs_fetched - return nil if prefs_raw.nil? - - @prefs_cache = parse_pref_string(prefs_raw) - @prefs_cache.clone - end - - # get a preference key - # @param key [String] The preferences key to look up - # @return [String] The preference value - def get_pref(key) - data = @prefs_fetched ? @prefs_cache : prefs - data[key] - end - - # underlying preference-setter. - # @param key [String] The preference name - # @param value [String] The value to set to - # @return [bool] whether the command succeeded - def _set_pref(key, value) - run_and_capture(flag_set_pref, "#{key}=#{value}", flag_save_prefs)[:success] - end - - # set a preference key/value pair, and update the cache. - # @param key [String] the preference key - # @param value [String] the preference value - # @return [bool] whether the command succeeded - def set_pref(key, value) - prefs unless @prefs_fetched # update cache first - success = _set_pref(key, value) - @prefs_cache[key] = value if success - success - end - - def _wrap_run(work_fn, *args, **kwargs) - # do some work to extract & merge environment variables if they exist - has_env = !args.empty? && args[0].class == Hash - env_vars = has_env ? args[0] : {} - actual_args = has_env ? args[1..-1] : args # need to shift over if we extracted args - full_args = @base_cmd + actual_args - full_cmd = env_vars.empty? ? full_args : [env_vars] + full_args - - shell_vars = env_vars.map { |k, v| "#{k}=#{v}" }.join(" ") - @last_msg = " $ #{shell_vars} #{full_args.join(' ')}" - work_fn.call(*full_cmd, **kwargs) - end - - # build and run the arduino command - def run_and_output(*args, **kwargs) - _wrap_run((proc { |*a, **k| Host.run_and_output(*a, **k) }), *args, **kwargs) - end - - # run a command and capture its output - # @return [Hash] {:out => String, :err => String, :success => bool} - def run_and_capture(*args, **kwargs) - ret = _wrap_run((proc { |*a, **k| Host.run_and_capture(*a, **k) }), *args, **kwargs) - @last_err = ret[:err] - @last_out = ret[:out] - ret - end - - # Board manager URLs - # @return [Array] The additional URLs used by the board manager - def board_manager_urls - url_list = get_pref("boardsmanager.additional.urls") - return [] if url_list.nil? - - url_list.split(",") - end - - # Set board manager URLs - # @return [Array] The additional URLs used by the board manager - def board_manager_urls=(all_urls) - set_pref("boardsmanager.additional.urls", all_urls.join(",")) - end - - # check whether a board is installed - # we do this by just selecting a board. - # the arduino binary will error if unrecognized and do a successful no-op if it's installed - # @param boardname [String] The board to test - # @return [bool] Whether the board is installed - def board_installed?(boardname) - run_and_capture(flag_use_board, boardname)[:success] - end - - # install a board by name - # @param name [String] the board name - # @return [bool] whether the command succeeded - def install_boards(boardfamily) - # TODO: find out why IO.pipe fails but File::NULL succeeds :( - result = run_and_capture(flag_install_boards, boardfamily) - already_installed = result[:err].include?("Platform is already installed!") - result[:success] || already_installed - end - - # install a library by name - # @param name [String] the library name - # @return [bool] whether the command succeeded - def _install_library(library_name) - result = run_and_capture(flag_install_library, library_name) - - already_installed = result[:err].include?("Library is already installed: #{library_name}") - success = result[:success] || already_installed - - @libraries_indexed = (@libraries_indexed || success) if library_name == WORKAROUND_LIB - success - end - - # index the set of libraries by installing a dummy library - # related to WORKAROUND_LIB and https://github.com/arduino/Arduino/issues/3535 - # TODO: unclear if this is still necessary - def index_libraries - return true if @libraries_indexed - - _install_library(WORKAROUND_LIB) - @libraries_indexed - end - - # install a library by name - # @param name [String] the library name - # @return [bool] whether the command succeeded - def install_library(library_name) - index_libraries - _install_library(library_name) - end - - # generate the (very likely) path of a library given its name - # @param library_name [String] The name of the library - # @return [Pathname] The fully qualified library name - def library_path(library_name) - Pathname.new(lib_dir) + library_name - end - - # Determine whether a library is present in the lib dir - # - # Note that `true` doesn't guarantee that the library is valid/installed - # and `false` doesn't guarantee that the library isn't built-in - # - # @param library_name [String] The name of the library - # @return [bool] - def library_present?(library_name) - library_path(library_name).exist? - end - - # update the library index - # @return [bool] Whether the update succeeded - def update_library_index - # install random lib so the arduino IDE grabs a new library index - # see: https://github.com/arduino/Arduino/issues/3535 - install_library(WORKAROUND_LIB) - end - - # use a particular board for compilation - # @param boardname [String] The board to use - # @return [bool] whether the command succeeded - def use_board(boardname) - run_and_capture(flag_use_board, boardname, flag_save_prefs)[:success] - end - - # use a particular board for compilation, installing it if necessary - # @param boardname [String] The board to use - # @return [bool] whether the command succeeded - def use_board!(boardname) - return true if use_board(boardname) - - boardfamily = boardname.split(":")[0..1].join(":") - puts "Board '#{boardname}' not found; attempting to install '#{boardfamily}'" - return false unless install_boards(boardfamily) # guess board family from first 2 :-separated fields - - use_board(boardname) - end - - # @param path [String] The sketch to verify - # @return [bool] whether the command succeeded - def verify_sketch(path) - ext = File.extname path - unless ext.casecmp(".ino").zero? - @last_msg = "Refusing to verify sketch with '#{ext}' extension -- rename it to '.ino'!" - return false - end - unless File.exist? path - @last_msg = "Can't verify Sketch at nonexistent path '#{path}'!" - return false - end - ret = run_and_capture(flag_verify, path) - ret[:success] - end - - # ensure that the given library is installed, or symlinked as appropriate - # return the path of the prepared library, or nil - # @param path [Pathname] library to use - # @return [String] the path of the installed library - def install_local_library(path) - src_path = path.realpath - library_name = src_path.basename - destination_path = library_path(library_name) - - # things get weird if the sketchbook contains the library. - # check that first - if destination_path.exist? - uhoh = "There is already a library '#{library_name}' in the library directory" - return destination_path if destination_path == src_path - - # maybe it's a symlink? that would be OK - if destination_path.symlink? - return destination_path if destination_path.readlink == src_path - - @last_msg = "#{uhoh} and it's not symlinked to #{src_path}" - return nil - end - - @last_msg = "#{uhoh}. It may need to be removed manually." - return nil - end - - # install the library - Host.symlink(src_path, destination_path) - destination_path - end - - # @param installed_library_path [String] The library to query - # @return [Array] Example sketch files - def library_examples(installed_library_path) - example_path = Pathname.new(installed_library_path) + "examples" - return [] unless File.exist?(example_path) - - examples = example_path.children.select(&:directory?).map(&:to_path).map(&File.method(:basename)) - files = examples.map do |e| - proj_file = example_path + e + "#{e}.ino" - proj_file.exist? ? proj_file.to_s : nil - end - files.reject(&:nil?).sort_by(&:to_s) - end - end -end diff --git a/lib/arduino_ci/arduino_cmd_linux.rb b/lib/arduino_ci/arduino_cmd_linux.rb deleted file mode 100644 index ff6c137a..00000000 --- a/lib/arduino_ci/arduino_cmd_linux.rb +++ /dev/null @@ -1,17 +0,0 @@ -require 'arduino_ci/arduino_cmd' -require 'timeout' - -module ArduinoCI - - # Implementation of Arduino linux IDE commands - class ArduinoCmdLinux < ArduinoCmd - flag :get_pref, "--get-pref" - flag :set_pref, "--pref" - flag :save_prefs, "--save-prefs" - flag :use_board, "--board" - flag :install_boards, "--install-boards" - flag :install_library, "--install-library" - flag :verify, "--verify" - end - -end diff --git a/lib/arduino_ci/arduino_cmd_linux_builder.rb b/lib/arduino_ci/arduino_cmd_linux_builder.rb deleted file mode 100644 index 679c72b6..00000000 --- a/lib/arduino_ci/arduino_cmd_linux_builder.rb +++ /dev/null @@ -1,19 +0,0 @@ -require "arduino_ci/host" -require 'arduino_ci/arduino_cmd' - -module ArduinoCI - - # Implementation of Arduino linux CLI commands - class ArduinoCmdLinuxBuilder < ArduinoCmd - - flag :get_pref, "--get-pref" # apparently doesn't exist - flag :set_pref, "--pref" # apparently doesn't exist - flag :save_prefs, "--save-prefs" # apparently doesn't exist - flag :use_board, "-fqbn" - flag :install_boards, "--install-boards" # apparently doesn't exist - flag :install_library, "--install-library" # apparently doesn't exist - flag :verify, "-compile" - - end - -end diff --git a/lib/arduino_ci/arduino_cmd_osx.rb b/lib/arduino_ci/arduino_cmd_osx.rb deleted file mode 100644 index 196e1453..00000000 --- a/lib/arduino_ci/arduino_cmd_osx.rb +++ /dev/null @@ -1,17 +0,0 @@ -require "arduino_ci/host" -require 'arduino_ci/arduino_cmd' - -module ArduinoCI - - # Implementation of OSX commands - class ArduinoCmdOSX < ArduinoCmd - flag :get_pref, "--get-pref" - flag :set_pref, "--pref" - flag :save_prefs, "--save-prefs" - flag :use_board, "--board" - flag :install_boards, "--install-boards" - flag :install_library, "--install-library" - flag :verify, "--verify" - end - -end diff --git a/lib/arduino_ci/arduino_cmd_windows.rb b/lib/arduino_ci/arduino_cmd_windows.rb deleted file mode 100644 index 3282dfea..00000000 --- a/lib/arduino_ci/arduino_cmd_windows.rb +++ /dev/null @@ -1,17 +0,0 @@ -require "arduino_ci/host" -require 'arduino_ci/arduino_cmd' - -module ArduinoCI - - # Implementation of OSX commands - class ArduinoCmdWindows < ArduinoCmd - flag :get_pref, "--get-pref" - flag :set_pref, "--pref" - flag :save_prefs, "--save-prefs" - flag :use_board, "--board" - flag :install_boards, "--install-boards" - flag :install_library, "--install-library" - flag :verify, "--verify" - end - -end diff --git a/lib/arduino_ci/arduino_downloader.rb b/lib/arduino_ci/arduino_downloader.rb index f619bcf2..12a2d791 100644 --- a/lib/arduino_ci/arduino_downloader.rb +++ b/lib/arduino_ci/arduino_downloader.rb @@ -1,4 +1,5 @@ require 'fileutils' +require 'pathname' require 'net/http' require 'open-uri' require 'zip' @@ -10,10 +11,10 @@ module ArduinoCI # Manage the OS-specific download & install of Arduino class ArduinoDownloader - # @param desired_ide_version [string] Version string e.g. 1.8.7 + # @param desired_version [string] Version string e.g. 1.8.7 # @param output [IO] $stdout, $stderr, File.new(/dev/null, 'w'), etc. where console output will be sent - def initialize(desired_ide_version, output = $stdout) - @desired_ide_version = desired_ide_version + def initialize(desired_version, output = $stdout) + @desired_version = desired_version @output = output end @@ -30,7 +31,7 @@ def prepare # The autolocated executable of the installation # - # @return [string] or nil + # @return [Pathname] or nil def self.autolocated_executable # Arbitrarily, I'm going to pick the force installed location first # if it exists. I'm not sure why we would have both, but if we did @@ -39,70 +40,54 @@ def self.autolocated_executable locations.find { |loc| !loc.nil? && File.exist?(loc) } end - # The autolocated directory of the installation - # - # @return [string] or nil - def self.autolocated_installation - # Arbitrarily, I'm going to pick the force installed location first - # if it exists. I'm not sure why we would have both, but if we did - # a force install then let's make sure we actually use it. - locations = [self.force_install_location, self.existing_installation] - locations.find { |loc| !loc.nil? && File.exist?(loc) } + # The executable Arduino file in an existing installation, or nil + # @return [Pathname] + def self.existing_executable + self.must_implement(__method__) end - # The path to the directory of an existing installation, or nil + # The local file (dir) name of the desired IDE package (zip/tar/etc) # @return [string] - def self.existing_installation - self.must_implement(__method__) + def package_file + self.class.must_implement(__method__) end - # The executable Arduino file in an existing installation, or nil + # The local filename of the extracted IDE package (zip/tar/etc) # @return [string] - def self.existing_executable + def self.extracted_file self.must_implement(__method__) end # The executable Arduino file in a forced installation, or nil - # @return [string] + # @return [Pathname] def self.force_installed_executable - self.must_implement(__method__) + Pathname.new(ENV['HOME']) + self.extracted_file end # The technology that will be used to complete the download # (for logging purposes) # @return [string] - def downloader + def self.downloader "open-uri" end # The technology that will be used to extract the download # (for logging purposes) # @return [string] - def extracter - "Zip" - end - - # The URL of the desired IDE package (zip/tar/etc) for this platform - # @return [string] - def package_url - "https://downloads.arduino.cc/#{package_file}" + def self.extracter + self.must_implement(__method__) end - # The local file (dir) name of the desired IDE package (zip/tar/etc) - # @return [string] - def package_file - self.class.must_implement(__method__) + # Extract the package_file to extracted_file + # @return [bool] whether successful + def self.extract(_package_file) + self.must_implement(__method__) end - # The local filename of the extracted IDE package (zip/tar/etc) + # The URL of the desired IDE package (zip/tar/etc) for this platform # @return [string] - def extracted_file - self.class.must_implement(__method__) - end - - # @return [String] The location where a forced install will go - def self.force_install_location - File.join(ENV['HOME'], 'arduino_ci_ide') + def package_url + "https://github.com/arduino/arduino-cli/releases/download/#{@desired_version}/#{package_file}" end # Download the package_url to package_file @@ -130,26 +115,10 @@ def download @output.puts "\nArduino force-install failed downloading #{package_url}: #{e}" end - # Extract the package_file to extracted_file - # @return [bool] whether successful - def extract - Zip::File.open(package_file) do |zip| - batch_size = [1, (zip.size / 100).to_i].max - dots = 0 - zip.each do |file| - @output.print "." if (dots % batch_size).zero? - file.restore_permissions = true - file.extract { accept_all } - dots += 1 - end - end - end - - # Move the extracted package file from extracted_file to the force_install_location + # Move the extracted package file from extracted_file to the force_installed_executable # @return [bool] whether successful def install - # Move only the content of the directory - FileUtils.mv extracted_file, self.class.force_install_location + FileUtils.mv self.class.extracted_file.to_s, self.class.force_installed_executable.to_s end # Forcibly install Arduino on linux from the web @@ -161,40 +130,40 @@ def execute return false end - arduino_package = "Arduino #{@desired_ide_version} package" + arduino_package = "Arduino #{@desired_version} package" attempts = 0 loop do - if File.exist? package_file - @output.puts "#{arduino_package} seems to have been downloaded already" if attempts.zero? + if File.exist?(package_file) + @output.puts "#{arduino_package} seems to have been downloaded already at #{package_file}" if attempts.zero? break elsif attempts >= DOWNLOAD_ATTEMPTS break @output.puts "After #{DOWNLOAD_ATTEMPTS} attempts, failed to download #{package_url}" else - @output.print "Attempting to download #{arduino_package} with #{downloader}" + @output.print "Attempting to download #{arduino_package} with #{self.class.downloader}" download @output.puts end attempts += 1 end - if File.exist? extracted_file - @output.puts "#{arduino_package} seems to have been extracted already" - elsif File.exist? package_file - @output.print "Extracting archive with #{extracter}" - extract + if File.exist?(self.class.extracted_file) + @output.puts "#{arduino_package} seems to have been extracted already at #{self.class.extracted_file}" + elsif File.exist?(package_file) + @output.print "Extracting archive with #{self.class.extracter}" + self.class.extract(package_file) @output.puts end - if File.exist? self.class.force_install_location - @output.puts "#{arduino_package} seems to have been installed already" - elsif File.exist? extracted_file + if File.exist?(self.class.force_installed_executable) + @output.puts "#{arduino_package} seems to have been installed already at #{self.class.force_installed_executable}" + elsif File.exist?(self.class.extracted_file) install else - @output.puts "Could not find extracted archive (tried #{extracted_file})" + @output.puts "Could not find extracted archive (tried #{self.class.extracted_file})" end - File.exist? self.class.force_install_location + File.exist?(self.class.force_installed_executable) end end diff --git a/lib/arduino_ci/arduino_downloader_linux.rb b/lib/arduino_ci/arduino_downloader_linux.rb index efbc34e2..487273d1 100644 --- a/lib/arduino_ci/arduino_downloader_linux.rb +++ b/lib/arduino_ci/arduino_downloader_linux.rb @@ -1,7 +1,5 @@ require "arduino_ci/arduino_downloader" -USE_BUILDER = false - module ArduinoCI # Manage the linux download & install of Arduino @@ -10,13 +8,25 @@ class ArduinoDownloaderLinux < ArduinoDownloader # The local filename of the desired IDE package (zip/tar/etc) # @return [string] def package_file - "#{extracted_file}-linux64.tar.xz" + "arduino-cli_#{@desired_version}_Linux_64bit.tar.gz" + end + + # The local file (dir) name of the extracted IDE package (zip/tar/etc) + # @return [string] + def self.extracted_file + "arduino-cli" + end + + # The executable Arduino file in an existing installation, or nil + # @return [string] + def self.existing_executable + Host.which("arduino-cli") end # Make any preparations or run any checks prior to making changes # @return [string] Error message, or nil if success def prepare - reqs = [extracter] + reqs = [self.class.extracter] reqs.each do |req| return "#{req} does not appear to be installed!" unless Host.which(req) end @@ -26,62 +36,14 @@ def prepare # The technology that will be used to extract the download # (for logging purposes) # @return [string] - def extracter + def self.extracter "tar" end # Extract the package_file to extracted_file # @return [bool] whether successful - def extract - system(extracter, "xf", package_file) - end - - # The local file (dir) name of the extracted IDE package (zip/tar/etc) - # @return [string] - def extracted_file - "arduino-#{@desired_ide_version}" - end - - # The path to the directory of an existing installation, or nil - # @return [string] - def self.existing_installation - exe = self.existing_executable - return nil if exe.nil? - - File.dirname(exe) # it's not really this - # but for this platform it doesn't really matter - end - - # The executable Arduino file in an existing installation, or nil - # @return [string] - def self.existing_executable - if USE_BUILDER - # builder_name = "arduino-builder" - # cli_place = Host.which(builder_name) - # unless cli_place.nil? - # ret = ArduinoCmdLinuxBuilder.new - # ret.base_cmd = [cli_place] - # return ret - # end - end - Host.which("arduino") - end - - # The executable Arduino file in a forced installation, or nil - # @return [string] - def self.force_installed_executable - if USE_BUILDER - # forced_builder = File.join(ArduinoCmdLinuxBuilder.force_install_location, builder_name) - # if File.exist?(forced_builder) - # ret = ArduinoCmdLinuxBuilder.new - # ret.base_cmd = [forced_builder] - # return ret - # end - end - forced_arduino = File.join(self.force_install_location, "arduino") - return forced_arduino if File.exist? forced_arduino - - nil + def self.extract(package_file) + system(extracter, "xf", package_file, extracted_file) end end diff --git a/lib/arduino_ci/arduino_downloader_osx.rb b/lib/arduino_ci/arduino_downloader_osx.rb index 02ea2347..89890599 100644 --- a/lib/arduino_ci/arduino_downloader_osx.rb +++ b/lib/arduino_ci/arduino_downloader_osx.rb @@ -8,54 +8,42 @@ class ArduinoDownloaderOSX < ArduinoDownloader # The local filename of the desired IDE package (zip/tar/etc) # @return [string] def package_file - "arduino-#{@desired_ide_version}-macosx.zip" + "arduino-cli_#{@desired_version}_macOS_64bit.tar.gz" end # The local file (dir) name of the extracted IDE package (zip/tar/etc) # @return [string] - def extracted_file - "Arduino.app" + def self.extracted_file + "arduino-cli" end - # @return [String] The location where a forced install will go - def self.force_install_location - # include the .app extension - File.join(ENV['HOME'], 'Arduino.app') - end - - # An existing Arduino directory in one of the given directories, or nil - # @param Array a list of places to look + # The executable Arduino file in an existing installation, or nil # @return [string] - def self.find_existing_arduino_dir(paths) - paths.find(&File.method(:exist?)) + def self.existing_executable + Host.which("arduino-cli") end - # An existing Arduino file in one of the given directories, or nil - # @param Array a list of places to look for the executable - # @return [string] - def self.find_existing_arduino_exe(paths) - paths.find do |path| - exe = File.join(path, "MacOS", "Arduino") - File.exist? exe + # Make any preparations or run any checks prior to making changes + # @return [string] Error message, or nil if success + def prepare + reqs = [self.class.extracter] + reqs.each do |req| + return "#{req} does not appear to be installed!" unless Host.which(req) end + nil end - # The path to the directory of an existing installation, or nil + # The technology that will be used to extract the download + # (for logging purposes) # @return [string] - def self.existing_installation - self.find_existing_arduino_dir(["/Applications/Arduino.app"]) + def self.extracter + "tar" end - # The executable Arduino file in an existing installation, or nil - # @return [string] - def self.existing_executable - self.find_existing_arduino_exe(["/Applications/Arduino.app"]) - end - - # The executable Arduino file in a forced installation, or nil - # @return [string] - def self.force_installed_executable - self.find_existing_arduino_exe([self.force_install_location]) + # Extract the package_file to extracted_file + # @return [bool] whether successful + def self.extract(package_file) + system(extracter, "xf", package_file, extracted_file) end end diff --git a/lib/arduino_ci/arduino_downloader_windows.rb b/lib/arduino_ci/arduino_downloader_windows.rb index f2c91669..9b7d1862 100644 --- a/lib/arduino_ci/arduino_downloader_windows.rb +++ b/lib/arduino_ci/arduino_downloader_windows.rb @@ -10,19 +10,6 @@ module ArduinoCI # Manage the POSIX download & install of Arduino class ArduinoDownloaderWindows < ArduinoDownloader - # Make any preparations or run any checks prior to making changes - # @return [string] Error message, or nil if success - def prepare - nil - end - - # The technology that will be used to complete the download - # (for logging purposes) - # @return [string] - def downloader - "open-uri" - end - # Download the package_url to package_file # @return [bool] whether successful def download @@ -35,29 +22,28 @@ def download @output.puts "\nArduino force-install failed downloading #{package_url}: #{e}" end - # Move the extracted package file from extracted_file to the force_install_location - # @return [bool] whether successful - def install - # Move only the content of the directory - FileUtils.mv extracted_file, self.class.force_install_location - end - # The local filename of the desired IDE package (zip/tar/etc) # @return [string] def package_file - "#{extracted_file}-windows.zip" + "arduino-cli_#{@desired_version}_Windows_64bit.zip" + end + + # The executable Arduino file in an existing installation, or nil + # @return [string] + def self.existing_executable + Host.which("arduino-cli") end # The technology that will be used to extract the download # (for logging purposes) # @return [string] - def extracter + def self.extracter "Expand-Archive" end # Extract the package_file to extracted_file # @return [bool] whether successful - def extract + def self.extract(package_file) Zip::File.open(package_file) do |zip| zip.each do |file| file.extract(file.name) @@ -67,36 +53,8 @@ def extract # The local file (dir) name of the extracted IDE package (zip/tar/etc) # @return [string] - def extracted_file - "arduino-#{@desired_ide_version}" - end - - # The path to the directory of an existing installation, or nil - # @return [string] - def self.existing_installation - exe = self.existing_executable - return nil if exe.nil? - - File.dirname(exe) - end - - # The executable Arduino file in an existing installation, or nil - # @return [string] - def self.existing_executable - arduino_reg = 'SOFTWARE\WOW6432Node\Arduino' - Win32::Registry::HKEY_LOCAL_MACHINE.open(arduino_reg).find do |reg| - path = reg.read_s('Install_Dir') - exe = File.join(path, "arduino_debug.exe") - File.exist? exe - end - rescue - nil - end - - # The executable Arduino file in a forced installation, or nil - # @return [string] - def self.force_installed_executable - File.join(self.force_install_location, "arduino_debug.exe") + def self.extracted_file + "arduino-cli.exe" end end diff --git a/lib/arduino_ci/arduino_installation.rb b/lib/arduino_ci/arduino_installation.rb index 9ca32ada..01d45ab2 100644 --- a/lib/arduino_ci/arduino_installation.rb +++ b/lib/arduino_ci/arduino_installation.rb @@ -1,15 +1,11 @@ require 'pathname' require "arduino_ci/host" -require "arduino_ci/arduino_cmd_osx" -require "arduino_ci/arduino_cmd_linux" -require "arduino_ci/arduino_cmd_windows" -require "arduino_ci/arduino_cmd_linux_builder" +require "arduino_ci/arduino_backend" require "arduino_ci/arduino_downloader_osx" require "arduino_ci/arduino_downloader_linux" - require "arduino_ci/arduino_downloader_windows" if ArduinoCI::Host.os == :windows -DESIRED_ARDUINO_IDE_VERSION = "1.8.6".freeze +DESIRED_ARDUINO_CLI_VERSION = "0.13.0".freeze module ArduinoCI @@ -23,80 +19,22 @@ class << self # attempt to find a workable Arduino executable across platforms # # Autolocation assumed to be an expensive operation - # @return [ArduinoCI::ArduinoCmd] an instance of the command or nil if it can't be found + # @return [ArduinoCI::ArduinoBackend] an instance of the command or nil if it can't be found def autolocate - ret = nil - case Host.os - when :osx then - ret = autolocate_osx - when :linux then - loc = ArduinoDownloaderLinux.autolocated_executable - return nil if loc.nil? - - ret = ArduinoCmdLinux.new - ret.base_cmd = [loc] - ret.binary_path = Pathname.new(loc) - when :windows then - loc = ArduinoDownloaderWindows.autolocated_executable - return nil if loc.nil? - - ret = ArduinoCmdWindows.new - ret.base_cmd = [loc] - ret.binary_path = Pathname.new(loc) + downloader_class = case Host.os + when :osx then ArduinoDownloaderOSX + when :linux then ArduinoDownloaderLinux + when :windows then ArduinoDownloaderWindows end - ret - end - # @return [ArduinoCI::ArduinoCmdOSX] an instance of the command or nil if it can't be found - def autolocate_osx - osx_root = ArduinoDownloaderOSX.autolocated_installation - return nil if osx_root.nil? - return nil unless File.exist? osx_root + loc = downloader_class.autolocated_executable + return nil if loc.nil? - launchers = [ - # try a hack that skips splash screen - # from https://github.com/arduino/Arduino/issues/1970#issuecomment-321975809 - [ - "java", - "-cp", - "#{osx_root}/Contents/Java/*", - "-DAPP_DIR=#{osx_root}/Contents/Java", - "-Dfile.encoding=UTF-8", - "-Dapple.awt.UIElement=true", - "-Xms128M", - "-Xmx512M", - "processing.app.Base", - ], - # failsafe way - [File.join(osx_root, "Contents", "MacOS", "Arduino")] - ] - - # create return and find a command launcher that works - ret = ArduinoCmdOSX.new - launchers.each do |launcher| - # test whether this method successfully launches the IDE - # note that "successful launch" involves a command that will fail, - # because that's faster than any command which succeeds. what we - # don't want to see is a java error. - args = launcher + ["--bogus-option"] - result = Host.run_and_capture(*args) - - # NOTE: Was originally searching for "Error: unknown option: --bogus-option" - # but also need to find "Erreur: option inconnue : --bogus-option" - # and who knows how many other languages. - # For now, just search for the end of the error and hope that the java-style - # launch of this won't include a similar string in it - next unless result[:err].include? ": --bogus-option" - - ret.base_cmd = launcher - ret.binary_path = Pathname.new(osx_root) - return ret - end - nil + ArduinoBackend.new(loc) end # Attempt to find a workable Arduino executable across platforms, and install it if we don't - # @return [ArduinoCI::ArduinoCmd] an instance of a command + # @return [ArduinoCI::ArduinoBackend] an instance of a command def autolocate!(output = $stdout) candidate = autolocate return candidate unless candidate.nil? @@ -109,7 +47,7 @@ def autolocate!(output = $stdout) # Forcibly install Arduino from the web # @return [bool] Whether the command succeeded - def force_install(output = $stdout, version = DESIRED_ARDUINO_IDE_VERSION) + def force_install(output = $stdout, version = DESIRED_ARDUINO_CLI_VERSION) worker_class = case Host.os when :osx then ArduinoDownloaderOSX when :windows then ArduinoDownloaderWindows diff --git a/lib/arduino_ci/cpp_library.rb b/lib/arduino_ci/cpp_library.rb index 810d32be..2d481f1b 100644 --- a/lib/arduino_ci/cpp_library.rb +++ b/lib/arduino_ci/cpp_library.rb @@ -14,15 +14,21 @@ module ArduinoCI # Information about an Arduino CPP library, specifically for compilation purposes class CppLibrary - # @return [Pathname] The path to the library being tested - attr_reader :base_dir + # @return [String] The official library properties file name + LIBRARY_PROPERTIES_FILE = "library.properties".freeze - # @return [Pathname] The path to the Arduino 3rd-party library directory - attr_reader :arduino_lib_dir + # @return [String] The "official" name of the library, which can include spaces (in a way that the lib dir won't) + attr_reader :name + + # @return [ArduinoBackend] The backend support for this library + attr_reader :backend # @return [Array] The set of artifacts created by this class (note: incomplete!) attr_reader :artifacts + # @return [Array] The set of directories that should be excluded from compilation + attr_reader :exclude_dirs + # @return [String] STDERR from the last command attr_reader :last_err @@ -35,30 +41,100 @@ class CppLibrary # @return [Array] Directories suspected of being vendor-bundle attr_reader :vendor_bundle_cache - # @param base_dir [Pathname] The path to the library being tested - # @param arduino_lib_dir [Pathname] The path to the libraries directory - # @param exclude_dirs [Array] Directories that should be excluded from compilation - def initialize(base_dir, arduino_lib_dir, exclude_dirs) - raise ArgumentError, 'base_dir is not a Pathname' unless base_dir.is_a? Pathname - raise ArgumentError, 'arduino_lib_dir is not a Pathname' unless arduino_lib_dir.is_a? Pathname - raise ArgumentError, 'exclude_dir is not an array of Pathnames' unless exclude_dirs.is_a?(Array) - raise ArgumentError, 'exclude_dir array contains non-Pathname elements' unless exclude_dirs.all? { |p| p.is_a? Pathname } - - @base_dir = base_dir - @exclude_dirs = exclude_dirs - @arduino_lib_dir = arduino_lib_dir.expand_path + # @param friendly_name [String] The "official" name of the library, which can contain spaces + # @param backend [ArduinoBackend] The support backend + def initialize(friendly_name, backend) + raise ArgumentError, "friendly_name is not a String (got #{friendly_name.class})" unless friendly_name.is_a? String + raise ArgumentError, 'backend is not a ArduinoBackend' unless backend.is_a? ArduinoBackend + + @name = friendly_name + @backend = backend + @info_cache = nil @artifacts = [] @last_err = "" @last_out = "" @last_msg = "" @has_libasan_cache = {} @vendor_bundle_cache = nil + @exclude_dirs = [] + end + + # Generate a guess as to the on-disk (coerced character) name of this library + # + # @TODO: delegate this to the backend in some way? It uses "official" names for install, but dir names in lists :( + # @param friendly_name [String] The library name as it might appear in library manager + # @return [String] How the path will be stored on disk -- spaces are coerced to underscores + def self.library_directory_name(friendly_name) + friendly_name.tr(" ", "_") + end + + # Generate a guess as to the on-disk (coerced character) name of this library + # + # @TODO: delegate this to the backend in some way? It uses "official" names for install, but dir names in lists :( + # @return [String] How the path will be stored on disk -- spaces are coerced to underscores + def name_on_disk + self.class.library_directory_name(@name) + end + + # Get the path to this library, whether or not it exists + # @return [Pathname] The fully qualified library path + def path + @backend.lib_dir + name_on_disk + end + + # Determine whether a library is present in the lib dir + # + # Note that `true` doesn't guarantee that the library is valid/installed + # and `false` doesn't guarantee that the library isn't built-in + # + # @return [bool] + def installed? + path.exist? + end + + # install a library by name + # @param version [String] the version to install + # @param recursive [bool] whether to also install its dependencies + # @return [bool] whether the command succeeded + def install(version = nil, recursive = false) + return true if installed? && !recursive + + fqln = version.nil? ? @name : "#{@name}@#{version}" + result = if recursive + @backend.run_and_capture("lib", "install", fqln) + else + @backend.run_and_capture("lib", "install", "--no-deps", fqln) + end + result[:success] + end + + # information about the library as reported by the backend + # @return [Hash] the metadata object + def info + return nil unless installed? + + # note that if the library isn't found, we're going to do a lot of cache attempts... + if @info_cache.nil? + @info_cache = @backend.installed_libraries.find do |l| + lib_info = l["library"] + Pathname.new(lib_info["install_dir"]).realpath == path.realpath + end + end + + @info_cache + end + + # @param installed_library_path [String] The library to query + # @return [Array] Example sketch files + def example_sketches + reported_dirs = info["library"]["examples"].map(&Pathname::method(:new)) + reported_dirs.map { |e| e + e.basename.sub_ext(".ino") }.select(&:exist?).sort_by(&:to_s) end # The expected path to the library.properties file (i.e. even if it does not exist) # @return [Pathname] def library_properties_path - @base_dir + "library.properties" + path + LIBRARY_PROPERTIES_FILE end # Whether library.properties definitions for this library exist @@ -68,16 +144,29 @@ def library_properties? lib_props.exist? && lib_props.file? end + # Library properties + # @return [LibraryProperties] The library.properties metadata wrapper for this library + def library_properties + return nil unless library_properties? + + LibraryProperties.new(library_properties_path) + end + + # Set directories that should be excluded from compilation + # @param rval [Array] Array of strings or pathnames that will be coerced to pathnames + def exclude_dirs=(rval) + @exclude_dirs = rval.map { |d| d.is_a?(Pathname) ? d : Pathname.new(d) } + end + # Decide whether this is a 1.5-compatible library # - # according to https://arduino.github.io/arduino-cli/latest/library-specification - # - # Should match logic from https://github.com/arduino/arduino-cli/blob/master/arduino/libraries/loader.go + # This should be according to https://arduino.github.io/arduino-cli/latest/library-specification + # but we rely on the cli to decide for us # @return [bool] def one_point_five? return false unless library_properties? - src_dir = (@base_dir + "src") + src_dir = path + "src" src_dir.exist? && src_dir.directory? end @@ -88,9 +177,9 @@ def one_point_five? # That gets us the vendor directory (or multiple directories). We can check # if the given path is contained by any of those. # - # @param path [Pathname] The path to check + # @param some_path [Pathname] The path to check # @return [bool] - def vendor_bundle?(path) + def vendor_bundle?(some_path) # Cache bundle information, as it is (1) time consuming to fetch and (2) not going to change while we run if @vendor_bundle_cache.nil? bundle_info = Host.run_and_capture("bundle show --paths") @@ -125,7 +214,7 @@ def vendor_bundle?(path) # With vendor bundles located, check this file against those @vendor_bundle_cache.any? do |gem_path| - path.ascend do |part| + some_path.ascend do |part| break true if gem_path == part end end @@ -135,13 +224,13 @@ def vendor_bundle?(path) # # @param path [Pathname] The path to check # @return [bool] - def in_tests_dir?(path) + def in_tests_dir?(sourcefile_path) return false unless tests_dir.exist? tests_dir_aliases = [tests_dir, tests_dir.realpath] # we could do this but some rubies don't return an enumerator for ascend # path.ascend.any? { |part| tests_dir_aliases.include?(part) } - path.ascend do |part| + sourcefile_path.ascend do |part| return true if tests_dir_aliases.include?(part) end false @@ -151,11 +240,11 @@ def in_tests_dir?(path) # # @param path [Pathname] The path to check # @return [bool] - def in_exclude_dir?(path) + def in_exclude_dir?(sourcefile_path) # we could do this but some rubies don't return an enumerator for ascend # path.ascend.any? { |part| tests_dir_aliases.include?(part) } - path.ascend do |part| - return true if exclude_dir.any? { |p| p.realpath == part } + sourcefile_path.ascend do |part| + return true if exclude_dir.any? { |p| p.realpath == part.realpath } end false end @@ -166,42 +255,26 @@ def in_exclude_dir?(path) # @param gcc_binary [String] def libasan?(gcc_binary) unless @has_libasan_cache.key?(gcc_binary) - file = Tempfile.new(["arduino_ci_libasan_check", ".cpp"]) - begin + Tempfile.create(["arduino_ci_libasan_check", ".cpp"]) do |file| file.write "int main(){}" file.close @has_libasan_cache[gcc_binary] = run_gcc(gcc_binary, "-o", "/dev/null", "-fsanitize=address", file.path) - ensure - file.delete end end @has_libasan_cache[gcc_binary] end - # Library properties - def library_properties - return nil unless library_properties? - - LibraryProperties.new(library_properties_path) - end - - # Get a list of all dependencies as defined in library.properties - # @return [Array] The library names of the dependencies (not the paths) - def arduino_library_dependencies - return nil unless library_properties? - - library_properties.depends - end - # Get a list of all CPP source files in a directory and its subdirectories # @param some_dir [Pathname] The directory in which to begin the search # @param extensions [Array] The set of allowable file extensions # @return [Array] The paths of the found files def code_files_in(some_dir, extensions) raise ArgumentError, 'some_dir is not a Pathname' unless some_dir.is_a? Pathname - return [] unless some_dir.exist? && some_dir.directory? - files = some_dir.realpath.children.reject(&:directory?) + full_dir = path + some_dir + return [] unless full_dir.exist? && full_dir.directory? + + files = full_dir.children.reject(&:directory?) cpp = files.select { |path| extensions.include?(path.extname.downcase) } not_hidden = cpp.reject { |path| path.basename.to_s.start_with?(".") } not_hidden.sort_by(&:to_s) @@ -215,17 +288,18 @@ def code_files_in_recursive(some_dir, extensions) raise ArgumentError, 'some_dir is not a Pathname' unless some_dir.is_a? Pathname return [] unless some_dir.exist? && some_dir.directory? - real = some_dir.realpath - Find.find(real).map { |p| Pathname.new(p) }.select(&:directory?).map { |d| code_files_in(d, extensions) }.flatten + Find.find(some_dir).map { |p| Pathname.new(p) }.select(&:directory?).map { |d| code_files_in(d, extensions) }.flatten end - # Header files that are part of the project library under test + # Source files that are part of the library under test + # @param extensions [Array] the allowed extensions (or, the ones we're looking for) # @return [Array] - def header_files + def source_files(extensions) + source_dir = Pathname.new(info["library"]["source_dir"]) ret = if one_point_five? - code_files_in_recursive(@base_dir + "src", HPP_EXTENSIONS) + code_files_in_recursive(source_dir, extensions) else - [@base_dir, @base_dir + "utility"].map { |d| code_files_in(d, HPP_EXTENSIONS) }.flatten + [source_dir, source_dir + "utility"].map { |d| code_files_in(d, extensions) }.flatten end # note to future troubleshooter: some of these tests may not be relevant, but at the moment at @@ -233,18 +307,16 @@ def header_files ret.reject { |p| vendor_bundle?(p) || in_tests_dir?(p) || in_exclude_dir?(p) } end + # Header files that are part of the project library under test + # @return [Array] + def header_files + source_files(HPP_EXTENSIONS) + end + # CPP files that are part of the project library under test # @return [Array] def cpp_files - ret = if one_point_five? - code_files_in_recursive(@base_dir + "src", CPP_EXTENSIONS) - else - [@base_dir, @base_dir + "utility"].map { |d| code_files_in(d, CPP_EXTENSIONS) }.flatten - end - - # note to future troubleshooter: some of these tests may not be relevant, but at the moment at - # least some of them are tied to existing features - ret.reject { |p| vendor_bundle?(p) || in_tests_dir?(p) || in_exclude_dir?(p) } + source_files(CPP_EXTENSIONS) end # CPP files that are part of the arduino mock library we're providing @@ -269,13 +341,13 @@ def cpp_files_libraries(aux_libraries) # Returns the Pathnames for all paths to exclude from testing and compilation # @return [Array] def exclude_dir - @exclude_dirs.map { |p| Pathname.new(@base_dir) + p }.select(&:exist?) + @exclude_dirs.map { |p| Pathname.new(path) + p }.select(&:exist?) end # The directory where we expect to find unit test defintions provided by the user # @return [Pathname] def tests_dir - Pathname.new(@base_dir) + "test" + Pathname.new(path) + "test" end # The files provided by the user that contain unit tests @@ -311,18 +383,33 @@ def gcc_version(gcc_binary) @last_err end - # Arduino library directories containing sources -- only those of the dependencies - # @return [Array] - def arduino_library_src_dirs(aux_libraries) + # Get a list of all dependencies as defined in library.properties + # @return [Array] The library names of the dependencies (not the paths) + def arduino_library_dependencies + return [] unless library_properties? + return [] if library_properties.depends.nil? + + library_properties.depends + end + + # Arduino library dependencies all the way down, installing if they are not present + # @return [Array] The library names of the dependencies (not the paths) + def all_arduino_library_dependencies!(additional_libraries = []) # Pull in all possible places that headers could live, according to the spec: # https://github.com/arduino/Arduino/wiki/Arduino-IDE-1.5:-Library-specification + recursive = (additional_libraries + arduino_library_dependencies).map do |n| + other_lib = self.class.new(n, @backend) + other_lib.install unless other_lib.installed? + other_lib.all_arduino_library_dependencies! + end.flatten + ret = (additional_libraries + recursive).uniq + ret + end - aux_libraries.map do |d| - # library manager coerces spaces in package names to underscores - # see https://github.com/ianfixes/arduino_ci/issues/132#issuecomment-518857059 - legal_dir = d.tr(" ", "_") - self.class.new(@arduino_lib_dir + legal_dir, @arduino_lib_dir, @exclude_dirs).header_dirs - end.flatten.uniq + # Arduino library directories containing sources -- only those of the dependencies + # @return [Array] + def arduino_library_src_dirs(aux_libraries) + all_arduino_library_dependencies!(aux_libraries).map { |l| self.class.new(l, @backend).header_dirs }.flatten.uniq end # GCC command line arguments for including aux libraries @@ -415,9 +502,9 @@ def build_for_test_with_configuration(test_file, aux_libraries, gcc_binary, ci_g # combine library.properties defs (if existing) with config file. # TODO: as much as I'd like to rely only on the properties file(s), I think that would prevent testing 1.0-spec libs - full_aux_libraries = arduino_library_dependencies.nil? ? aux_libraries : aux_libraries + arduino_library_dependencies - arg_sets << test_args(full_aux_libraries, ci_gcc_config) - arg_sets << cpp_files_libraries(full_aux_libraries).map(&:to_s) + full_dependencies = all_arduino_library_dependencies!(aux_libraries) + arg_sets << test_args(full_dependencies, ci_gcc_config) + arg_sets << cpp_files_libraries(full_dependencies).map(&:to_s) arg_sets << [test_file.to_s] args = arg_sets.flatten(1) return nil unless run_gcc(gcc_binary, *args) diff --git a/lib/arduino_ci/host.rb b/lib/arduino_ci/host.rb index 6882cff4..6d175bc4 100644 --- a/lib/arduino_ci/host.rb +++ b/lib/arduino_ci/host.rb @@ -6,6 +6,13 @@ module ArduinoCI # Tools for interacting with the host machine class Host + # TODO: this came from https://stackoverflow.com/a/22716582/2063546 + # and I'm not sure if it can be replaced by self.os == :windows + WINDOWS_VARIANT_REGEX = /mswin32|cygwin|mingw|bccwin/ + + # e.g. 11/27/2020 01:02 AM ExcludeSomething [C:\projects\arduino-ci\SampleProjects\ExcludeSomething] + DIR_SYMLINK_REGEX = %r{\d+/\d+/\d+\s+[^<]+\s+(.*) \[([^\]]+)\]} + # Cross-platform way of finding an executable in the $PATH. # via https://stackoverflow.com/a/5471032/2063546 # which('ruby') #=> /usr/bin/ruby @@ -38,21 +45,69 @@ def self.os return :windows if OS.windows? end + # Cross-platform symlinking # if on windows, call mklink, else self.symlink # @param [Pathname] old_path # @param [Pathname] new_path def self.symlink(old_path, new_path) - return FileUtils.ln_s(old_path.to_s, new_path.to_s) unless RUBY_PLATFORM =~ /mswin32|cygwin|mingw|bccwin/ + # we would prefer `new_path.make_symlink(old_path)` but "symlink function is unimplemented on this machine" with windows + return new_path.make_symlink(old_path) unless needs_symlink_hack? - # https://stackoverflow.com/a/22716582/2063546 + # via https://stackoverflow.com/a/22716582/2063546 # windows mklink syntax is reverse of unix ln -s # windows mklink is built into cmd.exe # vulnerable to command injection, but okay because this is a hack to make a cli tool work. - orp = old_path.realpath.to_s.tr("/", "\\") # HACK DUE TO REALPATH BUG where it - np = new_path.to_s.tr("/", "\\") # still joins windows paths with '/' + orp = pathname_to_windows(old_path.realpath) + np = pathname_to_windows(new_path) _stdout, _stderr, exitstatus = Open3.capture3('cmd.exe', "/C mklink /D #{np} #{orp}") exitstatus.success? end + + # Hack for "realpath" which on windows joins paths with slashes instead of backslashes + # @param path [Pathname] the path to render + # @return [String] A path that will work on windows + def self.pathname_to_windows(path) + path.to_s.tr("/", "\\") + end + + # Hack for "realpath" which on windows joins paths with slashes instead of backslashes + # @param str [String] the windows path + # @return [Pathname] A path that will be recognized by pathname + def self.windows_to_pathname(str) + Pathname.new(str.tr("\\", "/")) + end + + # Whether this OS requires a hack for symlinks + # @return [bool] + def self.needs_symlink_hack? + RUBY_PLATFORM =~ WINDOWS_VARIANT_REGEX + end + + # Cross-platform is-this-a-symlink function + # @param [Pathname] path + # @return [bool] Whether the file is a symlink + def self.symlink?(path) + return path.symlink? unless needs_symlink_hack? + + !readlink(path).nil? + end + + # Cross-platform "read link" function + # @param [Pathname] path + # @return [Pathname] the link target + def self.readlink(path) + return path.readlink unless needs_symlink_hack? + + the_dir = pathname_to_windows(path.parent) + the_file = path.basename.to_s + + stdout, _stderr, _exitstatus = Open3.capture3('cmd.exe', "/c dir /al #{the_dir}") + symlinks = stdout.lines.map { |l| DIR_SYMLINK_REGEX.match(l) }.compact + our_link = symlinks.find { |m| m[1] == the_file } + return nil if our_link.nil? + + windows_to_pathname(our_link[2]) + end end end diff --git a/lib/arduino_ci/library_properties.rb b/lib/arduino_ci/library_properties.rb index 1a080713..a28c6937 100644 --- a/lib/arduino_ci/library_properties.rb +++ b/lib/arduino_ci/library_properties.rb @@ -11,12 +11,22 @@ class LibraryProperties # @param path [Pathname] The path to the library.properties file def initialize(path) @fields = {} - File.foreach(path) do |line| + File.foreach(path) do |line_with_delim| + line = line_with_delim.chomp parts = line.split("=", 2) - @fields[parts[0]] = parts[1].chomp unless parts.empty? + next if parts[0].nil? + next if parts[0].empty? + next if parts[1].nil? + + @fields[parts[0]] = parts[1] unless parts[1].empty? end end + # @return [Hash] the properties as a hash, all strings + def to_h + @fields.clone + end + # Enable a shortcut syntax for library property accessors, in the style of `attr_accessor` metaprogramming. # This is used to create a named field pointing to a specific property in the file, optionally applying # a specific formatting function. diff --git a/spec/arduino_cmd_spec.rb b/spec/arduino_backend_spec.rb similarity index 51% rename from spec/arduino_cmd_spec.rb rename to spec/arduino_backend_spec.rb index 1b098694..1fbfa97e 100644 --- a/spec/arduino_cmd_spec.rb +++ b/spec/arduino_backend_spec.rb @@ -6,38 +6,37 @@ def get_sketch(dir, file) end -RSpec.describe ArduinoCI::ArduinoCmd do +RSpec.describe ArduinoCI::ArduinoBackend do next if skip_ruby_tests - next if skip_splash_screen_tests - arduino_cmd = ArduinoCI::ArduinoInstallation.autolocate! + backend = ArduinoCI::ArduinoInstallation.autolocate! after(:each) do |example| if example.exception - puts "Last message: #{arduino_cmd.last_msg}" + puts "Last message: #{backend.last_msg}" puts "========== Stdout:" - puts arduino_cmd.last_out + puts backend.last_out puts "========== Stderr:" - puts arduino_cmd.last_err + puts backend.last_err end end context "initialize" do it "sets base vars" do - expect(arduino_cmd.base_cmd).not_to be nil - expect(arduino_cmd.prefs.class).to be Hash + expect(backend.binary_path).not_to be nil end end context "board_installed?" do it "Finds installed boards" do - uno_installed = arduino_cmd.board_installed? "arduino:avr:uno" + backend.install_boards("arduino:avr") # we used to assume this was installed... not the case for arduino-cli + uno_installed = backend.board_installed? "arduino:avr:uno" expect(uno_installed).to be true expect(uno_installed).not_to be nil end it "Doesn't find bogus boards" do - bogus_installed = arduino_cmd.board_installed? "eggs:milk:wheat" + bogus_installed = backend.board_installed? "eggs:milk:wheat" expect(bogus_installed).to be false expect(bogus_installed).not_to be nil end @@ -45,47 +44,37 @@ def get_sketch(dir, file) context "installation of boards" do it "installs and sets boards" do - expect(arduino_cmd.install_boards("arduino:sam")).to be true - expect(arduino_cmd.use_board("arduino:sam:arduino_due_x")).to be true + expect(backend.install_boards("arduino:sam")).to be true end end context "libraries" do it "knows where to find libraries" do - fake_lib = "_____nope" - expected_dir = Pathname.new(arduino_cmd.lib_dir) + fake_lib - expect(arduino_cmd.library_path(fake_lib)).to eq(expected_dir) - expect(arduino_cmd.library_present?(fake_lib)).to be false + fake_lib_name = "_____nope" + expected_dir = Pathname.new(backend.lib_dir) + fake_lib_name + fake_lib = backend.library_of_name(fake_lib_name) + expect(fake_lib.path).to eq(expected_dir) + expect(fake_lib.installed?).to be false end end - context "set_pref" do - - it "Sets key to what it was before" do - upload_verify = arduino_cmd.get_pref("upload.verify") - result = arduino_cmd.set_pref("upload.verify", upload_verify) - expect(result).to be true - end - end - - context "board_manager" do it "Reads and writes board_manager URLs" do fake_urls = ["http://foo.bar", "http://arduino.ci"] - existing_urls = arduino_cmd.board_manager_urls + existing_urls = backend.board_manager_urls # try to ensure maxiumum variability in the test test_url_sets = (existing_urls.empty? ? [fake_urls, []] : [[], fake_urls]) + [existing_urls] test_url_sets.each do |urls| - arduino_cmd.board_manager_urls = urls - expect(arduino_cmd.board_manager_urls).to match_array(urls) + backend.board_manager_urls = urls + expect(backend.board_manager_urls).to match_array(urls) end end end - context "verify_sketch" do + context "compile_sketch" do sketch_path_ino = get_sketch("FakeSketch", "FakeSketch.ino") sketch_path_pde = get_sketch("FakeSketch", "FakeSketch.pde") @@ -93,19 +82,19 @@ def get_sketch(dir, file) sketch_path_bad = get_sketch("BadSketch", "BadSketch.ino") it "Rejects a PDE sketch at #{sketch_path_pde}" do - expect(arduino_cmd.verify_sketch(sketch_path_pde)).to be false + expect(backend.compile_sketch(sketch_path_pde, "arduino:avr:uno")).to be false end it "Fails a missing sketch at #{sketch_path_mia}" do - expect(arduino_cmd.verify_sketch(sketch_path_mia)).to be false + expect(backend.compile_sketch(sketch_path_mia, "arduino:avr:uno")).to be false end it "Fails a bad sketch at #{sketch_path_bad}" do - expect(arduino_cmd.verify_sketch(sketch_path_bad)).to be false + expect(backend.compile_sketch(sketch_path_bad, "arduino:avr:uno")).to be false end it "Passes a simple INO sketch at #{sketch_path_ino}" do - expect(arduino_cmd.verify_sketch(sketch_path_ino)).to be true + expect(backend.compile_sketch(sketch_path_ino, "arduino:avr:uno")).to be true end end end diff --git a/spec/arduino_downloader_spec.rb b/spec/arduino_downloader_spec.rb index 442b2bfc..9158b439 100644 --- a/spec/arduino_downloader_spec.rb +++ b/spec/arduino_downloader_spec.rb @@ -7,18 +7,15 @@ it "has correct class properties" do ad = ArduinoCI::ArduinoDownloader - expect{ad.autolocated_executable}.to raise_error(NotImplementedError) - expect{ad.autolocated_installation}.to raise_error(NotImplementedError) - expect{ad.existing_installation}.to raise_error(NotImplementedError) expect{ad.existing_executable}.to raise_error(NotImplementedError) - expect{ad.force_installed_executable}.to raise_error(NotImplementedError) - expect(ad.force_install_location).to eq(File.join(ENV['HOME'], 'arduino_ci_ide')) + expect{ad.extracted_file}.to raise_error(NotImplementedError) + expect{ad.extracter}.to raise_error(NotImplementedError) + expect{ad.extract("foo")}.to raise_error(NotImplementedError) end it "has correct instance properties" do ad = ArduinoCI::ArduinoDownloader.new(DESIRED_VERSION) expect(ad.prepare).to be nil - expect{ad.package_url}.to raise_error(NotImplementedError) expect{ad.package_file}.to raise_error(NotImplementedError) end end @@ -29,23 +26,20 @@ context "Basics" do it "has correct class properties" do ad = ArduinoCI::ArduinoDownloaderLinux - # these will vary with CI. Don't test them. - # expect(ad.autolocated_executable).to be nil - # expect(ad.autolocated_installation).to be nil - # expect(ad.existing_installation).to be nil + # these can vary with CI. Don't test them. # expect(ad.existing_executable).to be nil + # expect(ad.autolocated_executable).to be nil # expect(ad.force_installed_executable).to be nil - expect(ad.force_install_location).to eq(File.join(ENV['HOME'], 'arduino_ci_ide')) + expect(ad.downloader).to eq("open-uri") + expect(ad.extracter).to eq("tar") end it "has correct instance properties" do ad = ArduinoCI::ArduinoDownloaderLinux.new(DESIRED_VERSION) expect(ad.prepare).to be nil - expect(ad.downloader).to eq("open-uri") - expect(ad.extracter).to eq("tar") - expect(ad.package_url).to eq("https://downloads.arduino.cc/arduino-rhubarb-linux64.tar.xz") - expect(ad.package_file).to eq("arduino-rhubarb-linux64.tar.xz") + expect(ad.package_url).to eq("https://github.com/arduino/arduino-cli/releases/download/rhubarb/arduino-cli_rhubarb_Linux_64bit.tar.gz") + expect(ad.package_file).to eq("arduino-cli_rhubarb_Linux_64bit.tar.gz") end end end @@ -55,23 +49,48 @@ context "Basics" do it "has correct class properties" do ad = ArduinoCI::ArduinoDownloaderOSX - # these will vary with CI. Don't test them. - # expect(ad.autolocated_executable).to be nil - # expect(ad.autolocated_installation).to be nil - # expect(ad.existing_installation).to be nil + # these can vary with CI. Don't test them. # expect(ad.existing_executable).to be nil + # expect(ad.autolocated_executable).to be nil # expect(ad.force_installed_executable).to be nil - expect(ad.force_install_location).to eq(File.join(ENV['HOME'], 'Arduino.app')) + expect(ad.downloader).to eq("open-uri") + expect(ad.extracter).to eq("tar") end it "has correct instance properties" do ad = ArduinoCI::ArduinoDownloaderOSX.new(DESIRED_VERSION) expect(ad.prepare).to be nil - expect(ad.downloader).to eq("open-uri") - expect(ad.extracter).to eq("Zip") - expect(ad.package_url).to eq("https://downloads.arduino.cc/arduino-rhubarb-macosx.zip") - expect(ad.package_file).to eq("arduino-rhubarb-macosx.zip") + expect(ad.package_url).to eq("https://github.com/arduino/arduino-cli/releases/download/rhubarb/arduino-cli_rhubarb_macOS_64bit.tar.gz") + expect(ad.package_file).to eq("arduino-cli_rhubarb_macOS_64bit.tar.gz") + end + end +end + + +if ArduinoCI::Host.os == :windows + RSpec.describe ArduinoCI::ArduinoDownloaderWindows do + next if skip_ruby_tests + context "Basics" do + it "has correct class properties" do + ad = ArduinoCI::ArduinoDownloaderWindows + # these will vary with CI. Don't test them. + # expect(ad.autolocated_executable).to be nil + # expect(ad.existing_executable).to be nil + # expect(ad.force_installed_executable).to be nil + + expect(ad.downloader).to eq("open-uri") + expect(ad.extracter).to eq("Expand-Archive") + end + + it "has correct instance properties" do + ad = ArduinoCI::ArduinoDownloaderWindows.new(DESIRED_VERSION) + expect(ad.prepare).to be nil + expect(ad.package_url).to eq("https://github.com/arduino/arduino-cli/releases/download/rhubarb/arduino-cli_rhubarb_Windows_64bit.zip") + expect(ad.package_file).to eq("arduino-cli_rhubarb_Windows_64bit.zip") + end end end + + end diff --git a/spec/arduino_installation_spec.rb b/spec/arduino_installation_spec.rb index 551dc47f..f11bdaab 100644 --- a/spec/arduino_installation_spec.rb +++ b/spec/arduino_installation_spec.rb @@ -2,7 +2,6 @@ RSpec.describe ArduinoCI::ArduinoInstallation do next if skip_ruby_tests - next if skip_splash_screen_tests context "autolocate" do it "doesn't fail" do @@ -11,10 +10,10 @@ end context "autolocate!" do - arduino_cmd = ArduinoCI::ArduinoInstallation.autolocate! + backend = ArduinoCI::ArduinoInstallation.autolocate! it "doesn't fail" do - expect(arduino_cmd.base_cmd).not_to be nil - expect(arduino_cmd.lib_dir).not_to be nil + expect(backend.binary_path).not_to be nil + expect(backend.lib_dir).not_to be nil end end @@ -24,11 +23,10 @@ output.rewind expect(output.read.empty?).to be true # install a bogus version to save time downloading - arduino_cmd = ArduinoCI::ArduinoInstallation.force_install(output, "BOGUS VERSION") + backend = ArduinoCI::ArduinoInstallation.force_install(output, "BOGUS VERSION") output.rewind expect(output.read.empty?).to be false end end end - diff --git a/spec/ci_config_spec.rb b/spec/ci_config_spec.rb index 23d01e40..8ae3431a 100644 --- a/spec/ci_config_spec.rb +++ b/spec/ci_config_spec.rb @@ -1,6 +1,8 @@ require "spec_helper" require "pathname" +require "fake_lib_dir" + RSpec.describe ArduinoCI::CIConfig do next if skip_ruby_tests context "default" do @@ -156,11 +158,20 @@ end context "allowable_unittest_files" do + + # we will need to install some dummy libraries into a fake location, so do that on demand + fld = FakeLibDir.new + backend = fld.backend cpp_lib_path = Pathname.new(__dir__) + "fake_library" - cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, Pathname.new("my_fake_arduino_lib_dir"), []) + + around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) } + before(:each) { @cpp_library = backend.install_local_library(cpp_lib_path) } it "starts with a known set of files" do - expect(cpp_library.test_files.map { |f| File.basename(f) }).to match_array([ + expect(cpp_lib_path.exist?).to be(true) + expect(@cpp_library).to_not be(nil) + expect(@cpp_library.path.exist?).to be(true) + expect(@cpp_library.test_files.map(&:basename).map(&:to_s)).to match_array([ "sam-squamsh.cpp", "yes-good.cpp", "mars.cpp" @@ -168,9 +179,19 @@ end it "filters that set of files" do + expect(cpp_lib_path.exist?).to be(true) + expect(@cpp_library).to_not be(nil) + expect(@cpp_library.test_files.map(&:basename).map(&:to_s)).to match_array([ + "sam-squamsh.cpp", + "yes-good.cpp", + "mars.cpp" + ]) + override_file = File.join(File.dirname(__FILE__), "yaml", "o1.yaml") combined_config = ArduinoCI::CIConfig.default.with_override(override_file) - expect(combined_config.allowable_unittest_files(cpp_library.test_files).map { |f| File.basename(f) }).to match_array([ + expect(combined_config.unittest_info[:testfiles][:select]).to match_array(["*-*.*"]) + expect(combined_config.unittest_info[:testfiles][:reject]).to match_array(["sam-squamsh.*"]) + expect(combined_config.allowable_unittest_files(@cpp_library.test_files).map(&:basename).map(&:to_s)).to match_array([ "yes-good.cpp", ]) end diff --git a/spec/cpp_library_spec.rb b/spec/cpp_library_spec.rb index 6a25468f..48bae5bf 100644 --- a/spec/cpp_library_spec.rb +++ b/spec/cpp_library_spec.rb @@ -1,59 +1,70 @@ require "spec_helper" require "pathname" +require 'tmpdir' + +require 'fake_lib_dir' sampleproj_path = Pathname.new(__dir__).parent + "SampleProjects" -def get_relative_dir(sampleprojects_tests_dir) - base_dir = sampleprojects_tests_dir.ascend do |path| - break path if path.split[1].to_s == "SampleProjects" - end - sampleprojects_tests_dir.relative_path_from(base_dir) +def verified_install(backend, path) + ret = backend.install_local_library(path) + raise "backend.install_local_library from '#{path}' failed: #{backend.last_msg}" if ret.nil? + ret end - RSpec.describe "ExcludeSomething C++" do next if skip_cpp_tests - cpp_lib_path = sampleproj_path + "ExcludeSomething" + # we will need to install some dummy libraries into a fake location, so do that on demand + fld = FakeLibDir.new + backend = fld.backend + test_lib_name = "ExcludeSomething" + cpp_lib_path = sampleproj_path + test_lib_name + around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) } + before(:each) do + @base_dir = fld.libraries_dir + @cpp_library = verified_install(backend, cpp_lib_path) + end + context "without excludes" do - cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, - Pathname.new("my_fake_arduino_lib_dir"), - []) context "cpp_files" do it "finds cpp files in directory" do + expect(@cpp_library).to_not be(nil) excludesomething_cpp_files = [ Pathname.new("ExcludeSomething/src/exclude-something.cpp"), Pathname.new("ExcludeSomething/src/excludeThis/exclude-this.cpp") ] - relative_paths = cpp_library.cpp_files.map { |f| get_relative_dir(f) } + relative_paths = @cpp_library.cpp_files.map { |f| f.relative_path_from(@base_dir) } expect(relative_paths).to match_array(excludesomething_cpp_files) end end context "unit tests" do it "can't build due to files that should have been excluded" do - config = ArduinoCI::CIConfig.default.from_example(cpp_lib_path) - path = config.allowable_unittest_files(cpp_library.test_files).first - compiler = config.compilers_to_use.first - result = cpp_library.build_for_test_with_configuration(path, - [], - compiler, - config.gcc_config("uno")) + @cpp_library = verified_install(backend, cpp_lib_path) + config = ArduinoCI::CIConfig.default.from_example(cpp_lib_path) + path = config.allowable_unittest_files(@cpp_library.test_files).first + compiler = config.compilers_to_use.first + result = @cpp_library.build_for_test_with_configuration(path, + [], + compiler, + config.gcc_config("uno")) expect(result).to be nil end end end context "with excludes" do - cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, - Pathname.new("my_fake_arduino_lib_dir"), - ["src/excludeThis"].map(&Pathname.method(:new))) + context "cpp_files" do it "finds cpp files in directory" do + @cpp_library = verified_install(backend, cpp_lib_path) + @cpp_library.exclude_dirs = ["src/excludeThis"].map(&Pathname.method(:new)) + excludesomething_cpp_files = [ Pathname.new("ExcludeSomething/src/exclude-something.cpp") ] - relative_paths = cpp_library.cpp_files.map { |f| get_relative_dir(f) } + relative_paths = @cpp_library.cpp_files.map { |f| f.relative_path_from(@base_dir) } expect(relative_paths).to match_array(excludesomething_cpp_files) end end @@ -68,6 +79,7 @@ def get_relative_dir(sampleprojects_tests_dir) answers = { DoSomething: { one_five: false, + library_properties: true, cpp_files: [Pathname.new("DoSomething") + "do-something.cpp"], cpp_files_libraries: [], header_dirs: [Pathname.new("DoSomething")], @@ -80,6 +92,7 @@ def get_relative_dir(sampleprojects_tests_dir) }, OnePointOhDummy: { one_five: false, + library_properties: false, cpp_files: [ "OnePointOhDummy/YesBase.cpp", "OnePointOhDummy/utility/YesUtil.cpp", @@ -96,6 +109,7 @@ def get_relative_dir(sampleprojects_tests_dir) }, OnePointFiveMalformed: { one_five: false, + library_properties: false, cpp_files: [ "OnePointFiveMalformed/YesBase.cpp", "OnePointFiveMalformed/utility/YesUtil.cpp", @@ -110,6 +124,7 @@ def get_relative_dir(sampleprojects_tests_dir) }, OnePointFiveDummy: { one_five: true, + library_properties: true, cpp_files: [ "OnePointFiveDummy/src/YesSrc.cpp", "OnePointFiveDummy/src/subdir/YesSubdir.cpp", @@ -129,6 +144,7 @@ def get_relative_dir(sampleprojects_tests_dir) # easier to construct this one from the other test cases answers[:DependOnSomething] = { one_five: true, + library_properties: true, cpp_files: ["DependOnSomething/src/YesDeps.cpp"].map { |f| Pathname.new(f) }, cpp_files_libraries: answers[:OnePointOhDummy][:cpp_files] + answers[:OnePointFiveDummy][:cpp_files], header_dirs: ["DependOnSomething/src"].map { |f| Pathname.new(f) }, # this is not recursive! @@ -141,32 +157,52 @@ def get_relative_dir(sampleprojects_tests_dir) answers.freeze answers.each do |sampleproject, expected| + + # we will need to install some dummy libraries into a fake location, so do that on demand + fld = FakeLibDir.new + backend = fld.backend + context "#{sampleproject}" do cpp_lib_path = sampleproj_path + sampleproject.to_s - cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, sampleproj_path, []) - dependencies = cpp_library.arduino_library_dependencies.nil? ? [] : cpp_library.arduino_library_dependencies + around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) } + before(:each) do + @base_dir = fld.libraries_dir + @cpp_library = verified_install(backend, cpp_lib_path) + end + + it "is a sane test env" do + expect(sampleproject.to_s).to eq(@cpp_library.name) + end it "detects 1.5 format" do - expect(cpp_library.one_point_five?).to eq(expected[:one_five]) + expect(@cpp_library.one_point_five?).to eq(expected[:one_five]) + end + + it "detects library.properties" do + expect(@cpp_library.library_properties?).to eq(expected[:library_properties]) end + context "cpp_files" do it "finds cpp files in directory" do - relative_paths = cpp_library.cpp_files.map { |f| get_relative_dir(f) } + relative_paths = @cpp_library.cpp_files.map { |f| f.relative_path_from(@base_dir) } expect(relative_paths.map(&:to_s)).to match_array(expected[:cpp_files].map(&:to_s)) end end context "cpp_files_libraries" do it "finds cpp files in directories of dependencies" do - relative_paths = cpp_library.cpp_files_libraries(dependencies).map { |f| get_relative_dir(f) } + @cpp_library.all_arduino_library_dependencies! # side effect: installs them + dependencies = @cpp_library.arduino_library_dependencies.nil? ? [] : @cpp_library.arduino_library_dependencies + dependencies.each { |d| verified_install(backend, sampleproj_path + d) } + relative_paths = @cpp_library.cpp_files_libraries(dependencies).map { |f| f.relative_path_from(@base_dir) } expect(relative_paths.map(&:to_s)).to match_array(expected[:cpp_files_libraries].map(&:to_s)) end end context "header_dirs" do it "finds directories containing h files" do - relative_paths = cpp_library.header_dirs.map { |f| get_relative_dir(f) } + relative_paths = @cpp_library.header_dirs.map { |f| f.relative_path_from(@base_dir) } expect(relative_paths.map(&:to_s)).to match_array(expected[:header_dirs].map(&:to_s)) end end @@ -176,14 +212,14 @@ def get_relative_dir(sampleprojects_tests_dir) # since we don't know where the CI system will install this stuff, # we need to go looking for a relative path to the SampleProjects directory # just to get our "expected" value - relative_path = get_relative_dir(cpp_library.tests_dir) + relative_path = @cpp_library.tests_dir.relative_path_from(@base_dir) expect(relative_path.to_s).to eq("#{sampleproject}/test") end end context "test_files" do it "finds cpp files in directory" do - relative_paths = cpp_library.test_files.map { |f| get_relative_dir(f) } + relative_paths = @cpp_library.test_files.map { |f| f.relative_path_from(@base_dir) } expect(relative_paths.map(&:to_s)).to match_array(expected[:test_files].map(&:to_s)) end end @@ -191,7 +227,9 @@ def get_relative_dir(sampleprojects_tests_dir) context "arduino_library_src_dirs" do it "finds src dirs from dependent libraries" do # we explicitly feed in the internal dependencies - relative_paths = cpp_library.arduino_library_src_dirs(dependencies).map { |f| get_relative_dir(f) } + dependencies = @cpp_library.arduino_library_dependencies.nil? ? [] : @cpp_library.arduino_library_dependencies + dependencies.each { |d| verified_install(backend, sampleproj_path + d) } + relative_paths = @cpp_library.arduino_library_src_dirs(dependencies).map { |f| f.relative_path_from(@base_dir) } expect(relative_paths.map(&:to_s)).to match_array(expected[:arduino_library_src_dirs].map(&:to_s)) end end @@ -199,33 +237,39 @@ def get_relative_dir(sampleprojects_tests_dir) end context "test" do + + # we will need to install some dummy libraries into a fake location, so do that on demand + fld = FakeLibDir.new + backend = fld.backend cpp_lib_path = sampleproj_path + "DoSomething" - cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, Pathname.new("my_fake_arduino_lib_dir"), []) config = ArduinoCI::CIConfig.default + around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) } + before(:each) { @cpp_library = verified_install(backend, cpp_lib_path) } + after(:each) do |example| if example.exception - puts "Last command: #{cpp_library.last_cmd}" + puts "Last command: #{@cpp_library.last_cmd}" puts "========== Stdout:" - puts cpp_library.last_out + puts @cpp_library.last_out puts "========== Stderr:" - puts cpp_library.last_err + puts @cpp_library.last_err end end it "is going to test more than one library" do - test_files = cpp_library.test_files + test_files = @cpp_library.test_files expect(test_files.empty?).to be false end - test_files = cpp_library.test_files + test_files = Pathname.glob(Pathname.new(cpp_lib_path) + "test" + "*.cpp") test_files.each do |path| expected = path.basename.to_s.include?("good") config.compilers_to_use.each do |compiler| it "tests #{File.basename(path)} with #{compiler} expecting #{expected}" do - exe = cpp_library.build_for_test_with_configuration(path, [], compiler, config.gcc_config("uno")) + exe = @cpp_library.build_for_test_with_configuration(path, [], compiler, config.gcc_config("uno")) expect(exe).not_to be nil - expect(cpp_library.run_test_file(exe)).to eq(expected) + expect(@cpp_library.run_test_file(exe)).to eq(expected) end end end diff --git a/spec/fake_lib_dir.rb b/spec/fake_lib_dir.rb new file mode 100644 index 00000000..2c613213 --- /dev/null +++ b/spec/fake_lib_dir.rb @@ -0,0 +1,56 @@ +require "arduino_ci" + +class FakeLibDir + + attr_reader :config_dir + attr_reader :config_file + attr_reader :backend + attr_reader :arduino_dir + attr_reader :libraries_dir + + def initialize + # we will need to install some dummy libraries into a fake location, so do that on demand + @config_dir = Pathname.new(Dir.pwd).realpath + @config_file = @config_dir + ArduinoCI::ArduinoBackend::CONFIG_FILE_NAME + @backend = ArduinoCI::ArduinoInstallation.autolocate! + @backend.config_dir = @config_dir + end + + # designed to be called by rspec's "around" function + def in_pristine_fake_libraries_dir(example) + d = Dir.mktmpdir + begin + # write a yaml file containing the current directory + dummy_config = { "directories" => { "user" => d.to_s } } + @arduino_dir = Pathname.new(d) + @libraries_dir = @arduino_dir + "libraries" + Dir.mkdir(@libraries_dir) + + f = File.open(@config_file, "w") + begin + f.write dummy_config.to_yaml + f.close + example.run + ensure + begin + File.unlink(@config_file) + rescue Errno::ENOENT + # cool, already done + end + end + ensure + if ArduinoCI::Host.needs_symlink_hack? + stdout, stderr, exitstatus = Open3.capture3('cmd.exe', "/c rmdir /s /q #{ArduinoCI::Host.pathname_to_windows(d)}") + unless exitstatus.success? + puts "====== rmdir of #{d} failed" + puts stdout + puts stderr + end + else + FileUtils.remove_entry(d) + end + end + end + + +end diff --git a/spec/host_spec.rb b/spec/host_spec.rb new file mode 100644 index 00000000..f3523b1e --- /dev/null +++ b/spec/host_spec.rb @@ -0,0 +1,53 @@ +require "spec_helper" +require 'tmpdir' + + +def idempotent_delete(path) + path.delete +rescue Errno::ENOENT +end + +# creates a dir at then deletes it after block executes +# this will DESTROY any existing entry at that location in the filesystem +def with_tmpdir(path) + begin + idempotent_delete(path) + path.mkpath + yield + ensure + idempotent_delete(path) + end +end + + +RSpec.describe ArduinoCI::Host do + next if skip_ruby_tests + + context "symlinks" do + it "creates symlinks that we agree are symlinks" do + our_dir = Pathname.new(__dir__) + foo_dir = our_dir + "foo_dir" + bar_dir = our_dir + "bar_dir" + + with_tmpdir(foo_dir) do + foo_dir.unlink # we just want to place something at this location + expect(foo_dir.exist?).to be_falsey + + with_tmpdir(bar_dir) do + expect(bar_dir.exist?).to be_truthy + expect(bar_dir.symlink?).to be_falsey + + ArduinoCI::Host.symlink(bar_dir, foo_dir) + expect(ArduinoCI::Host.symlink?(bar_dir)).to be_falsey + expect(ArduinoCI::Host.symlink?(foo_dir)).to be_truthy + expect(ArduinoCI::Host.readlink(foo_dir).realpath).to eq(bar_dir.realpath) + end + end + + expect(foo_dir.exist?).to be_falsey + expect(bar_dir.exist?).to be_falsey + + end + end + +end diff --git a/spec/library_properties_spec.rb b/spec/library_properties_spec.rb index 3c6de1ee..84797077 100644 --- a/spec/library_properties_spec.rb +++ b/spec/library_properties_spec.rb @@ -3,7 +3,7 @@ RSpec.describe ArduinoCI::LibraryProperties do context "property extraction" do - library_properties = ArduinoCI::LibraryProperties.new(Pathname.new(__dir__) + "properties/example.library.properties") + library_properties = ArduinoCI::LibraryProperties.new(Pathname.new(__dir__) + "properties" + "example.library.properties") expected = { string: { @@ -36,10 +36,34 @@ end end + it "reads full_paragraph" do + expect(library_properties.full_paragraph).to eq ([ + expected[:string][:sentence], + expected[:string][:paragraph] + ].join(" ")) + end + it "doesn't crash on nonexistent fields" do expect(library_properties.dot_a_linkage).to be(nil) end end + context "Input handling" do + malformed_examples = [ + "extra_blank_line.library.properties", + "just_equals.library.properties", + "no_equals.library.properties", + "no_key.library.properties", + "no_value.library.properties", + ].map { |e| Pathname.new(__dir__) + "properties" + e } + + malformed_examples.each do |e| + quirk = e.basename.to_s.split(".library.").first + it "reads a properties file with #{quirk}" do + expect { ArduinoCI::LibraryProperties.new(e) }.to_not raise_error + end + end + end + end diff --git a/spec/properties/extra_blank_line.library.properties b/spec/properties/extra_blank_line.library.properties new file mode 100644 index 00000000..fd790211 --- /dev/null +++ b/spec/properties/extra_blank_line.library.properties @@ -0,0 +1,3 @@ +name=ExtraBlank + +sentence=We put the blank line in the middle so overzealous text editors dont trim it diff --git a/spec/properties/just_equals.library.properties b/spec/properties/just_equals.library.properties new file mode 100644 index 00000000..70f9c290 --- /dev/null +++ b/spec/properties/just_equals.library.properties @@ -0,0 +1,3 @@ +name=JustEquals +sentence=Bad file with just an equals on a line += diff --git a/spec/properties/no_equals.library.properties b/spec/properties/no_equals.library.properties new file mode 100644 index 00000000..0a25cb32 --- /dev/null +++ b/spec/properties/no_equals.library.properties @@ -0,0 +1,3 @@ +name=NoEquals +sentence=Bad file with no equals on a line +wat diff --git a/spec/properties/no_key.library.properties b/spec/properties/no_key.library.properties new file mode 100644 index 00000000..abdc161d --- /dev/null +++ b/spec/properties/no_key.library.properties @@ -0,0 +1,3 @@ +name=NoKey +sentence=Bad file with no key on a line +=profit diff --git a/spec/properties/no_value.library.properties b/spec/properties/no_value.library.properties new file mode 100644 index 00000000..8f30e099 --- /dev/null +++ b/spec/properties/no_value.library.properties @@ -0,0 +1,3 @@ +name=NoValue +sentence=Bad file with no value on a line +seriously_why_do_we_even_have_this_line= diff --git a/spec/spec_helper.rb b/spec/spec_helper.rb index eb6f5631..b79e2d07 100644 --- a/spec/spec_helper.rb +++ b/spec/spec_helper.rb @@ -1,3 +1,7 @@ +require 'simplecov' +SimpleCov.start do + add_filter %r{^/spec/} +end require "bundler/setup" require "arduino_ci" @@ -13,10 +17,6 @@ end end -def skip_splash_screen_tests - !ENV["ARDUINO_CI_SKIP_SPLASH_SCREEN_RSPEC_TESTS"].nil? -end - def skip_ruby_tests !ENV["ARDUINO_CI_SKIP_RUBY_RSPEC_TESTS"].nil? end diff --git a/spec/testsomething_unittests_spec.rb b/spec/testsomething_unittests_spec.rb index 4cb49541..c882399f 100644 --- a/spec/testsomething_unittests_spec.rb +++ b/spec/testsomething_unittests_spec.rb @@ -1,40 +1,48 @@ require "spec_helper" require "pathname" -sampleproj_path = Pathname.new(__dir__).parent + "SampleProjects" - -def get_relative_dir(sampleprojects_tests_dir) - base_dir = sampleprojects_tests_dir.ascend do |path| - break path if path.split[1].to_s == "SampleProjects" - end - sampleprojects_tests_dir.relative_path_from(base_dir) -end +require 'fake_lib_dir' +sampleproj_path = Pathname.new(__dir__).parent + "SampleProjects" RSpec.describe "TestSomething C++" do next if skip_cpp_tests - cpp_lib_path = sampleproj_path + "TestSomething" - cpp_library = ArduinoCI::CppLibrary.new(cpp_lib_path, - Pathname.new("my_fake_arduino_lib_dir"), - ["src/excludeThis"].map(&Pathname.method(:new))) + + # we will need to install some dummy libraries into a fake location, so do that on demand + fld = FakeLibDir.new + backend = fld.backend + test_lib_name = "TestSomething" + cpp_lib_path = sampleproj_path + test_lib_name + context "cpp_files" do + around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) } + before(:each) do + @base_dir = fld.libraries_dir + @cpp_library = backend.install_local_library(cpp_lib_path) + end + it "finds cpp files in directory" do testsomething_cpp_files = [Pathname.new("TestSomething/src/test-something.cpp")] - relative_paths = cpp_library.cpp_files.map { |f| get_relative_dir(f) } + relative_paths = @cpp_library.cpp_files.map { |f| f.relative_path_from(@base_dir) } expect(relative_paths).to match_array(testsomething_cpp_files) end end config = ArduinoCI::CIConfig.default.from_example(cpp_lib_path) context "unit tests" do + around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) } + before(:each) do + @base_dir = fld.libraries_dir + @cpp_library = backend.install_local_library(cpp_lib_path) + end it "is going to test more than one library" do - test_files = cpp_library.test_files + test_files = @cpp_library.test_files expect(test_files.empty?).to be false end it "has some allowable test files" do - allowed_files = config.allowable_unittest_files(cpp_library.test_files) + allowed_files = config.allowable_unittest_files(@cpp_library.test_files) expect(allowed_files.empty?).to be false end @@ -46,11 +54,12 @@ def get_relative_dir(sampleprojects_tests_dir) expect(config.platforms_to_unittest.length.zero?).to be(false) end + cpp_library = backend.install_local_library(cpp_lib_path) test_files = config.allowable_unittest_files(cpp_library.test_files) # filter the list based on a glob, if provided unless ENV["ARDUINO_CI_SELECT_CPP_TESTS"].nil? - Dir.chdir(cpp_library.tests_dir) do + Dir.chdir(@cpp_library.tests_dir) do globbed = Pathname.glob(ENV["ARDUINO_CI_SELECT_CPP_TESTS"]) test_files.select! { |p| globbed.include?(p.basename) } end @@ -61,19 +70,21 @@ def get_relative_dir(sampleprojects_tests_dir) config.compilers_to_use.each do |compiler| context "file #{tfn} (using #{compiler})" do + around(:example) { |example| fld.in_pristine_fake_libraries_dir(example) } before(:all) do - @exe = cpp_library.build_for_test_with_configuration(path, [], compiler, config.gcc_config("uno")) + @cpp_library = backend.install_local_library(cpp_lib_path) + @exe = @cpp_library.build_for_test_with_configuration(path, [], compiler, config.gcc_config("uno")) end # extra debug for c++ failures after(:each) do |example| if example.exception - puts "Last command: #{cpp_library.last_cmd}" + puts "Last command: #{@cpp_library.last_cmd}" puts "========== Stdout:" - puts cpp_library.last_out + puts @cpp_library.last_out puts "========== Stderr:" - puts cpp_library.last_err + puts @cpp_library.last_err end end @@ -82,7 +93,7 @@ def get_relative_dir(sampleprojects_tests_dir) end it "#{tfn} passes tests" do skip "Can't run the test program because it failed to build" if @exe.nil? - expect(cpp_library.run_test_file(@exe)).to_not be_falsey + expect(@cpp_library.run_test_file(@exe)).to_not be_falsey end end end