Use relocatable shebang with space-in-path robustness

[charliermarsh]

Summary

The current shebang seems to fail when the path itself contains spaces. For example:

❯ "/Users/crmarsh/Library/Application Support/uv/python/cpython-3.13.0-macos-aarch64-none/bin/pydoc3"
/Users/crmarsh/Library/Application Support/uv/python/cpython-3.13.0-macos-aarch64-none/bin/pydoc3: line 2: /Users/crmarsh/Library
Support/uv/python/cpython-3.13.0-macos-aarch64-none/bin/python3.13: No such file or directory
/Users/crmarsh/Library/Application Support/uv/python/cpython-3.13.0-macos-aarch64-none/bin/pydoc3: line 2: exec: /Users/crmarsh/Library
Support/uv/python/cpython-3.13.0-macos-aarch64-none/bin/python3.13: cannot execute: No such file or directory

This PR replaces it with the shebang that we use in uv for relocatable environments, which is outlined in the following series of PRs and issues:

• https://github.com/astral-sh/uv/pull/5515/files#r1694358328
• fix(venv): make relocatable activation scripts support ksh uv#5640
• fix(venv.relocatable): script entrypoints should work if symlinked to uv#8079

Closes #394.

Closes astral-sh/uv#9348.

[charliermarsh]

\cc @paveldikov

[paveldikov]

Are you sure ${PYTHON_MAJMIN_VERSION}${PYTHON_BINARY_SUFFIX} will be interpolated within the single quotes?

(I don't have good knowledge of the control flow in here, so genuinely unsure. If the interpolation is indeed broken, you might want to use implicit concatenation e.g. 'foo'"$BAR". I.e. just flip the pythonXY bit to double-quoted.)

[charliermarsh]

Oh good point. Mentally I assumed this was an f-string because in the written-out shebang, it seems to use the expanded versions (but clearly it's not). I'll review, modify, and test prior to merging.

[charliermarsh]

Ok, so the variables did expand, but the rest of the expressions were also evaluated...

#!/bin/sh
'''exec' "."/'python3.9d' "/var/folders/p1/44pzfl0j2m301zzfb0fv0p380000gn/T/tmppqyxz114/build-cpython.sh" ""
' '''

[charliermarsh]

(Still not doing quite the right thing, will revisit soon.)

[charliermarsh]

Success!

#!/bin/sh
'''exec' "$(dirname -- "$(realpath -- "$0")")/python3.11d" "$0" "$@"
' '''

import pydoc
if __name__ == '__main__':
    pydoc.cli()

[cipriancraciun]

Wouldn't it make more sense to have this complicated launcher only if the Python path contains a space?

Because the Python startup time is already slow enough, I don't think having two extra processes (per startup) (dirname and realpath) would help much. (Also it would break if one intends to use this in a minimal container / environment, where the coreutils or equivalent aren't present.)

Now I don't exactly know if this launcher should be relocatable, but if so, I think there can be a way to detect in pure POSIX sh if the $0 contains a space, and if so do this, else just leaving it alone.

---

Also, according to the POSIX specification https://pubs.opengroup.org/onlinepubs/9699919799/utilities/V3_chap02.html#tag_18_06_02 one could use something like the following (not tested, but could be worked out):

#!/bin/sh
''':' ## this is an alias for `true`
_path="${0}"
_path="${_path%/*}" ## remove the smallest suffix like `/*`
exec -- "${_path}/python3.11d" "${0}" "${@}"
' '''

This, should work, and should remove the need for any other additional processes, except the /bin/sh.

[charliermarsh]

I defer to @paveldikov on the shebang itself.

[charliermarsh]

Note however that we can't know if the path will contain a space, since the whole motivation here is that the path is relocatable. The user could put it anywhere!

[cipriancraciun]

[...] we can't know if the path will contain a space [...]

One could use the following simple check (that I believe is POSIX compliant):

## replace all spaces and check if the value differs from the original
_path="${0}"
if test "${_path}" == "${_path// }" ; then
    ## the path does not contain a space
else
    ## the path contains a space
fi

---

However, the other snippet I've given should work, (and after some extra thought about it) perhaps an extra check should be done to see if the path did contain a / or not, as in:

#!/bin/sh
''':'
_path_0="${0}"
_path="${_path_0%/*}"
if test "${_path_0}" == "${_path}" ; then
    ## there was no `/` fould in `_path_0`
    _path='.'
fi
exec -- "${_path}/python3.11d" "${0}" "${@}"
' '''

[zanieb]

Happy to track this as a performance improvement

#397

[charliermarsh]

Success!

❯ "/Users/crmarsh/Library/Application Support/uv/python/python/install/bin/pydoc3"
pydoc - the Python documentation tool

pydoc3 <name> ...
    Show text documentation on something.  <name> may be the name of a
    Python keyword, topic, function, module, or package, or a dotted
    reference to a class or function within a module or module in a
    package.  If <name> contains a '/', it is used as the path to a
    Python source file to document. If name is 'keywords', 'topics',
    or 'modules', a listing of these things is displayed.

pydoc3 -k <keyword>
    Search for a keyword in the synopsis lines of all available modules.

pydoc3 -n <hostname>
    Start an HTTP server with the given hostname (default: localhost).

pydoc3 -p <port>
    Start an HTTP server on the given port on the local machine.  Port
    number 0 can be used to get an arbitrary unused port.

pydoc3 -b
    Start an HTTP server on an arbitrary unused port and open a web browser
    to interactively browse documentation.  This option can be used in
    combination with -n and/or -p.

pydoc3 -w <name> ...
    Write out the HTML documentation for a module to a file in the current
    directory.  If <name> contains a '/', it is treated as a filename; if
    it names a directory, documentation is written for all the contents.

[charliermarsh]

Whereas before:

❯ "/Users/crmarsh/Library/Application Support/uv/python/cpython-3.11.10-macos-aarch64-none/bin/pydoc3"
/Users/crmarsh/Library/Application Support/uv/python/cpython-3.11.10-macos-aarch64-none/bin/pydoc3: line 2: /Users/crmarsh/Library
Support/uv/python/cpython-3.11.10-macos-aarch64-none/bin/python3.11: No such file or directory
/Users/crmarsh/Library/Application Support/uv/python/cpython-3.11.10-macos-aarch64-none/bin/pydoc3: line 2: exec: /Users/crmarsh/Library
Support/uv/python/cpython-3.11.10-macos-aarch64-none/bin/python3.11: cannot execute: No such file or directory