ISSUE-20173: [Bug] Python instance doesn't install dependency from requirements.txt

[sijie]

Original Issue: apache#20173

---

Search before asking

• I searched in the issues and found nothing similar.

Version

master, commit id: 35e9897

Minimal reproduce step

1. package a python function in zip:

❯ tree func
func
├── requirements.txt
└── src
    └── exclamation.py

❯ cat func/requirements.txt
requests==2.28.2

❯ cat func/src/exclamation.py
import requests # just check dependencies is installed

def process(input):
  return str(input) + '!'

❯ zip -r func.zip func
updating: func/ (stored 0%)
updating: func/requirements.txt (stored 0%)
updating: func/src/ (stored 0%)
updating: func/src/exclamation.py (deflated 16%)

2. create some required dir and file:

mkdir -p /pulsar/conf/python-log/ logs/functions && echo "[loggers]
      keys=root

      [handlers]
      keys=stream_handler

      [formatters]
      keys=formatter

      [logger_root]
      level=INFO
      handlers=stream_handler

      [handler_stream_handler]
      class=StreamHandler
      level=INFO
      formatter=formatter
      args=(sys.stdout,)

      [formatter_formatter]
      format=[%(asctime)s] [%(levelname)s] %(filename)s: %(message)s
      datefmt=%Y-%m-%d %H:%M:%S %z" > /pulsar/conf/python-log/python_instance_logging.ini

3. run python instance directly

python /pulsar/instances/python-instance/python_instance_main.py --py /pulsar/download/pulsar_functions/func.zip --logging_directory logs/functions --logging_file exclamation-py-9e1f170e-0 --logging_config_file /pulsar/conf/python-log/python_instance_logging.ini --install_usercode_dependencies true --instance_id 0 --function_id 0-74a988f8-6711-4a9c-9030-a07a54eb2fc4 --function_version 0 --function_details '{"tenant":"public","namespace":"default","name":"exclamation-py-9e1f170e","className":"exclamation","userConfig":"{}","runtime":"PYTHON","autoAck":true,"parallelism":1,"source":{"typeClassName":"[B","inputSpecs":{"persistent://public/default/test-py-input":{}},"cleanupSubscription":true},"sink":{"typeClassName":"[B","topic":"persistent://public/default/test-py-output","producerSpec":{},"forwardSourceMessageProperty":true},"resources":{"cpu":1,"ram":"1073741824"},"componentType":"FUNCTION"}' --pulsar_serviceurl pulsar://localhost:6650 --max_buffered_tuples 100 --port 39093 --metrics_port 39094 --expected_healthcheck_interval -1 --cluster_name sn-platform-pulsar --use_tls false

What did you expect to see?

python instance run successfully

What did you see instead?

[2023-04-24 09:25:45 +0000] [INFO] util.py: Failed to import class exclamation from path download/pulsar_functions
[2023-04-24 09:25:45 +0000] [INFO] util.py: No module named 'requests'
Traceback (most recent call last):
File "/pulsar/instances/python-instance/util.py", line 40, in import_class
return import_class_from_path(from_path, full_class_name)
File "/pulsar/instances/python-instance/util.py", line 60, in import_class_from_path
mod = importlib.import_module(class_name)
File "/usr/lib/python3.8/importlib/init.py", line 127, in import_module
return _bootstrap._gcd_import(name[level:], package, level)
File "", line 1014, in _gcd_import
File "", line 991, in _find_and_load
File "", line 975, in _find_and_load_unlocked
File "", line 671, in _load_unlocked
File "", line 848, in exec_module
File "", line 219, in _call_with_frames_removed
File "/pulsar/download/pulsar_functions/exclamation/src/exclamation.py", line 1, in
import requests # just check dependencies is installed
ModuleNotFoundError: No module named 'requests'

Anything else?

No response

Are you willing to submit a PR?

• I'm willing to submit a PR!