ERROR - Internal Python error in the inspect module.

[datascientistlin]

Hi team,

I have encountered the following error when launching a job in databricks.

ERROR - Internal Python error in the inspect module.

This was run in a spark_python_task from dbx with following commands

dbx deploy --job=E2E-test -e test --files-only
dbx launch --job=E2E-test -e test --as-run-submit --trace

The error was actually generated after the full python file has been executed. Here I used the framework from kedro, so I could see that all tasks in the pipeline has already been completed.

Content of the python file is pretty simple:

from project.__main__ import main

main(["--pipeline", "de"])

Below is the full error message:

2022-08-23 09:17:32,091 - kedro.runner.sequential_runner - INFO - Completed 4 out of 4 task
2022-08-23 09:17:32,091 - kedro.runner.sequential_runner - INFO - Pipeline execution completed successfully.
2022-08-23 09:17:32,091 - root - ERROR - Internal Python error in the inspect module.
Below is the traceback from this internal error.

Traceback (most recent call last):
File "/databricks/python/lib/python3.8/site-packages/click/core.py", line 1065, in main
ctx.exit()
File "/databricks/python/lib/python3.8/site-packages/click/core.py", line 687, in exit
raise Exit(code)
click.exceptions.Exit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/databricks/python/lib/python3.8/site-packages/IPython/core/interactiveshell.py", line 3437, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "", line 14, in
exec(compile(f.read(), filename, 'exec'))
File "/tmp/tmpywm076jl.py", line 10, in
main(["--pipeline", "de"])
File "/databricks/python/lib/python3.8/site-packages/ata_psd/main.py", line 47, in main
run(*args, **kwargs)
File "/databricks/python/lib/python3.8/site-packages/click/core.py", line 1130, in call
return self.main(*args, **kwargs)
File "/databricks/python/lib/python3.8/site-packages/click/core.py", line 1083, in main
sys.exit(e.exit_code)
SystemExit: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py", line 1101, in get_records
return _fixed_getinnerframes(etb, number_of_lines_of_context, tb_offset)
File "/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py", line 248, in wrapped
return f(*args, **kwargs)
File "/databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py", line 281, in _fixed_getinnerframes
records = fix_frame_records_filenames(inspect.getinnerframes(etb, context))
File "/usr/lib/python3.8/inspect.py", line 1515, in getinnerframes
frameinfo = (tb.tb_frame,) + getframeinfo(tb, context)
AttributeError: 'tuple' object has no attribute 'tb_frame'
╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /databricks/python/lib/python3.8/site-packages/click/core.py:1065 in main │
│ │
│ /databricks/python/lib/python3.8/site-packages/click/core.py:687 in exit │
╰──────────────────────────────────────────────────────────────────────────────╯
Exit: 0

During handling of the above exception, another exception occurred:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /databricks/python/lib/python3.8/site-packages/IPython/core/interactiveshell │
│ .py:3437 in run_code │
│ │
│ 3434 │ │ │ │ elif async_ : │
│ 3435 │ │ │ │ │ await eval(code_obj, self.user_global_ns, self.us │
│ 3436 │ │ │ │ else: │
│ ❱ 3437 │ │ │ │ │ exec(code_obj, self.user_global_ns, self.user_ns) │
│ 3438 │ │ │ finally: │
│ 3439 │ │ │ │ # Reset our crash handler in place │
│ 3440 │ │ │ │ sys.excepthook = old_excepthook │
│ :14 in │
│ │
│ /tmp/tmpywm076jl.py:10 in │
│ │
│ 7 # main(["--pipeline", "train1"]) │
│ 8 # main(["--pipeline", "train2"]) │
│ 9 # main(["--pipeline", "inference"]) │
│ ❱ 10 main(["--pipeline", "de"]) │
│ 11 │
│ │
│ /databricks/python/lib/python3.8/site-packages/ata_psd/main.py:47 in │
│ main │
│ │
│ 44 │ package_name = Path(file).parent.name │
│ 45 │ configure_project(package_name) │
│ 46 │ run = _find_run_command(package_name) │
│ ❱ 47 │ run(*args, **kwargs) │
│ 48 │
│ 49 │
│ 50 if name == "main": │
│ │
│ /databricks/python/lib/python3.8/site-packages/click/core.py:1130 in │
│ call │
│ │
│ /databricks/python/lib/python3.8/site-packages/click/core.py:1083 in main │
╰──────────────────────────────────────────────────────────────────────────────╯
SystemExit: 0

During handling of the above exception, another exception occurred:

╭───────────────────── Traceback (most recent call last) ──────────────────────╮
│ /databricks/python/lib/python3.8/site-packages/IPython/core/interactiveshell │
│ .py:3357 in run_ast_nodes │
│ │
│ 3354 │ │ │ │ │ with compiler.extra_flags(getattr(ast, 'PyCF_ALLO │
│ 3355 │ │ │ │ │ │ code = compiler(mod, cell_name, mode) │
│ 3356 │ │ │ │ │ │ asy = compare(code) │
│ ❱ 3357 │ │ │ │ │ if (await self.run_code(code, result, async_=asy │
│ 3358 │ │ │ │ │ │ return True │
│ 3359 │ │ │ │
│ 3360 │ │ │ # Flush softspace │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/interactiveshell │
│ .py:3444 in run_code │
│ │
│ 3441 │ │ except SystemExit as e: │
│ 3442 │ │ │ if result is not None: │
│ 3443 │ │ │ │ result.error_in_exec = e │
│ ❱ 3444 │ │ │ self.showtraceback(exception_only=True) │
│ 3445 │ │ │ warn("To exit: use 'exit', 'quit', or Ctrl-D.", stackleve │
│ 3446 │ │ except self.custom_exceptions: │
│ 3447 │ │ │ etype, value, tb = sys.exc_info() │
│ │
│ /databricks/python/lib/python3.8/site-packages/rich/traceback.py:99 in │
│ ipy_show_traceback │
│ │
│ 96 │ │ │ """wrap the default ip.showtraceback to store info for ip. │
│ 97 │ │ │ nonlocal tb_data │
│ 98 │ │ │ tb_data = kwargs │
│ ❱ 99 │ │ │ default_showtraceback(*args, **kwargs) │
│ 100 │ │ │
│ 101 │ │ def ipy_display_traceback( │
│ 102 │ │ │ *args: Any, is_syntax: bool = False, **kwargs: Any │
│ │
│ /databricks/python/lib/python3.8/site-packages/rich/traceback.py:99 in │
│ ipy_show_traceback │
│ │
│ 96 │ │ │ """wrap the default ip.showtraceback to store info for ip. │
│ 97 │ │ │ nonlocal tb_data │
│ 98 │ │ │ tb_data = kwargs │
│ ❱ 99 │ │ │ default_showtraceback(*args, **kwargs) │
│ 100 │ │ │
│ 101 │ │ def ipy_display_traceback( │
│ 102 │ │ │ *args: Any, is_syntax: bool = False, **kwargs: Any │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/interactiveshell │
│ .py:2054 in showtraceback │
│ │
│ 2051 │ │ │ │ if exception_only: │
│ 2052 │ │ │ │ │ stb = ['An exception has occurred, use %tb to see │
│ 2053 │ │ │ │ │ │ 'the full traceback.\n'] │
│ ❱ 2054 │ │ │ │ │ stb.extend(self.InteractiveTB.get_exception_only( │
│ 2055 │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ │
│ 2056 │ │ │ │ else: │
│ 2057 │ │ │ │ │ try: │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py:754 │
│ in get_exception_only │
│ │
│ 751 │ │ etype : exception type │
│ 752 │ │ value : exception value │
│ 753 │ │ """ │
│ ❱ 754 │ │ return ListTB.structured_traceback(self, etype, value) │
│ 755 │ │
│ 756 │ def show_exception_only(self, etype, evalue): │
│ 757 │ │ """Only print the exception type and message, without a trace │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py:629 │
│ in structured_traceback │
│ │
│ 626 │ │ │ chained_exc_ids.add(id(exception[1])) │
│ 627 │ │ │ chained_exceptions_tb_offset = 0 │
│ 628 │ │ │ out_list = ( │
│ ❱ 629 │ │ │ │ self.structured_traceback( │
│ 630 │ │ │ │ │ etype, evalue, (etb, chained_exc_ids), │
│ 631 │ │ │ │ │ chained_exceptions_tb_offset, context) │
│ 632 │ │ │ │ + chained_exception_message │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py:1367 │
│ in structured_traceback │
│ │
│ 1364 │ │ │ self.tb = tb[0] │
│ 1365 │ │ else: │
│ 1366 │ │ │ self.tb = tb │
│ ❱ 1367 │ │ return FormattedTB.structured_traceback( │
│ 1368 │ │ │ self, etype, value, tb, tb_offset, number_of_lines_of_con │
│ 1369 │
│ 1370 │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py:1267 │
│ in structured_traceback │
│ │
│ 1264 │ │ mode = self.mode │
│ 1265 │ │ if mode in self.verbose_modes: │
│ 1266 │ │ │ # Verbose modes need a full traceback │
│ ❱ 1267 │ │ │ return VerboseTB.structured_traceback( │
│ 1268 │ │ │ │ self, etype, value, tb, tb_offset, number_of_lines_of │
│ 1269 │ │ │ ) │
│ 1270 │ │ elif mode == 'Minimal': │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py:1124 │
│ in structured_traceback │
│ │
│ 1121 │ │ │ │ │ │ │ number_of_lines_of_context=5): │
│ 1122 │ │ """Return a nice text document describing the traceback.""" │
│ 1123 │ │ │
│ ❱ 1124 │ │ formatted_exception = self.format_exception_as_a_whole(etype, │
│ 1125 │ │ │ │ │ │ │ │ │ │ │ │ │ │ │ tb_off │
│ 1126 │ │ │
│ 1127 │ │ colors = self.Colors # just a shorthand + quicker name looku │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py:1082 │
│ in format_exception_as_a_whole │
│ │
│ 1079 │ │ records = self.get_records(etb, number_of_lines_of_context, t │
│ 1080 │ │ │
│ 1081 │ │ │
│ ❱ 1082 │ │ last_unique, recursion_repeat = find_recursion(orig_etype, ev │
│ 1083 │ │ │
│ 1084 │ │ frames = self.format_records(records, last_unique, recursion_ │
│ 1085 │
│ │
│ /databricks/python/lib/python3.8/site-packages/IPython/core/ultratb.py:382 │
│ in find_recursion │
│ │
│ 379 │ # quarter of the traceback (250 frames by default) is repeats, an │
│ 380 │ # first frame (from in to out) that looks different. │
│ 381 │ if not is_recursion_error(etype, value, records): │
│ ❱ 382 │ │ return len(records), 0 │
│ 383 │ │
│ 384 │ # Select filename, lineno, func_name to track frames with │
│ 385 │ records = [r[1:4] for r in records] │
╰──────────────────────────────────────────────────────────────────────────────╯
TypeError: object of type 'NoneType' has no len()

Thanks in advance for any tips and advise.

[renardeinside]

Hi @datascientistlin ,
the main method calls kedro, and kedro calls click under the hood. at the end of the command click calls sys.exit(0), which is not allowed by Databricks as described here.

I would recommend checking if there is another API that allows to call the command without click context.

[datascientistlin]

Thanks @renardeinside for your insightful update as always. So it means a sys.exit(0) is automatically appended to the end of the code after databricks has run through all the lines. Is my understanding correct?

Do you have any suggestion on how to avoid the sys.exit(0) to be called? Does wrapping the code body inside a if __name__ == '__main__': help?

Thank you.

[renardeinside]

So it means a sys.exit(0) is automatically appended to the end of the code

this is appended by click which is used by Kedro, it's not something coming from the Databricks. I assume kedro has a non-cli based programming interface which should allow you to launch your workload without this issue.

[datascientistlin]

Thanks @renardeinside , this is helpful. Let me investigate further how to tune kedro in this case.