Pickle by value datetime dependency

[sebrestin]

Hi folks,

I've came across a behavior which I cannot explain. I am not sure if this is a bug or a limitation or simply I am am not using cloudpickle correctly. My hope is we can clarify which one it is. I am trying to pickle by value an object and its dependencies. In this case I am using a datetime as a dependency. The result is an error: AttributeError: type object 'date' has no attribute '__add__'

In order to reproduce this you can use the following test case(I am using Python 3.8), which is a simplified version.

    def test_pickle_by_value_datetime_dependency(self):

        _prev_sys_path = sys.path.copy()
        try:
            _mock_interactive_session_cwd = os.path.dirname(__file__)

            _maybe_remove(sys.path, '')
            _maybe_remove(sys.path, _mock_interactive_session_cwd)

            sys.path.insert(0, _mock_interactive_session_cwd)

            with subprocess_worker(protocol=self.protocol) as w:
                w.run(
                    lambda p: sys.path.remove(p), _mock_interactive_session_cwd
                )

                class C:
                    def __init__(self, d):
                        self.d = d

                cloudpickle.register_pickle_by_value(__import__(C.__module__))
                cloudpickle.register_pickle_by_value(datetime)

                d = datetime.datetime.now()
                obj = C(d)

                dump = cloudpickle.dumps(obj)

                w.run(
                    lambda content: cloudpickle.loads(content), dump
                )
        finally:
            sys.path = _prev_sys_path

[sebrestin]

After a bit more debug, it seems the problem comes from the difficulty of pickling Python descriptors. In this particular case adding the following solves the __add__ problem:

    _dispatch_table[types.WrapperDescriptorType] = _desc_reduce
    _dispatch_table[types.ClassMethodDescriptorType] = _desc_reduce
    _dispatch_table[types.MethodDescriptorType] = _desc_reduce

But it fails when it reaches a more complex get/set descriptor: year.

[ogrisel]

Could you please provide a minimal reproduction script that we could copy and paste to reproduce the problem alongside with the full traceback you observe and the Python version?

[ogrisel]

Alternatively, please create a "draft" with your experiment so that we can have a look at the CI logs.

[sebrestin]

@ogrisel You cannot reproduce it using the test I posted in my first entry?