Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Resolve AttributeError: 'ScalaFunction1' object has no attribute 'hashCode'. #184

Open
wants to merge 3 commits into
base: master
Choose a base branch
from
Open

Resolve AttributeError: 'ScalaFunction1' object has no attribute 'hashCode'. #184

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
c79da8c
Resolve AttributeError: 'ScalaFunction1' object has no attribute 'has…
poolis Jan 3, 2024
c01b232
Address CI issue with return types.
poolis Jan 23, 2024
d417a24
Use Integer.hashCode.
poolis Jan 25, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 6 additions & 0 deletions pydeequ/scala_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,9 @@ def apply(self, arg):
"""Implements the apply function"""
return self.lambda_function(arg)

def hashCode(self):
return self.gateway.jvm.java.lang.Integer.hashCode(hash(self.lambda_function))

class Java:
"""scala.Function1: a function that takes one argument"""

Expand All @@ -55,6 +58,9 @@ def apply(self, t1, t2):
"""Implements the apply function"""
return self.lambda_function(t1, t2)

def hashCode(self):
return self.gateway.jvm.java.lang.Integer.hashCode(hash(self.lambda_function))

class Java:
"""scala.Function2: a function that takes two arguments"""

Expand Down
23 changes: 22 additions & 1 deletion tests/test_checks.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1366,4 +1366,25 @@ def test_fail_list_of_constraints(self):
.run()

df = VerificationResult.checkResultsAsDataFrame(self.spark, result)
self.assertEqual(df.select("constraint_status").collect(), [Row(constraint_status="Success"), Row(constraint_status="Success")])
self.assertEqual(df.select("constraint_status").collect(), [Row(constraint_status="Success"), Row(constraint_status="Success")])

def test_hash_code(self):
"""
Lack of Exception is passing. Previously this test would fail with:
AttributeError: 'ScalaFunction1' object has no attribute 'hashCode'
"""
vrb = VerificationSuite(self.spark) \
.onData(self.df)
check = Check(self.spark, CheckLevel.Error, "Enough checks to trigger a hashCode not an attribute of ScalaFunction1")
check.isComplete('b')
vrb.addCheck(check)
check.containsEmail('email')
vrb.addCheck(check)
check.isGreaterThanOrEqualTo("d", "b")
vrb.addCheck(check)
check.isLessThanOrEqualTo("b", "d")
vrb.addCheck(check)
check.hasDataType("d", ConstrainableDataTypes.String, lambda x: x >= 1)
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why do we need to add and verify one by one?

Copy link
Contributor Author

@poolis poolis Jan 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Because that is the use case for triggering the exception. If I change the test like so, it does not use hashCode.

        vrb = VerificationSuite(self.spark) \
            .onData(self.df)
        check = Check(self.spark, CheckLevel.Error, "Enough checks to trigger a hashCode not an attribute of ScalaFunction1")
        check.addConstraints([
            check.isComplete('b'),
            check.containsEmail('email'),
            check.isGreaterThanOrEqualTo("d", "b"),
            check.isLessThanOrEqualTo("b", "d"),
            check.hasDataType("d", ConstrainableDataTypes.String, lambda x: x >= 1)])

         result = vrb.addCheck(check).run()

Copy link
Contributor

@chenliu0831 chenliu0831 Jan 19, 2024 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Does it only fail at the magic 5th one? It's a big strange if so.. btw CI is failing on this test

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The issue write up has more info: #91

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I can't replicate the error in CI but submitted an attempt to fix it.

vrb.addCheck(check)

result = vrb.run()
13 changes: 12 additions & 1 deletion tests/test_scala_utils.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,13 +25,24 @@ def test_scala_function1(self):
self.assertFalse(notNoneTest.apply(None))
self.assertTrue(notNoneTest.apply("foo"))

# Test hashCode()
self.assertNotEqual(greaterThan10.hashCode(), notNoneTest.hashCode())
self.assertTrue(isinstance(greaterThan10.hashCode(), int))

appendTest = ScalaFunction1(self.sc._gateway, "{}test".format)
self.assertEqual("xtest", appendTest.apply("x"))

def test_scala_function2(self):
concatFunction = ScalaFunction2(self.sc._gateway, lambda x, y: x + y)
lambda_func = lambda x, y: x + y
concatFunction = ScalaFunction2(self.sc._gateway, lambda_func)
self.assertEqual("ab", concatFunction.apply("a", "b"))

anotherConcatFunction = ScalaFunction2(self.sc._gateway, lambda_func)

# Test hashCode()
self.assertEqual(concatFunction.hashCode(), anotherConcatFunction.hashCode())
self.assertTrue(isinstance(concatFunction.hashCode(), int))


if __name__ == "__main__":
unittest.main()
Loading