HIVE_DELEGATION_TOKEN error

[hamdiazz]

We had an error when launching beeline in hive-metastore_PF9SOBDP073.log

2023-11-08T16:14:33,209 - ERROR [pool-8-thread-67:TSaslTransport@315] - SASL negotiation failure javax.security.sasl.SaslException: DIGEST-MD5: IO error acquiring password at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:598) ~[?:1.8.0_382] at com.sun.security.sasl.digest.DigestMD5Server.evaluateResponse(DigestMD5Server.java:244) ~[?:1.8.0_382] at org.apache.thrift.transport.TSaslTransport$SaslParticipant.evaluateChallengeOrResponse(TSaslTransport.java:539) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:283) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_382] at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_382] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) ~[hadoop-common-3.1.1-0.0.jar:?] at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_382] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_382] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382] Caused by: org.apache.hadoop.security.token.SecretManager$InvalidToken: token expired or does not exist: HIVE_DELEGATION_TOKEN owner=hive, renewer=hive, realUser=hive/bdpnode03.bdpc01.applispfref.sipfref.local@bdpopf, issueDate=1699456473123, maxDate=1700061273123, sequenceNumber=21, masterKeyId=3 at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:104) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.hadoop.hive.metastore.security.TokenStoreDelegationTokenSecretManager.retrievePassword(TokenStoreDelegationTokenSecretManager.java:57) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.getPassword(HadoopThriftAuthBridge.java:511) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$SaslDigestCallbackHandler.handle(HadoopThriftAuthBridge.java:542) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at com.sun.security.sasl.digest.DigestMD5Server.validateClientResponse(DigestMD5Server.java:589) ~[?:1.8.0_382] ... 15 more 2023-11-08T16:14:33,210 - ERROR [pool-8-thread-67:TThreadPoolServer$WorkerProcess@297] - Error occurred during processing of message. java.lang.RuntimeException: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:219) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:694) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory$1.run(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at java.security.AccessController.doPrivileged(Native Method) ~[?:1.8.0_382] at javax.security.auth.Subject.doAs(Subject.java:360) ~[?:1.8.0_382] at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709) ~[hadoop-common-3.1.1-0.0.jar:?] at org.apache.hadoop.hive.metastore.security.HadoopThriftAuthBridge$Server$TUGIAssumingTransportFactory.getTransport(HadoopThriftAuthBridge.java:691) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:269) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) ~[?:1.8.0_382] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) ~[?:1.8.0_382] at java.lang.Thread.run(Thread.java:750) ~[?:1.8.0_382] Caused by: org.apache.thrift.transport.TTransportException: DIGEST-MD5: IO error acquiring password at org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.thrift.transport.TSaslServerTransport.open(TSaslServerTransport.java:41) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] at org.apache.thrift.transport.TSaslServerTransport$Factory.getTransport(TSaslServerTransport.java:216) ~[hive-exec-3.1.3-1.0.jar:3.1.3-1.0] ... 10 more

I saw that the parameter below is mentioned in metastore_site list :
metastore.cluster.delegation.token.store.class: org.apache.hadoop.hive.thrift.DBTokenStore

There is two solutions after many tests:

Set in hive-site.xml :

• hive.cluster.delegation.token.store.class=org.apache.hadoop.hive.thrift.DBTokenStore

Set in hive-site.xml :

• hive.cluster.delegation.token.store.class: "org.apache.hadoop.hive.thrift.ZooKeeperTokenStore"
• hive.metastore.kerberos.keytab.file: "/etc/security/keytabs/hive.service.keytab"
• hive.metastore.kerberos.principal: "hive/_HOST@{{ realm }}"