Skip to content

Latest commit

 

History

History
52 lines (51 loc) · 3.44 KB

TODO.org

File metadata and controls

52 lines (51 loc) · 3.44 KB
Raw

729

use “kubectl get kafkatopics.kafka.strimzi.io -n kafka” to check success of integration tests, maybe in fnkctl?

DELAYED BECAUSE IT NOT BLOCKS BUT WARN create topic in distribute before sending alerts in order to avoid error below: https://fink-broker.slack.com/archives/D03KJ390F17/p1692008729660549

Du coup ça fonctionne avec un compte utilisateur, par contre j’ai pas activé les autorisations dans kafka car le fink-alert-simulator aurait pu plus écrire dans le topic sans authentification. 12 h 28 J’ai maintenant ce message d’erreur: 23/08/14 10:26:52 WARN NetworkClient: [Producer clientId=producer-1] Error while fetching metadata with correlation id 29 : {fink_simbad_grav_candidates_ztf=LEADER_NOT_AVAILABLE} 12 h 32 En fait c’est du au fait que le topic existe pas, ça fonctionne si on relance lae job distribute… 12 h 33 Tu crois qu’on pourrais pré-créer les topic pour éviter ce problème @JulienPeloton ?

add user authentication in kafka https://stackoverflow.com/questions/65729535/how-to-do-i-connect-kafka-python-to-accept-username-and-password-for-jaas-like-i

Enable authZ in kafka (require authN setup in fink-alert-simulator)

[#B] distribute should wait for data to appear instead of crashing in connect_to_raw_database()

move nodeport to internal for svc kafka-cluster-kafka-external-bootstrap

improve final test in CI (check Kafka with fink-client https://github.com/astrolabsoftware/fink-client)

run code-check.sh in CI

add unit test for schema_converter

https://stackoverflow.com/questions/30385981/how-to-access-s3a-files-from-apache-spark

Document +add SO post?: Download hadoop binary release: https://www.apache.org/dyn/closer.cgi/hadoop/common/hadoop-3.2.4/hadoop-3.2.4.tar.gz extract and copy jar: fjammes@clrinfopo18  ~/Downloads/hadoop-3.2.4  cp ./share/hadoop/tools/lib/hadoop-aws-3.2.4.jar ~/src/k8s-spark-py/custom/jars fjammes@clrinfopo18  ~/Downloads/hadoop-3.2.4  cp ./share/hadoop/tools/lib/aws-java-sdk-bundle-1.11.901.jar ~/src/k8s-spark-py/custom/jars // WARNING package are not deployed in spark-executor // see https://stackoverflow.com/a/67299668/2784039

document hack to retrieve Maven URLs

kubectl logs stream2raw-py-f529af864f8dee60-driver | grep downlo | cut -d’ ’ -f2 > jars-urls.txt OR add mnv copy:dependencies when building the image?

manage dependencies

What to do with:

  1. hbase-spark-hbase2.4_spark3_scala2.12_hadoop3.2.jar

hbase-spark-protocol-shaded-hbase2.4_spark3_scala2.12_hadoop3.2.jar which are both in k8s-spark-py/custom and fink-broker/libs (cf. FINK_JARS) cf. Julien are they required?

  1. custom/jars/commons-pool2-2.6.2.jar which was in k8s-spark-py/custom

document minio install and bucket creation:

5 curl https://dl.min.io/client/mc/release/linux-amd64/mc –create-dirs -o $HOME/minio-binaries/mc 6 chmod +x $HOME/minio-binaries/mc 15 export PATH=$PATH:$HOME/minio-binaries/ 17 mc alias set s3 http://minio.minio-dev:9000 minioadmin minioadmin 19 mc ls s3 27 mc mb s3/fink-broker-online mc ls f1 –recursive fink-broker-online/

test removal of options below

  • –conf spark.driver.extraJavaOptions=”-Divy.cache.dir=/tmp -Divy.home=/tmp” \ –conf spark.hadoop.fs.s3a.path.style.access=true \
  • –conf spark.hadoop.fs.s3a.aws.credentials.provider=org.apache.hadoop.fs.s3a.SimpleAWSCredentialsProvider \

INSTALL MINIO https://min.io/docs/minio/kubernetes/upstream/index.html?