Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Does this connector work with Spark 3.3 and EMR 6.11 #191

Open
SravaniMaddala opened this issue Dec 25, 2023 · 1 comment
Open

Does this connector work with Spark 3.3 and EMR 6.11 #191

SravaniMaddala opened this issue Dec 25, 2023 · 1 comment

Comments

@SravaniMaddala
Copy link

I am trying to use this connector with Spark 3.3 and EMR 6.11 but the RDD is always empty. Wondering if there is any version mismatch that is causing this or if I am missing something.

SparkSession sparkSession = SparkSession.builder().getOrCreate();
SparkContext sparkContext = sparkSession.sparkContext();

JavaSparkContext sc = new JavaSparkContext(sparkContext)

JobConf jobConf = new JobConf(sc.hadoopConfiguration());
jobConf.set("dynamodb.input.tableName", "test-dynamo");
jobConf.set("mapred.output.format.class", "org.apache.hadoop.dynamodb.write.DynamoDBOutputFormat");
jobConf.set("mapred.input.format.class", "org.apache.hadoop.dynamodb.read.DynamoDBInputFormat");

JavaPairRDD<Text, DynamoDBItemWritable> rows = sc.hadoopRDD(jobConf, DynamoDBInputFormat.class,Text.class, DynamoDBItemWritable.class);

rows.count();
@custommonkey
Copy link

The unreleased master does seem to work with 3.3. I had the same experience with latest 4.16 release not working against 3.3.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

2 participants