Need help for configuring AthenaX

[Randika-Isuru]

Hi Folks,

I have raised my first question athenax-users - Google Groups This was that question Question

This question is extended version of that question.

I have setup my project as follows this is my .yaml configuration file

athenax.master.uri: http://master152:8083
catalog.impl: com.pearson.athenax.catalog.impl.MydashboardCatalogProvider
clusters:
1234567:
yarn.site.location: hdfs://master152:8020/athenax/yarn-site.xml
athenax.home.dir: hdfs://master152:8020/athenax
flink.uber.jar.location: hdfs://master152:8020/athenax/flink-dist_2.11-1.7-SNAPSHOT.jar
localize.resources:
- hdfs://master152:8020/athenax/log4j.properties
additional.jars:
- hdfs://master152:8020/athenax/connectors.jar
- hdfs://master152:8020/athenax/foo.jar
extras:
jobstore.leveldb.file: /home/cyclone/randika/athenax/db

After that I have start the AthenaX server with no errors. Then I have submitted the new job through the API call using postman here is that.

http://master152:8083/mydashboard/jobs/new-jobs

In here I need to mention i have changed the BASE_PATH

/ws/v1 as /mydashboard

and most important thing is I have change the implementation of this /jobs/new-jobs api call as follows

`@Override
  public Response allocateNewJob(SecurityContext securityContext) throws NotFoundException {
/*return Response.ok().entity(
    Collections.singletonMap("job-uuid", ctx.jobManager().newJobUUID())
).build();*/
// above is the original code...
  LOG.info("Allocate new job with default values... START...");
  JobsApi api = new JobsApi();
  UUID uuid = ctx.jobManager().newJobUUID();
    try {
		JobDefinitionDesiredstate state = new JobDefinitionDesiredstate()
		    .clusterId("1234567")
		    .resource(new JobDefinitionResource().vCores(1L).memory(2048L));
		JobDefinition job = new JobDefinition()
		    .query("SELECT * FROM input.foo")
		    .addDesiredStateItem(state);
		api.updateJob(UUID.fromString(uuid.toString()), job);
		LOG.info("Allocate new job with default values... END...");
		return Response.ok().entity(
		        Collections.singletonMap("job-uuid", uuid)
		    ).build();
	} catch (ApiException e) {
		LOG.error("Exception occur inside the");
		e.printStackTrace();
	}
    return null;

}`

After calling that api call via postman, I think job will successfully stored inside the path jobstore.leveldb.file which is mention on configuration file. And it has return job id

{ "job-uuid": "33df9d53-420e-45cf-bd4c-669f8fec66d1" }

My First question is, earlier (which is i have configure on very first time) after submitting the job, yarn it will try to run that job on top of yarn (As per my understanding this can be totally wrong) and fail (I have mention how it fail at my First question which I have asked on here. but now it won't happen. what I figure out is, earlier it has call scanAll() method and while execution that method it has fail. but now I think scan all method is not execute

My second question is I need to understand what was the real purpose of using AthenaX and how to use that proper way. and can you give some brief Explanation about this AthenaX platform It was highly appreciated.
for Ex.
Earlier I thought using athenaX platform we can submit our own custom flink Streaming job and we can create dynamical query and get some result from that. After that using that result we can do some predictive analytics things.

This was the basic idea I had my own about athenaX

One more thing that i need to tell about my configuration file. on my Configuration file it has mention foo.jar and connections.jar under additional.jars: section this two jar are just jars which is have no content in side that i have added those jar because of AthenaX getting started doc it has mention those jars are required.

Thanks