Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Iris: Display ingestion state for lecture slide upload #9090

Merged
merged 141 commits into from
Nov 29, 2024
Merged
Show file tree
Hide file tree
Changes from 138 commits
Commits
Show all changes
141 commits
Select commit Hold shift + click to select a range
d80da31
First UI addon's mockup for now
yassinsws Jul 9, 2024
f2dc002
Merge remote-tracking branch 'origin/HEAD' into feature/iris/AddInges…
yassinsws Jul 15, 2024
d5fb0c0
Add links to citations
yassinsws Jul 18, 2024
cadcf14
Add Ingestion Status server updates.
yassinsws Jul 19, 2024
e66ab6e
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Jul 19, 2024
babef53
Linter
yassinsws Jul 19, 2024
6900069
Linter
yassinsws Jul 19, 2024
53c3e6d
Update tests to the new Lecture initialization method from findAllByC…
yassinsws Jul 20, 2024
6e1a67c
fix test initialisation, and edit class for ingestion button
yassinsws Jul 20, 2024
744d460
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Jul 20, 2024
762ce50
Fix Error by course.id
yassinsws Jul 20, 2024
b6f19f4
fix initiliase profile place and return back css to original for comp…
yassinsws Jul 21, 2024
692ffd7
Add client tests
yassinsws Jul 21, 2024
9969fc5
Fix lint and erase unused code
yassinsws Jul 21, 2024
267e84b
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Jul 21, 2024
09fcfe8
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Jul 21, 2024
70d945a
Fix Css
yassinsws Jul 22, 2024
6ef48d5
Merge remote-tracking branch 'origin/feature/iris/AddIngestionStateUI…
yassinsws Jul 22, 2024
aeb6210
Add feedback Server Side
yassinsws Aug 20, 2024
818a484
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Aug 21, 2024
040a3dc
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Aug 21, 2024
f735406
Client Side feedback
yassinsws Aug 21, 2024
55a785f
Merge remote-tracking branch 'origin/feature/iris/AddIngestionStateUI…
yassinsws Aug 21, 2024
7b8f6a7
Fix optional updateIngestionState
yassinsws Aug 21, 2024
c76a4bc
Fix optional updateIngestionState
yassinsws Aug 21, 2024
68eaf4c
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Aug 21, 2024
6d0beba
Fix server test
yassinsws Aug 21, 2024
02db2b4
Merge remote-tracking branch 'origin/feature/iris/AddIngestionStateUI…
yassinsws Aug 21, 2024
31b0351
merge1
yassinsws Sep 1, 2024
cf1e5c8
merge2
yassinsws Sep 1, 2024
3fd0fe6
merge3
yassinsws Sep 1, 2024
7e598e0
merge4
yassinsws Sep 1, 2024
fc32803
merge5
yassinsws Sep 1, 2024
f5be815
merge5
yassinsws Sep 1, 2024
e909760
merge5
yassinsws Sep 1, 2024
caa421e
merge5
yassinsws Sep 1, 2024
e288e35
merge5
yassinsws Sep 1, 2024
ebeff52
merge5
yassinsws Sep 1, 2024
5181aa0
merge5
yassinsws Sep 1, 2024
59b9d6b
merge5
yassinsws Sep 1, 2024
e15070a
merge5
yassinsws Sep 1, 2024
67421fd
merge5
yassinsws Sep 1, 2024
42215c4
merge5
yassinsws Sep 1, 2024
3ff48f3
merge5
yassinsws Sep 1, 2024
0ac814d
merge5
yassinsws Sep 1, 2024
9309113
merge5
yassinsws Sep 1, 2024
88e5b26
merge5
yassinsws Sep 1, 2024
5f674c5
merge5
yassinsws Sep 1, 2024
dc96efa
merge5
yassinsws Sep 1, 2024
8d23e4d
merge5
yassinsws Sep 1, 2024
75db9a2
merge5
yassinsws Sep 1, 2024
48faf22
merge5
yassinsws Sep 1, 2024
6853968
merge5
yassinsws Sep 1, 2024
69c13f0
merge5
yassinsws Sep 1, 2024
fcefee5
merge5
yassinsws Sep 1, 2024
215b9fc
merge5
yassinsws Sep 1, 2024
fcf08d6
merge5
yassinsws Sep 1, 2024
0319ff8
merge5
yassinsws Sep 1, 2024
da79fb3
merge5
yassinsws Sep 1, 2024
3297316
merge5
yassinsws Sep 1, 2024
3d2fb88
merge5
yassinsws Sep 1, 2024
dc29c9f
merge5
yassinsws Sep 1, 2024
0f29d06
merge5
yassinsws Sep 1, 2024
582579c
merge5
yassinsws Sep 1, 2024
2dd8a97
merge5
yassinsws Sep 1, 2024
ff19eae
merge5
yassinsws Sep 1, 2024
cc4d227
merge5
yassinsws Sep 1, 2024
2ad3b63
merge5
yassinsws Sep 1, 2024
4456498
merge5
yassinsws Sep 1, 2024
21ab67a
merge5
yassinsws Sep 1, 2024
7520e87
merge5
yassinsws Sep 1, 2024
0fd5292
merge5
yassinsws Sep 1, 2024
f78f999
merge5
yassinsws Sep 1, 2024
1b292f4
merge5
yassinsws Sep 1, 2024
dddacc0
merge5
yassinsws Sep 1, 2024
f5c31eb
merge5
yassinsws Sep 1, 2024
422cc05
merge5
yassinsws Sep 1, 2024
23f5db8
merge5
yassinsws Sep 1, 2024
cb37124
merge5
yassinsws Sep 1, 2024
8df44e2
Michael review and change database to enum.
yassinsws Sep 1, 2024
c7659bb
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Sep 3, 2024
5f634d9
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Sep 4, 2024
ec071bf
Make IngestionState updated through api calls.
yassinsws Sep 5, 2024
4c10f75
Merge
yassinsws Sep 5, 2024
912fdbf
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Sep 6, 2024
eeb2785
Remove ingestion state from attachment unit and remove the tests base…
yassinsws Sep 6, 2024
8126a54
Merge remote-tracking branch 'origin/feature/iris/AddIngestionStateUI…
yassinsws Sep 6, 2024
42747d4
Lint
yassinsws Sep 6, 2024
4a0f746
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Sep 6, 2024
28c7350
Merge
yassinsws Oct 1, 2024
ecb0d38
Lint
yassinsws Oct 2, 2024
cf2ae6b
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 2, 2024
6c02a09
check if job is still not done and keep the in progress sign
yassinsws Oct 2, 2024
1969862
merge
yassinsws Oct 2, 2024
8722d67
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 7, 2024
49ce286
Fix server test
yassinsws Oct 7, 2024
86a9113
fix javadoc
yassinsws Oct 7, 2024
fbe1800
Update application-artemis.yml
yassinsws Oct 8, 2024
484c361
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 8, 2024
1eb33ab
implement feedback
yassinsws Oct 13, 2024
50fb122
implement feedback
yassinsws Oct 13, 2024
ed58b85
merge and reduce api calls for lecture ingestion state
yassinsws Oct 13, 2024
98750d2
reduce api calls for lecture ingestion state
yassinsws Oct 13, 2024
137cec9
client tests
yassinsws Oct 13, 2024
33f2a52
client tests
yassinsws Oct 13, 2024
cfabad9
client tests
yassinsws Oct 13, 2024
7e31cc3
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 15, 2024
c5d0f96
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 18, 2024
dbda433
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 19, 2024
de5cf94
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 19, 2024
3754f7e
Enforce atleast instructor properly and handle errors properly
Oct 21, 2024
bac5d9d
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 21, 2024
e75d3c4
Enforce atleast instructor properly
Oct 21, 2024
7fd8c18
Enforce atleast instructor properly
Oct 21, 2024
fda7170
Merge remote-tracking branch 'origin/feature/iris/AddIngestionStateUI…
Oct 21, 2024
b61ec62
Adress Michaels review
Oct 21, 2024
e22a04e
server stzle
Oct 21, 2024
a07cfdd
server stzle
Oct 21, 2024
4cc1c28
Address Stephan's Feedback and make a deep review of the code.
Oct 25, 2024
85ea14a
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 25, 2024
a4ab9f1
fix translation
Oct 25, 2024
548b805
Merge remote-tracking branch 'origin/feature/iris/AddIngestionStateUI…
Oct 25, 2024
3dd43ff
fix style
Oct 25, 2024
80c019b
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 26, 2024
f27b5f5
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 28, 2024
526c0ed
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 29, 2024
0828d1a
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Oct 31, 2024
68fb000
Merge remote-tracking branch 'origin/develop' into feature/iris/AddIn…
yassinsws Nov 3, 2024
d3fea7d
fix Merge
yassinsws Nov 3, 2024
9626b93
fix Merge conflict
yassinsws Nov 3, 2024
5bd2984
fix Merge conflict
yassinsws Nov 11, 2024
1899201
Fix if statement
yassinsws Nov 11, 2024
cb0599a
Fix test fail due to merge
yassinsws Nov 12, 2024
9a97704
Merge
yassinsws Nov 17, 2024
58ce5b1
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Nov 18, 2024
46a7ebc
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Nov 21, 2024
f7858cf
Merge branch 'develop' into feature/iris/AddIngestionStateUI
yassinsws Nov 28, 2024
450948c
Merge branch 'develop' into feature/iris/AddIngestionStateUI
bassner Nov 29, 2024
2523654
silence ci complaints
bassner Nov 29, 2024
2360a1b
Merge remote-tracking branch 'origin/develop' into feature/iris/AddIn…
bassner Nov 29, 2024
4fab77f
Merge branch 'develop' into feature/iris/AddIngestionStateUI
bassner Nov 29, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same for here, is this file pushed by accident?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

somelinting happened nothing changed in here

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this file pushed by accident?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

same for both linting nothing changed in the files

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Original file line number Diff line number Diff line change
@@ -0,0 +1,5 @@
package de.tum.cit.aet.artemis.iris.dto;

public enum IngestionState {
NOT_STARTED, IN_PROGRESS, PARTIALLY_INGESTED, DONE, ERROR
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@
package de.tum.cit.aet.artemis.iris.dto;

public record IngestionStateResponseDTO(IngestionState state) {
}
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,8 @@

import static de.tum.cit.aet.artemis.core.config.Constants.PROFILE_IRIS;

import java.net.URLEncoder;
import java.nio.charset.StandardCharsets;
import java.util.Arrays;
import java.util.List;

Expand All @@ -20,11 +22,15 @@
import com.fasterxml.jackson.databind.ObjectMapper;

import de.tum.cit.aet.artemis.iris.domain.settings.IrisSubSettingsType;
import de.tum.cit.aet.artemis.iris.dto.IngestionState;
import de.tum.cit.aet.artemis.iris.dto.IngestionStateResponseDTO;
import de.tum.cit.aet.artemis.iris.exception.IrisException;
import de.tum.cit.aet.artemis.iris.exception.IrisForbiddenException;
import de.tum.cit.aet.artemis.iris.exception.IrisInternalPyrisErrorException;
import de.tum.cit.aet.artemis.iris.service.pyris.dto.PyrisVariantDTO;
import de.tum.cit.aet.artemis.iris.service.pyris.dto.lectureingestionwebhook.PyrisWebhookLectureDeletionExecutionDTO;
import de.tum.cit.aet.artemis.iris.service.pyris.dto.lectureingestionwebhook.PyrisWebhookLectureIngestionExecutionDTO;
import de.tum.cit.aet.artemis.iris.service.pyris.job.IngestionWebhookJob;
import de.tum.cit.aet.artemis.iris.web.open.PublicPyrisStatusUpdateResource;

/**
Expand All @@ -42,12 +48,19 @@ public class PyrisConnectorService {

private final ObjectMapper objectMapper;

private final PyrisJobService pyrisJobService;

@Value("${server.url}")
private String artemisBaseUrl;

@Value("${artemis.iris.url}")
private String pyrisUrl;

public PyrisConnectorService(@Qualifier("pyrisRestTemplate") RestTemplate restTemplate, MappingJackson2HttpMessageConverter springMvcJacksonConverter) {
public PyrisConnectorService(@Qualifier("pyrisRestTemplate") RestTemplate restTemplate, MappingJackson2HttpMessageConverter springMvcJacksonConverter,
PyrisJobService pyrisJobService) {
this.restTemplate = restTemplate;
this.objectMapper = springMvcJacksonConverter.getObjectMapper();
this.pyrisJobService = pyrisJobService;
}

/**
Expand Down Expand Up @@ -97,15 +110,63 @@ public void executePipeline(String feature, String variant, Object executionDTO)
* @param variant The variant of the feature to execute
* @param executionDTO The DTO sent as a body for the execution
*/
public void executeLectureWebhook(String variant, PyrisWebhookLectureIngestionExecutionDTO executionDTO) {
public void executeLectureAddtionWebhook(String variant, PyrisWebhookLectureIngestionExecutionDTO executionDTO) {
yassinsws marked this conversation as resolved.
Show resolved Hide resolved
var endpoint = "/api/v1/webhooks/lectures/" + variant;
try {
restTemplate.postForEntity(pyrisUrl + endpoint, objectMapper.valueToTree(executionDTO), Void.class);
}
catch (HttpStatusCodeException e) {
log.error("Failed to send lecture unit {} to Pyris: {}", executionDTO.pyrisLectureUnit().lectureUnitId(), e.getMessage());
throw toIrisException(e);
}
catch (RestClientException | IllegalArgumentException e) {
log.error("Failed to send lecture unit {} to Pyris: {}", executionDTO.pyrisLectureUnit().lectureUnitId(), e.getMessage());
throw new PyrisConnectorException("Could not fetch response from Pyris");
}
}

/**
* Retrieves the ingestion state of the lecture unit specified by retrieving the ingestion state from the vector database in Pyris.
*
* @param courseId id of the course
* @param lectureId id of the lecture
* @param lectureUnitId id of the lectureUnit to check in the Pyris vector database
* @return The ingestion state of the lecture Unit
*
*/
IngestionState getLectureUnitIngestionState(long courseId, long lectureId, long lectureUnitId) {
try {
String encodedBaseUrl = URLEncoder.encode(artemisBaseUrl, StandardCharsets.UTF_8);
String url = pyrisUrl + "/api/v1/courses/" + courseId + "/lectures/" + lectureId + "/lectureUnits/" + lectureUnitId + "/ingestion-state?base_url=" + encodedBaseUrl;
IngestionStateResponseDTO response = restTemplate.getForObject(url, IngestionStateResponseDTO.class);
IngestionState state = response.state();
if (state != IngestionState.DONE) {
if (pyrisJobService.currentJobs().stream().filter(job -> job instanceof IngestionWebhookJob).map(job -> (IngestionWebhookJob) job)
.anyMatch(ingestionJob -> ingestionJob.courseId() == courseId && ingestionJob.lectureId() == lectureId && ingestionJob.lectureUnitId() == lectureUnitId)) {
return IngestionState.IN_PROGRESS;
}
}
return state;
}
catch (RestClientException | IllegalArgumentException e) {
log.error("Error fetching ingestion state for lecture {}, lecture unit {}", lectureId, lectureUnitId, e);
throw new PyrisConnectorException("Error fetching ingestion state for lecture unit" + lectureUnitId);
}
}

/**
* Executes a webhook and send lectures to the webhook with the given variant
*
* @param executionDTO The DTO sent as a body for the execution
*/
public void executeLectureDeletionWebhook(PyrisWebhookLectureDeletionExecutionDTO executionDTO) {
var endpoint = "/api/v1/webhooks/lectures/delete";
try {
restTemplate.postForEntity(pyrisUrl + endpoint, objectMapper.valueToTree(executionDTO), Void.class);
}
catch (HttpStatusCodeException e) {
log.error("Failed to send lectures to Pyris", e);
throw toIrisException(e);
// TODO : add error ingestion UI.
}
catch (RestClientException | IllegalArgumentException e) {
log.error("Failed to send lectures to Pyris", e);
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -3,6 +3,7 @@
import static de.tum.cit.aet.artemis.core.config.Constants.PROFILE_IRIS;

import java.security.SecureRandom;
import java.util.Collection;
import java.util.concurrent.TimeUnit;
import java.util.function.Function;

Expand Down Expand Up @@ -93,11 +94,14 @@ public String addCourseChatJob(Long courseId, Long sessionId) {
/**
* Adds a new ingestion webhook job to the job map with a timeout.
*
* @param courseId the ID of the course associated with the webhook job
* @param lectureId the ID of the lecture associated with the webhook job
* @param lectureUnitId the ID of the lecture unit associated with the webhook job
* @return a unique token identifying the created webhook job
*/
public String addIngestionWebhookJob() {
public String addIngestionWebhookJob(long courseId, long lectureId, long lectureUnitId) {
var token = generateJobIdToken();
var job = new IngestionWebhookJob(token);
var job = new IngestionWebhookJob(token, courseId, lectureId, lectureUnitId);
long timeoutWebhookJob = 60;
TimeUnit unitWebhookJob = TimeUnit.MINUTES;
jobMap.put(token, job, timeoutWebhookJob, unitWebhookJob);
Expand All @@ -122,6 +126,15 @@ public void updateJob(PyrisJob job) {
jobMap.put(job.jobId(), job);
}

/**
* Get all current jobs.
*
* @return the all current jobs
*/
public Collection<PyrisJob> currentJobs() {
return jobMap.values();
}

/**
* Get the job of a token.
*
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -126,14 +126,12 @@ private void removeJobIfTerminatedElseUpdate(List<PyrisStageDTO> stages, PyrisJo
}

/**
* Handles the status update of a lecture ingestion job and logs the results for now => will change later
* TODO: Update this method to handle changes beyond logging
* Handles the status update of a lecture ingestion job.
*
* @param job the job that is updated
* @param statusUpdate the status update
*/
public void handleStatusUpdate(IngestionWebhookJob job, PyrisLectureIngestionStatusUpdateDTO statusUpdate) {
statusUpdate.stages().forEach(stage -> log.info(stage.name() + ":" + stage.message()));
removeJobIfTerminatedElseUpdate(statusUpdate.stages(), job);
}
}
Loading
Loading