Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Backport 2.17] Update threat intel job mapping to new version #1286

Merged
merged 1 commit into from
Sep 4, 2024
Merged

[Backport 2.17] Update threat intel job mapping to new version #1286

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 5 additions & 16 deletions src/main/java/org/opensearch/securityanalytics/threatIntel/common/TIFLockService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -73,24 +73,13 @@
*
* @param lockModel the lock model
*/
public void releaseLock(final LockModel lockModel) {
log.debug("Releasing lock with id [{}]", lockModel.getLockId());
public void releaseLock(final LockModel lockModel, final ActionListener<Boolean> listener) {
lockService.release(
lockModel,
ActionListener.wrap(released -> {}, exception -> log.error("Failed to release the lock", exception))
);
}

/**
* Wrapper method of LockService#release
*
* @param lockModel the lock model
*/
public void releaseLockEventDriven(final LockModel lockModel, final ActionListener<Boolean> listener) {
log.debug("Releasing lock with id [{}]", lockModel.getLockId());
lockService.release(
lockModel,
ActionListener.wrap(listener::onResponse, exception -> log.error("Failed to release the lock", exception))
ActionListener.wrap(listener::onResponse, exception -> {
log.error("Failed to release the lock", exception);
listener.onFailure(exception);
})

Check warning on line 82 in src/main/java/org/opensearch/securityanalytics/threatIntel/common/TIFLockService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/common/TIFLockService.java#L80-L82 

Added lines #L80 - L82 were not covered by tests
);
}

Expand Down
4 changes: 2 additions & 2 deletions src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFJobRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,7 +115,7 @@
ActionListener.wrap(lock -> {
updateJobParameter(jobParameter, lockService.getRenewLockRunnable(new AtomicReference<>(lock)),
ActionListener.wrap(
r -> lockService.releaseLockEventDriven(lock, ActionListener.wrap(
r -> lockService.releaseLock(lock, ActionListener.wrap(

Check warning on line 118 in src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFJobRunner.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFJobRunner.java#L118 

Added line #L118 was not covered by tests
response -> {
log.debug("Released tif job parameter lock with id [{}]", lock.getLockId());
},
Expand All @@ -125,7 +125,7 @@
)),
e -> {
log.error("Failed to update job parameter " + jobParameter.getName(), e);
lockService.releaseLockEventDriven(lock, ActionListener.wrap(
lockService.releaseLock(lock, ActionListener.wrap(

Check warning on line 128 in src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFJobRunner.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFJobRunner.java#L128 

Added line #L128 was not covered by tests
response -> {
log.debug("Released tif job parameter lock with id [{}]", lock.getLockId());
},
Expand Down
4 changes: 2 additions & 2 deletions ...java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFSourceConfigRunner.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -108,7 +108,7 @@
updateSourceConfigAndIOCs(saTifSourceConfig, lockService.getRenewLockRunnable(new AtomicReference<>(lock)),
ActionListener.wrap(
r -> {
lockService.releaseLockEventDriven(lock, ActionListener.wrap(
lockService.releaseLock(lock, ActionListener.wrap(

Check warning on line 111 in src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFSourceConfigRunner.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFSourceConfigRunner.java#L111 

Added line #L111 was not covered by tests
response -> {
log.debug("Released threat intel source config lock with id [{}]", lock.getLockId());
},
Expand All @@ -119,7 +119,7 @@
},
e -> {
log.error("Failed to update threat intel source config " + saTifSourceConfig.getName(), e);
lockService.releaseLockEventDriven(lock, ActionListener.wrap(
lockService.releaseLock(lock, ActionListener.wrap(

Check warning on line 122 in src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFSourceConfigRunner.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/jobscheduler/TIFSourceConfigRunner.java#L122 

Added line #L122 was not covered by tests
response -> {
log.debug("Released threat intel source config lock with id [{}]", lock.getLockId());
},
Expand Down
94 changes: 68 additions & 26 deletions ...n/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -50,10 +50,10 @@
import org.opensearch.securityanalytics.commons.model.IOCType;
import org.opensearch.securityanalytics.threatIntel.action.monitor.SearchThreatIntelMonitorAction;
import org.opensearch.securityanalytics.threatIntel.action.monitor.request.SearchThreatIntelMonitorRequest;
import org.opensearch.securityanalytics.threatIntel.common.StashedThreadContext;
import org.opensearch.securityanalytics.threatIntel.common.TIFLockService;
import org.opensearch.securityanalytics.threatIntel.model.DefaultIocStoreConfig;
import org.opensearch.securityanalytics.threatIntel.model.SATIFSourceConfig;
import org.opensearch.securityanalytics.util.IndexUtils;
import org.opensearch.securityanalytics.util.SecurityAnalyticsException;
import org.opensearch.threadpool.ThreadPool;

Expand All @@ -80,6 +80,7 @@
import static org.opensearch.securityanalytics.threatIntel.model.SATIFSourceConfig.SOURCE_CONFIG_FIELD;
import static org.opensearch.securityanalytics.threatIntel.model.SATIFSourceConfig.STATE_FIELD;
import static org.opensearch.securityanalytics.transport.TransportIndexDetectorAction.PLUGIN_OWNER_FIELD;
import static org.opensearch.securityanalytics.util.IndexUtils.shouldUpdateIndex;

/**
* CRUD for threat intel feeds source config object
Expand All @@ -93,7 +94,6 @@
private final NamedXContentRegistry xContentRegistry;
private final TIFLockService lockService;


public SATIFSourceConfigService(final Client client,
final ClusterService clusterService,
ThreadPool threadPool,
Expand Down Expand Up @@ -139,7 +139,7 @@
}
}, exception -> {
log.error("Failed to create threat intel source config index", exception);
lockService.releaseLockEventDriven(lock, ActionListener.wrap(
lockService.releaseLock(lock, ActionListener.wrap(

Check warning on line 142 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L142 

Added line #L142 was not covered by tests
r -> {
log.debug("Released threat intel source config lock with id [{}]", lock.getLockId());
actionListener.onFailure(exception);
Expand Down Expand Up @@ -197,33 +197,75 @@
/**
* Index name: .opensearch-sap--job
* Mapping: /mappings/threat_intel_job_mapping.json
* Updates the job index mapping if currently on a previous version
*
* @param stepListener setup listener
*/
public void createJobIndexIfNotExists(final StepListener<Void> stepListener) {
// check if job index exists
if (clusterService.state().metadata().hasIndex(SecurityAnalyticsPlugin.JOB_INDEX_NAME) == true) {
stepListener.onResponse(null);
return;
}
final CreateIndexRequest createIndexRequest = new CreateIndexRequest(SecurityAnalyticsPlugin.JOB_INDEX_NAME).mapping(getIndexMapping())
.settings(SecurityAnalyticsPlugin.TIF_JOB_INDEX_SETTING);
StashedThreadContext.run(client, () -> client.admin().indices().create(createIndexRequest, ActionListener.wrap(
r -> {
log.debug("[{}] index created", SecurityAnalyticsPlugin.JOB_INDEX_NAME);
stepListener.onResponse(null);
}, e -> {
if (e instanceof ResourceAlreadyExistsException) {
log.info("Index [{}] already exists", SecurityAnalyticsPlugin.JOB_INDEX_NAME);
checkAndUpdateJobIndexMapping(stepListener);
} else {
final CreateIndexRequest createIndexRequest = new CreateIndexRequest(SecurityAnalyticsPlugin.JOB_INDEX_NAME).mapping(getIndexMapping())
.settings(SecurityAnalyticsPlugin.TIF_JOB_INDEX_SETTING);
client.admin().indices().create(createIndexRequest, ActionListener.wrap(
r -> {
log.debug("[{}] index created", SecurityAnalyticsPlugin.JOB_INDEX_NAME);
stepListener.onResponse(null);
return;
}, e -> {
if (e instanceof ResourceAlreadyExistsException) {
log.info("Index [{}] already exists", SecurityAnalyticsPlugin.JOB_INDEX_NAME);
stepListener.onResponse(null);
return;

Check warning on line 219 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L217-L219 

Added lines #L217 - L219 were not covered by tests
}
log.error("Failed to create [{}] index", SecurityAnalyticsPlugin.JOB_INDEX_NAME, e);
stepListener.onFailure(e);

Check warning on line 222 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L221-L222 

Added lines #L221 - L222 were not covered by tests
}
log.error("Failed to create [{}] index", SecurityAnalyticsPlugin.JOB_INDEX_NAME, e);
stepListener.onFailure(e);
}
)));
));
}
}

private void checkAndUpdateJobIndexMapping(StepListener<Void> stepListener) {
try {
// Check if job index contains old mapping, if so update index mapping (current version = 2)
if (shouldUpdateIndex(clusterService.state().metadata().index(SecurityAnalyticsPlugin.JOB_INDEX_NAME), getIndexMapping())) {
log.info("Old schema version found for [{}] index, updating the index mapping", SecurityAnalyticsPlugin.JOB_INDEX_NAME);
IndexUtils.updateIndexMapping(
SecurityAnalyticsPlugin.JOB_INDEX_NAME,
getIndexMapping(), clusterService.state(), client.admin().indices(),
ActionListener.wrap(
r -> {
log.info("Successfully updated index mapping for [{}] index", SecurityAnalyticsPlugin.JOB_INDEX_NAME);
stepListener.onResponse(null);
}, e -> {
// Check if version is updated despite failure
try {
// Check if job index still contains older mapping
if (shouldUpdateIndex(clusterService.state().metadata().index(SecurityAnalyticsPlugin.JOB_INDEX_NAME), getIndexMapping())) {
log.error("Job index still contains older mapping, failed to update job index mapping", e);
stepListener.onFailure(e);

Check warning on line 246 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L245-L246 

Added lines #L245 - L246 were not covered by tests
} else {
// If job index contains newest mapping, then return success
log.info("Successfully updated index mapping for [{}] index", SecurityAnalyticsPlugin.JOB_INDEX_NAME);
stepListener.onResponse(null);

Check warning on line 250 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L249-L250 

Added lines #L249 - L250 were not covered by tests
}
} catch (IOException exception) {
log.error("Failed to check if job index contains older mapping. Failed to update job index mapping", e);
stepListener.onFailure(e);
}
}

Check warning on line 256 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L252-L256 

Added lines #L252 - L256 were not covered by tests
),
false
);
} else {
// If job index contains newest mapping, then do nothing
stepListener.onResponse(null);
}
} catch (IOException e) {
log.error("Failed to check and update job index mapping", e);
stepListener.onFailure(e);

Check warning on line 266 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L264-L266 

Added lines #L264 - L266 were not covered by tests
}
}

// Get TIF source config
public void getTIFSourceConfig(
Expand All @@ -234,7 +276,7 @@
client.get(getRequest, ActionListener.wrap(
getResponse -> {
if (!getResponse.isExists()) {
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.getDefault(),"Threat intel source config [%s] not found.", tifSourceConfigId), RestStatus.NOT_FOUND)));
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.ROOT, "Threat intel source config [%s] not found.", tifSourceConfigId), RestStatus.NOT_FOUND)));

Check warning on line 279 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L279 

Added line #L279 was not covered by tests
return;
}
SATIFSourceConfig saTifSourceConfig = null;
Expand All @@ -246,7 +288,7 @@
saTifSourceConfig = SATIFSourceConfig.docParse(xcp, getResponse.getId(), getResponse.getVersion());
}
if (saTifSourceConfig == null) {
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.getDefault(),"No threat intel source config exists [%s]", tifSourceConfigId), RestStatus.BAD_REQUEST)));
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.ROOT, "No threat intel source config exists [%s]", tifSourceConfigId), RestStatus.BAD_REQUEST)));

Check warning on line 291 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L291 

Added line #L291 was not covered by tests
} else {
log.debug("Threat intel source config with id [{}] fetched", getResponse.getId());
actionListener.onResponse(saTifSourceConfig);
Expand Down Expand Up @@ -366,9 +408,9 @@
log.info("Deleted threat intel source config [{}] successfully", saTifSourceConfig.getId());
actionListener.onResponse(deleteResponse);
} else if (deleteResponse.status().equals(RestStatus.NOT_FOUND)) {
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.getDefault(), "Threat intel source config with id [{%s}] not found", saTifSourceConfig.getId()), RestStatus.NOT_FOUND)));
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.ROOT, "Threat intel source config with id [{%s}] not found", saTifSourceConfig.getId()), RestStatus.NOT_FOUND)));

Check warning on line 411 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L411 

Added line #L411 was not covered by tests
} else {
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.getDefault(), "Failed to delete threat intel source config [{%s}]", saTifSourceConfig.getId()), deleteResponse.status())));
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.ROOT, "Failed to delete threat intel source config [{%s}]", saTifSourceConfig.getId()), deleteResponse.status())));

Check warning on line 413 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L413 

Added line #L413 was not covered by tests
}
}, e -> {
log.error("Failed to delete threat intel source config with id [{}]", saTifSourceConfig.getId());
Expand Down Expand Up @@ -407,7 +449,7 @@
log.info("Threat intel job scheduler lock with id [{}] not found", id);
actionListener.onResponse(deleteResponse);
} else {
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.getDefault(), "Failed to delete threat intel job scheduler lock with id [{%s}]", id), deleteResponse.status())));
actionListener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.ROOT, "Failed to delete threat intel job scheduler lock with id [{%s}]", id), deleteResponse.status())));

Check warning on line 452 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L452 

Added line #L452 was not covered by tests
}
}, e -> {
log.error("Failed to delete threat intel job scheduler lock with id [{}]", id);
Expand Down Expand Up @@ -453,7 +495,7 @@
if (!response.isAcknowledged()) {
log.error("Could not delete one or more IOC indices: " + index);
if (backgroundJob == false) {
listener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.getDefault(), "Could not delete one or more IOC indices: " + index), RestStatus.BAD_REQUEST)));
listener.onFailure(SecurityAnalyticsException.wrap(new OpenSearchStatusException(String.format(Locale.ROOT, "Could not delete one or more IOC indices: " + index), RestStatus.BAD_REQUEST)));

Check warning on line 498 in src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/service/SATIFSourceConfigService.java#L498 

Added line #L498 was not covered by tests
}
} else {
log.debug("Successfully deleted one or more IOC indices:" + index);
Expand Down
8 changes: 3 additions & 5 deletions ...ensearch/securityanalytics/threatIntel/transport/TransportIndexTIFSourceConfigAction.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,10 +20,8 @@
import org.opensearch.securityanalytics.threatIntel.action.SAIndexTIFSourceConfigAction;
import org.opensearch.securityanalytics.threatIntel.action.SAIndexTIFSourceConfigRequest;
import org.opensearch.securityanalytics.threatIntel.action.SAIndexTIFSourceConfigResponse;
import org.opensearch.securityanalytics.threatIntel.common.SourceConfigType;
import org.opensearch.securityanalytics.threatIntel.common.TIFLockService;
import org.opensearch.securityanalytics.threatIntel.model.SATIFSourceConfigDto;
import org.opensearch.securityanalytics.threatIntel.model.UrlDownloadSource;
import org.opensearch.securityanalytics.threatIntel.service.SATIFSourceConfigManagementService;
import org.opensearch.securityanalytics.transport.SecureTransportAction;
import org.opensearch.securityanalytics.util.SecurityAnalyticsException;
Expand Down Expand Up @@ -106,7 +104,7 @@
user,
ActionListener.wrap(
saTifSourceConfigDtoResponse -> {
lockService.releaseLockEventDriven(lock, ActionListener.wrap(
lockService.releaseLock(lock, ActionListener.wrap(
r -> {
log.debug("Released threat intel source config lock with id [{}]", lock.getLockId());
listener.onResponse(new SAIndexTIFSourceConfigResponse(
Expand All @@ -129,7 +127,7 @@
}, e -> {
String action = RestRequest.Method.PUT.equals(request.getMethod()) ? "update" : "create";
log.error(String.format("Failed to %s IOCs and threat intel source config", action), e);
lockService.releaseLockEventDriven(lock, ActionListener.wrap(
lockService.releaseLock(lock, ActionListener.wrap(
r -> {
log.debug("Released threat intel source config lock with id [{}]", lock.getLockId());
listener.onFailure(e);
Expand All @@ -146,7 +144,7 @@
} catch (Exception e) {
String action = RestRequest.Method.PUT.equals(request.getMethod()) ? "update" : "create";
log.error(String.format("Failed to %s IOCs and threat intel source config", action), e);
lockService.releaseLockEventDriven(lock, ActionListener.wrap(
lockService.releaseLock(lock, ActionListener.wrap(

Check warning on line 147 in src/main/java/org/opensearch/securityanalytics/threatIntel/transport/TransportIndexTIFSourceConfigAction.java

View check run for this annotation

Codecov / codecov/patch

src/main/java/org/opensearch/securityanalytics/threatIntel/transport/TransportIndexTIFSourceConfigAction.java#L147 

Added line #L147 was not covered by tests
r -> {
log.debug("Released threat intel source config lock with id [{}]", lock.getLockId());
listener.onFailure(e);
Expand Down
Loading
Loading