Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

detect/filestore: fix options handling and impact #12312

Open
wants to merge 9 commits into
base: master
Choose a base branch
from
+110 −66
Open

detect/filestore: fix options handling and impact #12312

Show file tree
Hide file tree
Changes from 4 commits
Commits
Show all changes
9 commits
Select commit Hold shift + click to select a range
e1a4441
detect/filestore: fix options handling and impact
regit Oct 29, 2024
d9c0844
file/store: WIP fix on filestore_cnt
regit Jan 22, 2025
e2734d6
formatting
regit Jan 22, 2025
6173435
remove useless else
regit Jan 22, 2025
f2a4c49
doc/userguide: use safe words
regit Jan 23, 2025
50db058
doc/userguide: use sticky buffer variant for magic
regit Jan 23, 2025
0496fde
doc/userguide: remove invalid examples
regit Jan 23, 2025
25eeaca
rules/files: modernize file magic rules
regit Jan 23, 2025
75664ef
rules/files: remove invalid examples
regit Jan 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions src/app-layer-htp.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5809,12 +5809,12 @@ libhtp:\n\
/* do detect */
SigMatchSignatures(&th_v, de_ctx, det_ctx, p1);

FAIL_IF((PacketAlertCheck(p1, 1)));
FAIL_IF((PacketAlertCheck(p1, 2)));
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What does this change mean ? Can there be a comment ?


/* do detect */
SigMatchSignatures(&th_v, de_ctx, det_ctx, p1);

FAIL_IF((PacketAlertCheck(p1, 1)));
FAIL_IF((PacketAlertCheck(p1, 2)));

r = AppLayerParserParse(
&th_v, alp_tctx, &f, ALPROTO_HTTP1, STREAM_TOCLIENT, httpbuf2, httplen2);
Expand Down
22 changes: 21 additions & 1 deletion src/detect-engine-file.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -186,14 +186,34 @@ uint8_t DetectFileInspectGeneric(DetectEngineCtx *de_ctx, DetectEngineThreadCtx
SCEnter();
DEBUG_VALIDATE_BUG_ON(f->alstate != alstate);

const uint8_t direction = flags & (STREAM_TOSERVER|STREAM_TOCLIENT);
const uint8_t direction = flags & (STREAM_TOSERVER | STREAM_TOCLIENT);
AppLayerGetFileState files = AppLayerParserGetTxFiles(f, tx, direction);
FileContainer *ffc = files.fc;
SCLogDebug("tx %p tx_id %" PRIu64 " ffc %p ffc->head %p sid %u", tx, tx_id, ffc,
ffc ? ffc->head : NULL, s->id);
if (ffc == NULL) {
SCReturnInt(DETECT_ENGINE_INSPECT_SIG_CANT_MATCH_FILES);
} else if (ffc->head == NULL) {
if (s->flags & SIG_FLAG_FILESTORE) {
/* If the signature has filestore, we need to check if we are in
a scope of capture where we need to prepare the capture for
an upcoming file. */
if (s->filestore_ctx && (s->filestore_ctx->scope == FILESTORE_SCOPE_TX)) {
/* In scope TX, we need to prepare file storage for file that could
appear on the transaction so we store the transaction.
We need to only increment filestore_cnt if the tx_id is changed.
*/
if (det_ctx->filestore_cnt == 0 ||
det_ctx->filestore[det_ctx->filestore_cnt - 1].tx_id != tx_id) {
det_ctx->filestore[det_ctx->filestore_cnt].file_id = 0;
det_ctx->filestore[det_ctx->filestore_cnt].tx_id = tx_id;
det_ctx->filestore_cnt++;
}
}
/* Other scopes than TX are going to be handled in post match without
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

But postmatch does not make signature fail to match, does it ?

I see (void)sigmatch_table[smd->type].Match(det_ctx, p, s, smd->ctx); return code ignored in DetectRunPostMatch

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, before we were not going to postmatch as the filestore without file was the cases where ffc or ffc->head were null and as a result it was a NO_MATCH.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I understand a bit more.
Why not handle FILESTORE_SCOPE_TX in postmatch as well ?

Is it not a problem if the signature matched so far but will not fully match in the end ?

any setup needed here so we can just return a match for them. */
SCReturnInt(DETECT_ENGINE_INSPECT_SIG_MATCH);
}
SCReturnInt(DETECT_ENGINE_INSPECT_SIG_NO_MATCH);
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

When do we get ffc != NULL && ffc->head == NULL ?

This still looks fishy to me as a signature with file.data that does not match today because of this SCReturnInt(DETECT_ENGINE_INSPECT_SIG_NO_MATCH), will cause FP with this PR...

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Talking about fishy, I think it is a good catch ;) I'm going to add a suricata-verify test so we are sure this case is handled.

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok, but do you know, in plain English, when we have the cases :

  • ffc == NULL (protocol or direction not supporting files ?)
  • ffc != NULL && ffc->head == NULL protocol supporting files but tx does not have a file in this direction ? (that is what I expected but this seems a wrong expectation)

}

Expand Down
80 changes: 65 additions & 15 deletions src/detect-filestore.c
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -98,6 +98,47 @@ void DetectFilestoreRegister(void)
g_file_match_list_id = DetectBufferTypeRegister("files");
}

static void FilestoreTriggerFlowStorage(
Flow *f, int toserver_dir, int toclient_dir, uint64_t init_tx_id)
{
SCLogDebug("init tx_id is: %ld", init_tx_id);
/* set flags in Flow and AppLayerStateData */
AppLayerStateData *sd = AppLayerParserGetStateData(f->proto, f->alproto, f->alstate);
if (toclient_dir) {
f->file_flags |= FLOWFILE_STORE_TC;
if (sd != NULL) {
sd->file_flags |= FLOWFILE_STORE_TC;
}
}
if (toserver_dir) {
f->file_flags |= FLOWFILE_STORE_TS;
if (sd != NULL) {
sd->file_flags |= FLOWFILE_STORE_TS;
}
}
/* Start storage for all existing files attached to the flow in correct direction */
void *alstate = FlowGetAppState(f);
if (alstate != NULL) {
uint64_t total_txs = AppLayerParserGetTxCnt(f, alstate);
for (uint64_t tx_id = init_tx_id; tx_id < total_txs; tx_id++) {
void *txv = AppLayerParserGetTx(f->proto, f->alproto, alstate, tx_id);
DEBUG_VALIDATE_BUG_ON(txv == NULL);
if (txv != NULL) {
AppLayerTxData *txd = AppLayerParserGetTxData(f->proto, f->alproto, txv);
DEBUG_VALIDATE_BUG_ON(txd == NULL);
if (txd != NULL) {
if (toclient_dir) {
txd->file_flags |= FLOWFILE_STORE_TC;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I do not think we need to set AppLayerTxData file_flags here as we already set AppLayerStateData one, and in the end we use a logical or of both

}
if (toserver_dir) {
txd->file_flags |= FLOWFILE_STORE_TS;
}
}
}
}
}
}

/**
* \brief apply the post match filestore with options
*/
Expand Down Expand Up @@ -170,20 +211,7 @@ static int FilestorePostMatchWithOptions(Packet *p, Flow *f, const DetectFilesto
}
}
} else if (this_flow) {
/* set in flow and AppLayerStateData */
AppLayerStateData *sd = AppLayerParserGetStateData(f->proto, f->alproto, f->alstate);
if (toclient_dir) {
f->file_flags |= FLOWFILE_STORE_TC;
if (sd != NULL) {
sd->file_flags |= FLOWFILE_STORE_TC;
}
}
if (toserver_dir) {
f->file_flags |= FLOWFILE_STORE_TS;
if (sd != NULL) {
sd->file_flags |= FLOWFILE_STORE_TS;
}
}
FilestoreTriggerFlowStorage(f, toserver_dir, toclient_dir, tx_id);
} else {
FileStoreFileById(fc, file_id);
}
Expand All @@ -207,11 +235,33 @@ static int DetectFilestorePostMatch(DetectEngineThreadCtx *det_ctx,
{
SCEnter();

DEBUG_VALIDATE_BUG_ON(p->flow == NULL);

if (det_ctx->filestore_cnt == 0) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By the way, det_ctx->filestore_cnt looks fishy to me cf OISF/suricata-verify#1999

Its scope is one packet when a signature can take multiple packets to match...

/* here we have no file but the signature is fully matched and
filestore option indicate we need to extract for file for the session
so we trigger flow storage. */
if (s->filestore_ctx && (s->filestore_ctx->scope == FILESTORE_SCOPE_SSN)) {
int toserver_dir = 0;
int toclient_dir = 0;
switch (s->filestore_ctx->direction) {
case FILESTORE_DIR_BOTH:
toserver_dir = 1;
toclient_dir = 1;
break;
case FILESTORE_DIR_TOSERVER:
toserver_dir = 1;
break;
case FILESTORE_DIR_TOCLIENT:
toclient_dir = 1;
break;
}
FilestoreTriggerFlowStorage(p->flow, toserver_dir, toclient_dir, det_ctx->tx_id);
}
SCReturnInt(0);
}

if ((s->filestore_ctx == NULL && !(s->flags & SIG_FLAG_FILESTORE)) || p->flow == NULL) {
if (s->filestore_ctx == NULL && !(s->flags & SIG_FLAG_FILESTORE)) {
#ifndef DEBUG
SCReturnInt(0);
#else
Expand Down
Loading