Skip to content

Commit

Permalink
Sort records fully for checksums
Browse files Browse the repository at this point in the history
Summary:
When reading a VRS file, records are sorted by timestamp, streamID, and position in the file, but not by record type, which means that two files can have the same records, but in slightly different order, and not be equivalent for checksums. As we added the `sort_records` option to generate file that might not be fully sorted, it's easy to generate file that are equivalent, but have different checksums.
When opening a file, you can now specify the same `sort_records` options to get the file fully sorted, taking into account the record's type, to get a deterministic state.

Reviewed By: kiminoue7

Differential Revision: D60942282

fbshipit-source-id: cdf94109f29ea794f0cedafc0655e38873123045
  • Loading branch information
Georges Berenger authored and facebook-github-bot committed Aug 8, 2024
1 parent 5adf845 commit d7277ae
Showing 1 changed file with 27 additions and 0 deletions.
27 changes: 27 additions & 0 deletions vrs/RecordFileReader.cpp
Original file line number Diff line number Diff line change
Expand Up @@ -43,6 +43,30 @@

using namespace std;

namespace {

using namespace vrs;

bool before(Record::Type lhs, Record::Type rhs) {
if (lhs == Record::Type::CONFIGURATION && rhs == Record::Type::STATE) {
return true;
} else if (lhs == Record::Type::STATE && rhs == Record::Type::CONFIGURATION) {
return false;
}
return static_cast<int>(lhs) < static_cast<int>(rhs);
}

bool fullRecordCompare(const IndexRecord::RecordInfo& lhs, const IndexRecord::RecordInfo& rhs) {
return lhs.timestamp < rhs.timestamp ||
(lhs.timestamp <= rhs.timestamp &&
(lhs.streamId < rhs.streamId ||
(lhs.streamId == rhs.streamId &&
(before(lhs.recordType, rhs.recordType) ||
(lhs.recordType == rhs.recordType && lhs.fileOffset < rhs.fileOffset)))));
}

} // namespace

namespace vrs {

StreamPlayer::~StreamPlayer() = default;
Expand Down Expand Up @@ -311,6 +335,9 @@ int RecordFileReader::doOpenFile(
streamRecordCounts_[record.streamId][record.recordType]++;
}
}
if (error == 0 && fileSpec.getExtraAsBool("sort_records", false)) {
sort(recordIndex_.begin(), recordIndex_.end(), fullRecordCompare);
}
return error;
}

Expand Down

0 comments on commit d7277ae

Please sign in to comment.