Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Malware protection 6: Malware integration #3604

Merged
merged 82 commits into from
Dec 10, 2024
Merged
Show file tree
Hide file tree
Changes from 63 commits
Commits
Show all changes
82 commits
Select commit Hold shift + click to select a range
afd0746
Rename PhishingDetection to MaliciousSiteProtection; code adjustments…
mallexxx Nov 22, 2024
45c982d
fix linter issues
mallexxx Nov 22, 2024
f041a1a
Update UserText key names; Bump BSK; disable TODO warnings
mallexxx Nov 25, 2024
3cdf77d
refactor Malicious Site Protection APIClient
mallexxx Nov 25, 2024
aca479c
fix test
mallexxx Nov 25, 2024
712f7d6
Update Malicious site data storing
mallexxx Nov 26, 2024
21f4585
fix BSK ref
mallexxx Nov 26, 2024
d135c21
fix BSK ref
mallexxx Nov 26, 2024
5687819
Refactor malicious site data update manager
mallexxx Nov 26, 2024
35cf020
Refactor Special Error Types
mallexxx Nov 27, 2024
8cb5739
Package.resolved
mallexxx Nov 27, 2024
6f29625
Integrate Malware protection
mallexxx Nov 27, 2024
87fd08c
Update protections update script and data
mallexxx Nov 28, 2024
9ef094d
validate revision returned by api in update script; fix revision upda…
mallexxx Nov 28, 2024
be5bde1
temporarily include malware test page to the filters
mallexxx Nov 28, 2024
bafd937
fix sh linter issue
mallexxx Nov 28, 2024
fcc6627
Package.resolved
mallexxx Nov 28, 2024
eb8be0b
bump BSK
mallexxx Nov 28, 2024
906ccd4
Malware protection 2: API Client refactoring (#3595)
mallexxx Nov 28, 2024
59b989a
Malware protection 3: Refactor Data storing (#3598)
mallexxx Nov 29, 2024
4d40826
Merge remote-tracking branch 'origin/main' into alex/malware-protecti…
mallexxx Nov 29, 2024
87f5f47
bump BSK
mallexxx Nov 29, 2024
fc9db2d
Merge branch 'alex/malware-protection-1' into alex/malware-protection-5
mallexxx Nov 29, 2024
e380729
Merge branch 'alex/malware-protection-5' into alex/malware-protection-6
mallexxx Nov 29, 2024
5b82f9b
fix linter warning
mallexxx Nov 29, 2024
30460c2
fix update_phishing_detection_data.sh; update malicious site data
mallexxx Nov 29, 2024
94f1595
Merge branch 'alex/malware-protection-5' into alex/malware-protection-6
mallexxx Nov 29, 2024
961e60f
fix updating hashes by update_phishing_detection_data.sh, add more in…
mallexxx Nov 29, 2024
556ec25
Merge branch 'alex/malware-protection-5' into alex/malware-protection-6
mallexxx Nov 29, 2024
caccaff
fix linter issue
mallexxx Nov 29, 2024
5445910
fix feature flag usage
mallexxx Nov 29, 2024
df70113
Merge branch 'alex/malware-protection-5' into alex/malware-protection-6
mallexxx Nov 29, 2024
46dce01
add Malware tests
mallexxx Nov 29, 2024
a5efd99
fix tests
mallexxx Nov 29, 2024
200c1b2
remove ambiguity for DuckURLSchemeHandler
mallexxx Nov 29, 2024
78640c2
Merge branch 'alex/malware-protection-5' into alex/malware-protection-6
mallexxx Nov 29, 2024
5c53151
fix tests
mallexxx Nov 29, 2024
6a688ac
increase timeout
mallexxx Nov 29, 2024
0eac1a9
fix tests
mallexxx Nov 29, 2024
83338a8
fix fire overlay kept showing after closing fire window
mallexxx Nov 29, 2024
6b92133
more debug info
mallexxx Nov 29, 2024
df4639b
rollback unneeded DuckURLSchemeHandler changes
mallexxx Nov 29, 2024
c9c7a46
rollback config
mallexxx Nov 29, 2024
5449841
Merge branch 'alex/malware-protection-1' into alex/malware-protection-5
mallexxx Nov 29, 2024
5afbe3b
address PR review comments
mallexxx Nov 29, 2024
31a6325
rm OnboardingTabExtension
mallexxx Nov 29, 2024
8b4c30e
fix tests
mallexxx Nov 29, 2024
82553b8
remove localOverrides checks
mallexxx Nov 29, 2024
b14f1a3
disable daily os version pixel for test runs
mallexxx Nov 29, 2024
4d2ac61
Merge remote-tracking branch 'origin/alex/malware-protection-5' into …
mallexxx Dec 2, 2024
7efd315
Merge remote-tracking branch 'origin/main' into alex/malware-protecti…
mallexxx Dec 2, 2024
6062a7f
fix build
mallexxx Dec 3, 2024
aeee5df
Malware protection 5: Refactor Special Error Types (#3603)
mallexxx Dec 3, 2024
4bbae0a
rollback lost line
mallexxx Dec 3, 2024
89bfa8b
cleanup
mallexxx Dec 3, 2024
fee4525
fix build
mallexxx Dec 3, 2024
18656dc
fix build
mallexxx Dec 3, 2024
7fd0faf
Merge branch 'alex/malware-protection-1' into alex/malware-protection-6
mallexxx Dec 3, 2024
b5ea63e
rename phishing, adjust settings
mallexxx Dec 3, 2024
8b62077
Merge remote-tracking branch 'origin/main' into alex/malware-protecti…
mallexxx Dec 3, 2024
ae17370
Pinned tab favicon
mallexxx Dec 4, 2024
e465120
upd filter ver
mallexxx Dec 4, 2024
61c94a0
ship review fixes
mallexxx Dec 4, 2024
88dbfb1
fix tests
mallexxx Dec 5, 2024
be2f2ee
add Warning: to Malware title
mallexxx Dec 5, 2024
941332a
go back for Ssl error page on Leave site action
mallexxx Dec 5, 2024
3ee8985
Merge remote-tracking branch 'origin/main' into alex/malware-protecti…
mallexxx Dec 6, 2024
8893893
update copy xcstrings
mallexxx Dec 6, 2024
d672847
update data
mallexxx Dec 6, 2024
c92cb45
fix page reloading on "Accept risk"
mallexxx Dec 6, 2024
4564fcf
remove TODO-s
mallexxx Dec 6, 2024
01c9932
fix tests
mallexxx Dec 6, 2024
f39d20f
update BSK dependency
SabrinaTardio Dec 6, 2024
3b80c27
update copy
mallexxx Dec 9, 2024
6307071
fix privacy dashboard button not updated
mallexxx Dec 10, 2024
dfb58cb
BSK -> 221.0.0
mallexxx Dec 10, 2024
d53d4f6
rm privacy-dashboard
mallexxx Dec 10, 2024
28f752e
Merge remote-tracking branch 'origin/main' into alex/malware-protecti…
mallexxx Dec 10, 2024
890480a
cleanup
mallexxx Dec 10, 2024
d4c60ee
maliciousSiteProtectionErrorPage feature flag -> maliciousSiteProtection
mallexxx Dec 10, 2024
244467d
BSK -> 221.0.1
mallexxx Dec 10, 2024
e14814f
Merge remote-tracking branch 'origin/main' into alex/malware-protecti…
mallexxx Dec 10, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
20 changes: 16 additions & 4 deletions DuckDuckGo.xcodeproj/project.pbxproj
Original file line number Diff line number Diff line change
Expand Up @@ -1972,6 +1972,10 @@
84B49F0F2CB10F0900FF08BB /* OHHTTPStubsSwift in Frameworks */ = {isa = PBXBuildFile; productRef = 84B49F0E2CB10F0900FF08BB /* OHHTTPStubsSwift */; };
84BBC7FF2CFA0D2F00BAE57A /* TestUtils in Frameworks */ = {isa = PBXBuildFile; productRef = 84BBC7FE2CFA0D2F00BAE57A /* TestUtils */; };
84BBC8012CFA0D3800BAE57A /* TestUtils in Frameworks */ = {isa = PBXBuildFile; productRef = 84BBC8002CFA0D3800BAE57A /* TestUtils */; };
84C96E462CF9BB6400A80A01 /* malwareFilterSet.json in Resources */ = {isa = PBXBuildFile; fileRef = 84C96E442CF9BB6400A80A01 /* malwareFilterSet.json */; };
84C96E472CF9BB6400A80A01 /* malwareHashPrefixes.json in Resources */ = {isa = PBXBuildFile; fileRef = 84C96E452CF9BB6400A80A01 /* malwareHashPrefixes.json */; };
84C96E482CF9BB6400A80A01 /* malwareFilterSet.json in Resources */ = {isa = PBXBuildFile; fileRef = 84C96E442CF9BB6400A80A01 /* malwareFilterSet.json */; };
84C96E492CF9BB6400A80A01 /* malwareHashPrefixes.json in Resources */ = {isa = PBXBuildFile; fileRef = 84C96E452CF9BB6400A80A01 /* malwareHashPrefixes.json */; };
84DC715A2C1C1E9000033B8C /* UserDefaultsWrapperTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 84DC71582C1C1E8A00033B8C /* UserDefaultsWrapperTests.swift */; };
84DC715B2C1C1E9000033B8C /* UserDefaultsWrapperTests.swift in Sources */ = {isa = PBXBuildFile; fileRef = 84DC71582C1C1E8A00033B8C /* UserDefaultsWrapperTests.swift */; };
84DDB90A2C92B66E008C997B /* WKVisitedLinkStoreWrapper.swift in Sources */ = {isa = PBXBuildFile; fileRef = 84DDB9092C92B667008C997B /* WKVisitedLinkStoreWrapper.swift */; };
Expand Down Expand Up @@ -4219,6 +4223,8 @@
84537A022C998C24008723BC /* FireWindowSession.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = FireWindowSession.swift; sourceTree = "<group>"; };
848648A02C76F4B20082282D /* BookmarksBarMenuViewController.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = BookmarksBarMenuViewController.swift; sourceTree = "<group>"; };
84B479072CCA7A3900F40329 /* Logger+UnitTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = "Logger+UnitTests.swift"; sourceTree = "<group>"; };
84C96E442CF9BB6400A80A01 /* malwareFilterSet.json */ = {isa = PBXFileReference; lastKnownFileType = text.json; path = malwareFilterSet.json; sourceTree = "<group>"; };
84C96E452CF9BB6400A80A01 /* malwareHashPrefixes.json */ = {isa = PBXFileReference; lastKnownFileType = text.json; path = malwareHashPrefixes.json; sourceTree = "<group>"; };
84DC71582C1C1E8A00033B8C /* UserDefaultsWrapperTests.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = UserDefaultsWrapperTests.swift; sourceTree = "<group>"; };
84DDB9092C92B667008C997B /* WKVisitedLinkStoreWrapper.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = WKVisitedLinkStoreWrapper.swift; sourceTree = "<group>"; };
84F1C8CE2C7705B500716446 /* BookmarksBarMenuPopover.swift */ = {isa = PBXFileReference; lastKnownFileType = sourcecode.swift; path = BookmarksBarMenuPopover.swift; sourceTree = "<group>"; };
Expand Down Expand Up @@ -9818,11 +9824,13 @@
CDE248A22C821FD500F9399D /* MaliciousSiteProtection */ = {
isa = PBXGroup;
children = (
CDE248A62C821FFE00F9399D /* phishingFilterSet.json */,
CDE248A32C821FFE00F9399D /* phishingHashPrefixes.json */,
CDE248A42C821FFE00F9399D /* MaliciousSiteProtectionManager.swift */,
CDE248A52C821FFE00F9399D /* MaliciousSiteProtectionPreferences.swift */,
CDE248A72C821FFE00F9399D /* MaliciousSiteProtectionState.swift */,
84C96E442CF9BB6400A80A01 /* malwareFilterSet.json */,
84C96E452CF9BB6400A80A01 /* malwareHashPrefixes.json */,
CDE248A62C821FFE00F9399D /* phishingFilterSet.json */,
CDE248A32C821FFE00F9399D /* phishingHashPrefixes.json */,
);
path = MaliciousSiteProtection;
sourceTree = "<group>";
Expand Down Expand Up @@ -10721,6 +10729,8 @@
3706FCD0293F65D500E42796 /* BookmarksBarCollectionViewItem.xib in Resources */,
3706FCD2293F65D500E42796 /* shield.json in Resources */,
7B5A23762C46A4A8007213AC /* ExcludedDomains.storyboard in Resources */,
84C96E482CF9BB6400A80A01 /* malwareFilterSet.json in Resources */,
84C96E492CF9BB6400A80A01 /* malwareHashPrefixes.json in Resources */,
3706FCD6293F65D500E42796 /* httpsMobileV2FalsePositives.json in Resources */,
3706FCD8293F65D500E42796 /* BookmarksBar.storyboard in Resources */,
3706FCD9293F65D500E42796 /* trackers-1.json in Resources */,
Expand Down Expand Up @@ -10915,6 +10925,8 @@
4BE5336B286912D40019DBFD /* BookmarksBarCollectionViewItem.xib in Resources */,
AA34396C2754D4E300B241FA /* shield.json in Resources */,
7B5A23752C46A4A8007213AC /* ExcludedDomains.storyboard in Resources */,
84C96E462CF9BB6400A80A01 /* malwareFilterSet.json in Resources */,
84C96E472CF9BB6400A80A01 /* malwareHashPrefixes.json in Resources */,
4B677435255DBEB800025BD8 /* httpsMobileV2FalsePositives.json in Resources */,
4BD18F05283F151F00058124 /* BookmarksBar.storyboard in Resources */,
AA3439792754D55100B241FA /* trackers-1.json in Resources */,
Expand Down Expand Up @@ -15334,8 +15346,8 @@
isa = XCRemoteSwiftPackageReference;
repositoryURL = "https://github.com/duckduckgo/BrowserServicesKit";
requirement = {
kind = exactVersion;
version = 217.0.0;
kind = revision;
revision = 30bc6507b5684045edaa9e7c90359772e9e38451;
};
};
9FF521422BAA8FF300B9819B /* XCRemoteSwiftPackageReference "lottie-spm" */ = {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -32,8 +32,7 @@
"kind" : "remoteSourceControl",
"location" : "https://github.com/duckduckgo/BrowserServicesKit",
"state" : {
"revision" : "befc1f1094353d8d88a99ac08885684c978b2016",
"version" : "217.0.0"
"revision" : "30bc6507b5684045edaa9e7c90359772e9e38451"
}
},
{
Expand Down
1 change: 0 additions & 1 deletion DuckDuckGo/Bookmarks/Services/BookmarkStoreMock.swift
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,6 @@ final class BookmarkStoreMock: BookmarkStore, CustomDebugStringConvertible {
queue.append(contentsOf: folder.children)
}
}
var indexInFavoritesArray = 0
store?.save(entitiesAtIndices: entities.map { ($0, nil, nil) }, completion: { _ in })
}
}
Expand Down
2 changes: 2 additions & 0 deletions DuckDuckGo/Common/Extensions/URLExtension.swift
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,8 @@ extension URL {
return URL(string: "https://duckduckgo.com/duckduckgo-help-pages/sync-and-backup/password-manager-security/")!
}

static var maliciousSiteProtectionLearnMore = URL(string: "https://duckduckgo.com/duckduckgo-help-pages/privacy/phishing-and-malware-protection/")!

static var searchSettings: URL {
return URL(string: "https://duckduckgo.com/settings/")!
}
Expand Down
1 change: 1 addition & 0 deletions DuckDuckGo/Common/Localizables/UserText.swift
Original file line number Diff line number Diff line change
Expand Up @@ -238,6 +238,7 @@ struct UserText {
static let webProcessCrashPageMessage = NSLocalizedString("page.crash.message", value: "Try reloading the page or come back later.", comment: "Error page message text shown when a Web Page process had crashed")
static let sslErrorPageTabTitle = NSLocalizedString("ssl.error.page.tab.title", value: "Warning: Site May Be Insecure", comment: "Title shown in an error page tab that warn users of security risks on a website due to SSL issues")
static let phishingErrorPageTabTitle = NSLocalizedString("phishing.error.page.tab.title", value: "Warning: Site May Be Deceptive", comment: "Title shown in an error page tab that warn users of security risks on a website that has been flagged as Phishing.")
static let malwareErrorPageTabTitle = NSLocalizedString("malware.error.page.tab.title", value: "Malware on this site can harm your device", comment: "Title shown in an error page tab that warn users of security risks on a website that has been flagged as Malware.")

static let openSystemPreferences = NSLocalizedString("open.preferences", value: "Open System Preferences", comment: "Open System Preferences (to re-enable permission for the App) (up to and including macOS 12")
static let openSystemSettings = NSLocalizedString("open.settings", value: "Open System Settings…", comment: "This string represents a prompt or button label prompting the user to open system settings")
Expand Down
2 changes: 1 addition & 1 deletion DuckDuckGo/Common/Utilities/UserDefaultsWrapper.swift
Original file line number Diff line number Diff line change
Expand Up @@ -101,7 +101,7 @@ public struct UserDefaultsWrapper<T> {

case homeButtonPosition = "preferences.appeareance.home-button-position"

case phishingDetectionEnabled = "preferences.security.phishing-detection-enabled"
case maliciousSiteDetectionEnabled = "preferences.security.phishing-detection-enabled"

// ATB
case installDate = "statistics.installdate.key"
Expand Down
12 changes: 12 additions & 0 deletions DuckDuckGo/Localizable.xcstrings
Original file line number Diff line number Diff line change
Expand Up @@ -35845,6 +35845,18 @@
}
}
},
"malware.error.page.tab.title" : {
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

we need translations

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, as soon as we get our copy approved

"comment" : "Title shown in an error page tab that warn users of security risks on a website that has been flagged as Malware.",
"extractionState" : "extracted_with_value",
"localizations" : {
"en" : {
"stringUnit" : {
"state" : "new",
"value" : "Malware on this site can harm your device"
}
}
}
},
"Manage Bookmarks" : {
"comment" : "Main Menu History item",
"localizations" : {
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -31,8 +31,8 @@
switch (dataType, dataType.threatKind) {
case (.hashPrefixSet, .phishing): "phishingHashPrefixes.json"
case (.filterSet, .phishing): "phishingFilterSet.json"
// case (.hashPrefixes, .malware): "malwareHashPrefixes.json"
// case (.filters, .malware): "malwareFilterSet.json"
case (.hashPrefixSet, .malware): "malwareHashPrefixes.json"
case (.filterSet, .malware): "malwareFilterSet.json"
}
}

Expand All @@ -46,11 +46,12 @@
struct EmbeddedDataProvider: MaliciousSiteProtection.EmbeddedDataProviding {

private enum Constants {
static let embeddedDataRevision = 1694418
static let phishingEmbeddedHashPrefixDataSHA = "d9eccd24d05ce16d4ab877574df728f69be6c7840aea00e1be11aeafffb0b1dc"
// TODO: Rollback the revision and filter (with -f) set when malware is available on the server

Check failure on line 49 in DuckDuckGo/MaliciousSiteProtection/MaliciousSiteProtectionManager.swift

View workflow job for this annotation

GitHub Actions / SwiftLint

TODOs should be resolved (Rollback the revision and filt...) (todo)
Copy link
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we remove it?

Copy link
Collaborator Author

@mallexxx mallexxx Dec 5, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, this is to be removed before merging. for now it‘s there to test malware.html handling using local filters as remote doesn‘t catch it

static let embeddedDataRevision = 1695638
static let phishingEmbeddedHashPrefixDataSHA = "a9fdca1e3c852ac288b738314732847c0daf27caf3ed30a349ecaaae04682e7e"
static let phishingEmbeddedFilterSetDataSHA = "5452a5a36651c3edb5f87716042175b5a3074acb5cc62a279dbca75479fc1eda"
// static let malwareEmbeddedHashPrefixDataSHA = "be5a2320307ed0dd8b8b2f2702dbade752dfb886aae24f212b0c3009524636aa"
// static let malwareEmbeddedFilterSetDataSHA = "37517e5f3dc66819f61f5a7bb8ace1921282415f10551d2defa5c3eb0985b570"
static let malwareEmbeddedHashPrefixDataSHA = "b248a417fbd003de95fd273ae01d1b336fe2599b7d1c60b8f71786a943ab4a83"
static let malwareEmbeddedFilterSetDataSHA = "6c63a19c15868083f80b9936cafc482ae9e6f01d27246aa91789197b6f5ba9e7"
}

func revision(for dataType: MaliciousSiteProtection.DataManager.StoredDataType) -> Int {
Expand All @@ -69,8 +70,8 @@
switch (dataType, dataType.threatKind) {
case (.hashPrefixSet, .phishing): Constants.phishingEmbeddedHashPrefixDataSHA
case (.filterSet, .phishing): Constants.phishingEmbeddedFilterSetDataSHA
// case (.hashPrefixes, .malware): Constants.malwareEmbeddedHashPrefixDataSHA
// case (.filters, .malware): Constants.malwareEmbeddedFilterSetDataSHA
case (.hashPrefixSet, .malware): Constants.malwareEmbeddedHashPrefixDataSHA
case (.filterSet, .malware): Constants.malwareEmbeddedFilterSetDataSHA
}
}

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ protocol MaliciousSiteProtectionPreferencesPersistor {

struct MaliciousSiteProtectionPreferencesUserDefaultsPersistor: MaliciousSiteProtectionPreferencesPersistor {

@UserDefaultsWrapper(key: .phishingDetectionEnabled, defaultValue: true)
@UserDefaultsWrapper(key: .maliciousSiteDetectionEnabled, defaultValue: true)
var isEnabled: Bool
}

Expand Down
Loading
Loading