LibClamAV Error: cl_scandesc_callback: Can't fstat descriptor 3 when scanning a large video file( 2.12 GB )

[eangelov]

Describe the bug

When trying to scan a large video file( 2.12 GB ) an error is thrown"LibClamAV Error: cl_scandesc_callback: Can't fstat descriptor 3"

OS Version: Windows 10 Pro 21H2 19044.1526 64bit
Clam Version: ClamAV 0.104.2/26460/Mon Feb 21 11:25:32 2022

How to reproduce the problem

Try to scan the video file via clamscan.exe
clamscan.exe "D:\clam stuff\vid_1.mp4"

ClamAV command output:

Loading:    19s, ETA:   0s [========================>]    8.61M/8.61M sigs
Compiling:   3s, ETA:   0s [========================>]       41/41 tasks

LibClamAV Error: cl_scandesc_callback: Can't fstat descriptor 3
D:\clam stuff\vid_1.mp4: Can't get file status ERROR

----------- SCAN SUMMARY -----------
Known viruses: 8606376
Engine version: 0.104.2
Scanned directories: 0
Scanned files: 0
Infected files: 0
Total errors: 1
Data scanned: 0.00 MB
Data read: 4192384.15 MB (ratio 0.00:1)
Time: 23.260 sec (0 m 23 s)
Start Date: 2022:02:22 15:20:30
End Date:   2022:02:22 15:20:54

Attachments

Link to test file

[Sod-Almighty]

Anyone plan to fix this incredibly serious issue that makes it impossible to scan large files on Windows? It's been over 18 months since this bug was reported.

[micahsnyder]

Anyone plan to fix this incredibly serious issue that makes it impossible to scan large files on Windows? It's been over 18 months since this bug was reported.

I am able to reproduce this error when trying to scan large files. We should fix it. But, we don't support scanning files larger than 2GB anyways. This feature limitation sucks for everyone (not just Windows), but it is certainly not "incredibly serious" as once we fix this bug you still won't be able to scan large files.

[Sod-Almighty]

we don't support scanning files larger than 2GB anyways

Why not?

[micahsnyder]

we don't support scanning files larger than 2GB anyways

Why not?

A bunch of clamav code was written using 32bit signed and unsigned integer variable types that can't handle file offsets larger than 2GB or 4GB. The code needs to be thoroughly audited to upgrade any of those variable types. Without a careful audit and upgrade, large files may cause math operations to fail unexpectedly which could have disastrous consequences.

It's not to say that we don't WANT to support larger files, but we have other priorities and we have a particularly small team right now.

[Sod-Almighty]

I hate to be the one to break it to you dude, but uh......64-bit has been standard for over a decade, my man. And many files are more than 2GB these days. An antivirus unable to scan a file larger than 2GB is de facto practically useless.

I'd say it should be your number one priority.

[micahsnyder]

I hate to be the one to break it to you dude, but uh......64-bit has been standard for over a decade, my man.

You don't have to tell me. But this software is 2-decades old and is a huge codebase. It takes work to update everything.

And many files are more than 2GB these days. An antivirus unable to scan a file larger than 2GB is de facto practically useless. An antivirus unable to scan a file larger than 2GB is de facto practically useless.

In ClamAV 0.105 we bumped up the default max file size for ClamAV. The result was a bunch of complaints from users about scan time. And then VirusTotal manually set lower limits for ClamAV scans in their service, because scanning files that big is not scalable. It takes too many resources.

But speaking practically, ... nobody is distributing 2GB+ malware. And none of our signatures are designed for files that big or tested against clean files that big. So you're much more likely to encounter false positives in huge files.

I'd say it should be your number one priority.

If you paid all our bills then setting priorities could be your job! Unfortunately for you, we have professional threat researchers that help us set priorities for the project. For them, higher priorities include things like:

• improvements to macro and script extraction,
• improvements to packed PE extraction,
• improvements to our signature language,
• fixing or adding support for unsupported archive formats to include things like OneNote document attachments extraction,
• finding new ways to detecting malware (such as identifying images distributed with malware and phishing).

Other high priorities for me include stuff like:

• fixing security issues as soon as possible when they're reported,
• keeping ClamAV up to date with LLVM library API changes so it can still be compiled,
• shoring up other security concerns. A big one a couple years ago was moving On-Access scanning out of ClamD so you don't have to run a malware scanner written in the least memory safe language (C) as root in order for it to function. A couple upcoming ones are:
• ditching MD5 hashes for the clean file cache and the signature database,
• scan process sandboxing so the scanning logic is further isolated to protect against the threat of code execution and denial of service vulnerabilities.

But there are simply too many other high priority things for me to list.

In short, there are a ton of things that someone thinks should be our number one priority. I thoroughly appreciate your needs for your use case, but this concern is far from our actual number one priority.

But I do have good news for you. We have a project in progress to unpack (or mount) and scan large archives. It's close to done and we'll share it when it is ready. It is a practical solution to work around ClamAV's present file size limitations.

[Sod-Almighty]

nobody is distributing 2GB+ malware

Incorrect. Auto-extracting archives are one example. "Onefiles" – which are fairly common on Linux – are another. AppImages are a third example. There are others. And mounting an archive only works with common archive formats.

But I'm clearly wasting my breath. I wish you the best of luck with your so-called "priority list" for a product that was obsolete a decade ago.

[micahsnyder]

for a product that was obsolete a decade ago.

Listen mate, we're doing the best we can with the resources we have to provide you something absolutely free at no cost to you in anyway, whatsoever. You are absolutely wasting your breath because you're being a complete jerk.

Take a moment to reflect before you go trash talking strangers on the internet.

[Sod-Almighty]

Oh, it's completed is it? That mean I can scan 2GB files now?

[micahsnyder]

The GitHub issue for scanning larger files is #344