Use random filenames or hashes for blobstorage #4309
Assignees
Labels
Comments
iequidoo
added a commit
that referenced
this issue
Jul 18, 2023
iequidoo
added a commit
that referenced
this issue
Jul 18, 2023
|
There is an attempt to fix at #4555, but it does not consider the discussion at #3817, the issue closed in favor of this one. As a first step I suggest adding As a second step we need new APIs to avoid leaking the blob path to the UI, I have drafted them at #3817 (comment) Once blob paths are not visible in the non-deprecated API anymore and are not sent over the network, we can merge #4555 or even use completely random filenames. |
Will the attachment filename be also encrypted to not leak the file name? |
|
On Wed, Jul 19, 2023 at 14:37 -0700, dumblob wrote:
Will the attachment filename be also encrypted to not leak the file name?
yes, typically the mime attachment is fully contained in the encrypted part, including the filename.
|
iequidoo
added a commit
that referenced
this issue
Jul 25, 2023
#4309) - Deprecate dc_msg_get_file(). It's kept for compatibility and now creates a temporary directory inside the blobdir, saves a file copy with unmangled filename, and returns its path. In the housekeeping we regularly clear these directories. - Add dc_msg_save_file() which saves file copy at the provided path and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file(). - Make dc_msg_get_filename() return the original filename for the use as a default in the file saving dialog. For this purpose add Param::Filename in addition to Param::File, to store the original filename. Param::Filename is used as an attachment filename in sent emails. - Use random filename suffixes for blobstorage. Thanks to the added Param::Filename these random suffixes is not sent over the network.
|
This will also allow us to get rid of the |
iequidoo
added a commit
that referenced
this issue
Jul 26, 2023
…ame (#4309) It can be used e.g. as a default in the file saving dialog. Also display the original filename in the message info. For these purposes add Param::Filename in addition to Param::File and use it as an attachment filename in sent emails.
iequidoo
added a commit
that referenced
this issue
Jul 26, 2023
…ame (#4309) It can be used e.g. as a default in the file saving dialog. Also display the original filename in the message info. For these purposes add Param::Filename in addition to Param::File and use it as an attachment filename in sent emails.
iequidoo
added a commit
that referenced
this issue
Jul 27, 2023
…ame (#4309) It can be used e.g. as a default in the file saving dialog. Also display the original filename in the message info. For these purposes add Param::Filename in addition to Param::File and use it as an attachment filename in sent emails.
iequidoo
added a commit
that referenced
this issue
Jul 27, 2023
…ame (#4309) It can be used e.g. as a default in the file saving dialog. Also display the original filename in the message info. For these purposes add Param::Filename in addition to Param::File and use it as an attachment filename in sent emails.
iequidoo
added a commit
that referenced
this issue
Jul 27, 2023
…ame (#4309) It can be used e.g. as a default in the file saving dialog. Also display the original filename in the message info. For these purposes add Param::Filename in addition to Param::File and use it as an attachment filename in sent emails.
iequidoo
added a commit
that referenced
this issue
Jul 27, 2023
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
Jul 27, 2023
Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Jul 27, 2023
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
Jul 27, 2023
Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Jul 28, 2023
Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Jul 30, 2023
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
Jul 30, 2023
Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Jul 30, 2023
This way filenames in the blobstorage are just random hex numbers. This also allows us to get rid of the `sanitize-filename` dependency.
iequidoo
added a commit
that referenced
this issue
Aug 3, 2023
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
Aug 3, 2023
Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Mar 11, 2024
`dc_msg_get_file()` should create a temporary directory inside the blobdir, save a copy with unmangled filename, and return its path. In the housekeeping we regularly clear these directories. This API is kept for compatibility, but deprecated. Instead, add `dc_msg_get_filedata_path()` that avoids unnecessary copying if an app needs just to open a file and its name doesn't matter.
iequidoo
added a commit
that referenced
this issue
Mar 11, 2024
) This way filenames in the blobstorage are just random hex numbers. This also allows us to get rid of the `sanitize-filename` dependency. This also requires `Param::Filename` to be set to "debug_logging*.xdc" for messages containing logging webxdc-s, otherwise they are not detected properly. This is done in "fix: Message::set_file_from_bytes(): Set Param::Filename", so don't forget to update senders as well.
iequidoo
added a commit
that referenced
this issue
Mar 18, 2024
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
Mar 18, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, we add random filename suffixes for the blobdir objects. Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Mar 18, 2024
`dc_msg_get_file()` should create a temporary directory inside the blobdir, save a copy with unmangled filename, and return its path. In the housekeeping we regularly clear these directories. This API is kept for compatibility, but deprecated. Instead, add `dc_msg_get_filedata_path()` that avoids unnecessary copying if an app needs just to open a file and its name doesn't matter.
iequidoo
added a commit
that referenced
this issue
Mar 18, 2024
) This way filenames in the blobstorage are just random hex numbers. This also allows us to get rid of the `sanitize-filename` dependency. This also requires `Param::Filename` to be set to "debug_logging*.xdc" for messages containing logging webxdc-s, otherwise they are not detected properly. This is done in "fix: Message::set_file_from_bytes(): Set Param::Filename", so don't forget to update senders as well.
iequidoo
added a commit
that referenced
this issue
Apr 24, 2024
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
Apr 24, 2024
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
Apr 24, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, we add random filename suffixes for the blobdir objects. Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Apr 24, 2024
`dc_msg_get_file()` should create a temporary directory inside the blobdir, save a copy with unmangled filename, and return its path. In the housekeeping we regularly clear these directories. This API is kept for compatibility, but deprecated. Instead, add `dc_msg_get_filedata_path()` that avoids unnecessary copying if an app needs just to open a file and its name doesn't matter.
iequidoo
added a commit
that referenced
this issue
Apr 24, 2024
) This way filenames in the blobstorage are just random hex numbers. This also allows us to get rid of the `sanitize-filename` dependency. This also requires `Param::Filename` to be set to "debug_logging*.xdc" for messages containing logging webxdc-s, otherwise they are not detected properly. This is done in "fix: Message::set_file_from_bytes(): Set Param::Filename", so don't forget to update senders as well.
iequidoo
added a commit
that referenced
this issue
Apr 24, 2024
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
Apr 25, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, we add random filename suffixes for the blobdir objects. Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Apr 25, 2024
`dc_msg_get_file()` should create a temporary directory inside the blobdir, save a copy with unmangled filename, and return its path. In the housekeeping we regularly clear these directories. This API is kept for compatibility, but deprecated. Instead, add `dc_msg_get_filedata_path()` that avoids unnecessary copying if an app needs just to open a file and its name doesn't matter.
iequidoo
added a commit
that referenced
this issue
Apr 25, 2024
) This way filenames in the blobstorage are just random hex numbers. This also allows us to get rid of the `sanitize-filename` dependency. This also requires `Param::Filename` to be set to "debug_logging*.xdc" for messages containing logging webxdc-s, otherwise they are not detected properly. This is done in "fix: Message::set_file_from_bytes(): Set Param::Filename", so don't forget to update senders as well.
iequidoo
added a commit
that referenced
this issue
Apr 26, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, we add random filename suffixes for the blobdir objects. Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Apr 27, 2024
) This way filenames in the blobstorage are just random hex numbers. This also allows us to get rid of the `sanitize-filename` dependency. This also requires `Param::Filename` to be set to "debug_logging*.xdc" for messages containing logging webxdc-s, otherwise they are not detected properly. This is done in "fix: Message::set_file_from_bytes(): Set Param::Filename", so don't forget to update senders as well.
hagenest
pushed a commit
that referenced
this issue
Apr 30, 2024
#4309) ... and fails if file already exists. The UI should open the file saving dialog, defaulting to Downloads and original filename, when asked to save the file. After confirmation it should call dc_msg_save_file().
iequidoo
added a commit
that referenced
this issue
May 21, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, we add random filename suffixes for the blobdir objects. Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
May 21, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, we add random filename suffixes for the blobdir objects. Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
May 23, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, we add random filename suffixes for the blobdir objects. Thanks to the added Param::Filename these random suffixes aren't sent over the network.
iequidoo
added a commit
that referenced
this issue
Oct 2, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, store blobs in blobdir subdirs with random names. Also this helps when we receive multiple attachments having the same name -- before, random filename suffixes were added to subsequent attachments, now attachments preserve their filenames which is important if they are opened in external programs.
iequidoo
added a commit
that referenced
this issue
Oct 3, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, store blobs in blobdir subdirs with random names. Also this helps when we receive multiple attachments having the same name -- before, random filename suffixes were added to subsequent attachments, now attachments preserve their filenames which is important if they are opened in external programs.
iequidoo
added a commit
that referenced
this issue
Oct 4, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, store blobs in blobdir subdirs with random names. Also this helps when we receive multiple attachments having the same name -- before, random filename suffixes were added to subsequent attachments, now attachments preserve their filenames which is important if they are opened in external programs.
iequidoo
added a commit
that referenced
this issue
Oct 5, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, store blobs in blobdir subdirs with random names. Also this helps when we receive multiple attachments having the same name -- before, random filename suffixes were added to subsequent attachments, now attachments preserve their filenames which is important if they are opened in external programs.
iequidoo
added a commit
that referenced
this issue
Nov 15, 2024
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the `selfavatar` setting and the contact avatar `i` param pointed to `$BLOBDIR/avatar.png` at the time it was detected. How this happened is unclear, but it is possible that `avatar.png` was removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact as `avatar.png`, while `selfavatar` config still pointed to `$BLOBDIR/avatar.png`. Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling `$BLOBDIR/...` references in the database which may accidentally point to unrelated files, could even be an `avatar.png` file sent to the bot in private. To prevent such bugs, store blobs in blobdir subdirs with random names. Also this helps when we receive multiple attachments having the same name -- before, random filename suffixes were added to subsequent attachments, now attachments preserve their filenames which is important if they are opened in external programs.
|
this is now tracked by #6265 |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Recently there was an accident with a chatbot that replaced its avatar set from the command line with an unrelated avatar of a contact. Both the
selfavatarsetting and the contact avatariparam pointed to$BLOBDIR/avatar.pngat the time it was detected. How this happened is unclear, but it is possible thatavatar.pngwas removed, unmounted or otherwise not detected by the core, and the core stored avatar received from the contact asavatar.png, whileselfavatarconfig still pointed to$BLOBDIR/avatar.png.Such bugs are unavoidable even if the core itself has no bugs as we cannot rely on blobdir not reside on the faulty network filesystem, being incorrectly backed up and restored etc., so we should assume that files may be randomly removed. Then there may be dangling
$BLOBDIR/...references in the database which may accidentally point to unrelated files, could even be anavatar.pngfile sent to the bot in private.To prevent such bugs, we should choose a random filename for the blobdir objects or as a hash of the contents.
Related issue: #3817
The text was updated successfully, but these errors were encountered: