Syncing symbolic links as reference

[david-jointech]

Currently when I add a symlink inside my Nextcloud folder it doesn't sync because "Symbolic links are not supported in syncing".
I am trying to use Nextcloud as a home for my data, to have it synced between my PCs. Now I'd also like to be able to access my git-repositories (mostly) code via the directory-structure in Nextcloud but I don't want to sync the git-repositories via Nextcloud. This is why it would be really nice for me to be able syncing only the link to the repository which is on a fixed position on any system. eg: ~/Nextcloud/current_projects/fizzbuzz would point to ~/repositories/fizzbuzz.
If I happen to be on a system where this git-repository isn't present I can just clone the repository there.
This option could be optional (like the syncing of hidden files).

There's also a issue about this for the owncloud-client here: owncloud/client#1440

[camilasan]

I see the use case, but we also would have to deal with windows links on mac, mac links on Linux... you get the drill. So between confusion and technical difficulty, and adding that most users don't even use symlinks all that much, this mostly seems a low-priority thing for me. Of course, anyone is free to work on it but he/she will have to do a full (cross-platform) solution, and that won't be easy.

[camilasan]

For reference: https://help.nextcloud.com/t/symbolic-link-support/220/27

[david-jointech]

Would the proposed implementation in point three of this post https://help.nextcloud.com/t/symbolic-link-support/220/18?u=callegar be a viable way to solve the problem?

[ferdiga]

I agree with the arguments mentioned in the link above.
a symbolic link is a very useful way (especially on Linux) to extend local storage capacity using other partitions and / or disk. at least on Linux the nextcloud client shouldn't exclude links as these are transparent to all Linux programs

[nefelin]

I understand the difficult involved but I would love to see a solution to this. Or at least an option to sync links across compatible platforms...

I use symlinks to sort a very large image collection that I use for drawing practice into smaller collections, which lets me have image sets exist in multiple collections, it is very useful and intuitive but would be even better if it synced.

[m1cm1c]

I see the use case, but we also would have to deal with windows links on mac, mac links on Linux... you get the drill.

Are you saying that if Nextcloud were to support symlinks, it would have to be able to convert them to *.lnk files on Windows? Converting between symlinks and *.lnk files is impossible in either direction, simply because you can't find a generic way of converting between paths on Linux and paths of Windows in either direction. Why not simply only create symlinks on the other system if it's also a Unix-like system? You can't unify Windows with the rest. They chose to be fundamentally incompatible with Linux / Mac / FreeBSD / OpenBSD / Solaris / whatever.

On the windows client, you can choose between:

• doing nothing
• dumping a file there that states where the link leads on a Unix-like system

I suppose you currently just dump Windows' *.lnk files as what they are on Unix-like systems, even if they are of no use there (well, they hardly are of any use on Windows too because they can't be used in the middle of paths, but that's a different topic).

[fwsGonzo]

No one is asking for conversion between symlinks and "windows links". There is no one-to-one comparison anyway. The dream solution here is to just not care that you found a symlink on linux and just treat it normally. If its a folder, its just a folder - extending storage. I can`t understand why the nextcloud client is hindering this to begin with, as you would have to actively disallow it.

EDIT: This is actually making it hard to use this cloud for me at all. I absolutely needed this feature to work exactly like Dropbox (and others).

[piranhaphish]

I agree that this is needed. I would go as far as to say that ignoring symlinks is a bug. Like was mentioned above, a symlinked file/folder can simply be treated as a normal file/folder; you have to intentionally go out of your way to avoid doing so (by using lstat() instead of stat(), for instance).

I don't think anybody is asking for the symlink itself to be represented on the cloud-side, moreso just simply recognized by the client side.

[david-jointech]

I actually was asking for symlinks to be synced as "symlinks" and not to sync the content. The main reason for me is that I want to work out of my nextcloud-directory by default (I have an organization system in place there) but I do not want to sync git-repositories via nextcloud.

[AncalagonTheBlack]

Such a Feature would be awesome. So many times it would come handy and today again.

[alexeymuranov]

I've noticed some seemingly random behaviour of my nextcloud-client 2.5.0 on Ubuntu (installed from nextcloud-devs PPA) with respect to symlinked folders inside a synchronised folder.

I first noticed that files in some local folder A which was not synced itself, but was symlinked from inside a synced folder S_local seemed to have disappeared by themselves. I looked into it and found that in the local synced folder S_local i had a symlink A to the "external" folder A, but in the remote synced folder S_remote there was an actual non-empty subfolder A (with the same name) in that place. When trying to sync, the nextcloud client was simply wiping the contents in my local folder A (which was just symlinked from inside S_local), and it was not downloading anywhere the contents of the remote actual folder A.

So far i didn't manage to reproduce this behaviour with a fresh folder (but i can reproduce it in the A folder). However, i observed some other strange behaviour. For example, i have currently a symlink to a folder locally, but a real folder with the same name remotely, and their contents is only partially synchronising.

Is this related to any known bug? I am not ready to report it myself, because i cannot yet reproduce the file-eating behaviour from scratch.

[alexeymuranov]

After some more investigation, i decided to report the problems i observed, because when synchronisation client removes files for no reason, it is a serious problem IMO: #899.

[fkbreitl]

Like many others I also need to sync my folders via symlinks and request this important feature.
From what I read above there are no good reasons not to support it and other cloud clients like Dropbox and tools like rsync support it, too.
It is also not necessary to support all platforms at once. You can start with one platform and later others.
One just needs to start and it is a rather simple thing to do.
So what are the plans?

[sphakka]

Some ideas might come from this recent update to rclone

[basos9]

EDIT 2023-10-22: This node is actually for Symlink dereferencing #3335
removed comment
see referred issue

[jcklpe]

I'm actually having this problem with the client right now. It has for a long time complained to me about symlinks but now it's refusing to sync them at all, which is causing problems for me. Any suggestions on how to fix this in the short term please let me know!

[rickdoesdev]

On the point that is commonly being raised as a counter re crossplatform; it's worth noting that Windows/NFTS -does- support symlinks (and hardlinks, can we please preserve hardlinks while we're at it) which are read and followed same as linux. It's had this support for years and years but hardly anyone uses them. Lately MS has been desparatly trying to claw back a developer user base that was migrating away so they've made the symlinks more obvious; and seem to have removed the previously required privilege escalation required to use them, so sym and hardlinks are a 1st class citizen on windows. .lnk files should not factor into this feature request at all.

It's been many years since I've been on a mac though, so I'm not sure what proprietary nonsense apple is doing, but osx is bsd, and last I checked it also still used sym and hardlinks under the covers (ln / ln -s command at least, I assume the fundamental node created on the HDD is the same)

(Edit: for people wanting a nice explorer ui for handling links on windows; http://schinagl.priv.at/nt/hardlinkshellext/hardlinkshellext.html is an amazing bit of software and a day one installation for me on any fresh windows setup)

[jcklpe]

I can confirm the moves by MS to include symlinks and hardlinks more and the link shell extension linked by rcuddy is cray invaluable. Highly recommend!

[scholer]

On the point that is commonly being raised as a counter re crossplatform; it's worth noting that Windows/NFTS -does- support symlinks (and hardlinks, can we please preserve hardlinks while we're at it) which are read and followed same as linux.

I really would like to emphasize this point by rcuddy. Symbolic links are super valuable as-is, which is why Microsoft has supported them for a long time. While symbolic links weren't originally supported in Windows XP, they have been supported since Vista. NTFS has full support for both symlinks and hardlinks (and junctions!). Microsoft initially made the mistake of requiring admin privileges to create symlinks, mostly out of fear of how Windows XP apps would react to symlinks. But Microsoft has now realized that symlinks has fantastic value when organizing the filesystem (just see how ubiquitous symlinks are Linux!). Microsoft has steadily been making it easier and easier for normal users to create symbolic links. (Last I checked, to allow normal users to create symlinks, you still either had to enable "developer mode", or use "Local Security Policies" to allow non-admin users to create them. But I imagine Microsoft will eventually remove this restriction - and until then, it wouldn't be too difficult to just ask NextCloud users to configure this if they want to synchronize symlinks.)

My recommendation is as follows:

Preserve symbolic links, as-is. Relative symbolic links are practically the same on all systems. That is, if I have a relative symlink pointing to ../very/deep/sub/directory/, that symlink can be preserved across all systems. Absolute symlinks are perhaps a bit trickier because of the NTFS drive-letter prefix (although Windows also support "absolute" paths without a drive letter, e.g. cd \Users to go to C:\Users if you are working on the C-drive, and D:\Users if you are on the D-drive). My recommendation would be to ignore symbolic links with a drive-letter specification, with an option to strip the drive-prefix (e.g. C:\Users is converted to /Users). Alternatively, absolute symlinks could be ignored entirely.

If people wants to "sync folders outside the NextCloud folder", NTFS junction points is the obvious solution: They are basically directory hardlinks*, but can be used to link to directories on other drives/partitions.

(*Junctions are not actually hardlinks, since making actual directory hardlinks opens a can of worms. Junction points uses NTFS "reparse points", which basically tells the software to stop and reparse the path in a specific way, depending on the type of reparse point.)

Of course, it would also be possible to just have client-side settings to configure the symlink sync behavior, e.g.

• Relative symlink:   ☒ Sync, as-is   ☐ Sync, dereferencing link   ☐ Ignore.
• Absolute symlink: ☐ Sync, as-is   ☐ Sync, dereferencing link   ☒ Ignore.

PS: Windows' .lnk shortcut files have always been an inferior substitute for symbolic links: They must be referencing an absolute path, and they cannot be traversed/followed by e.g. the terminal/command prompt or search utilities.

Refs:

• https://docs.microsoft.com/en-us/windows/desktop/fileio/naming-a-file
• https://www.tuxera.com/community/ntfs-3g-advanced/junction-points-and-symbolic-links/
• https://www.2brightsparks.com/resources/articles/NTFS-Hard-Links-Junctions-and-Symbolic-Links.pdf

[Adrien-Luxey]

Hi there, just want to provide my use-case where I don't want NextCloud to follow the symlinks and synchronize their content:

I have this big "Code" directory (with 100MB+ CSVs and the like) that has been causing constant 100% CPU from NextCloud for a while. You might know that it is not trivial to exclude a folder from synchronization locally (to ask NextCloud to stop trying to upload a folder to the server).
Following suggestions in the aforementioned Reddit thread, I just moved my "Code" folder outside of the synced directory, and created a symlink to its previous location.
This solved my problem. So I'm glad symlinks are not followed by NextCloud, as I don't see any other easy solution to prevent uploading subfolders of the synced directory.

[taminob]

Just wanted to give you all a short update because I made some progress after experimenting with different server representations.
In the end, I settled on a regular file in the filesystem containing the symlink target. Additionally, to determine if a file is a symlink, I added a new table to the database (oc_symlinks).
Using the existing oc_filecache to store the type as e.g. mimetype has a couple of disadvantages like that occ files:scan --all will overwrite any manually set mimetype. Also, I would have a pretty bad feeling with storing essential data in something called "cache".

Using this, I successfully synchronized basic symlinks to and from the server (PROPFIND, PUT, POST, GET and DELETE).
However, there are still lots of bugs left which I'll have to fix one by one. Like that the symlinks are getting synchronized every time or that the symlinks aren't getting deleted from the database.
Writing tests, cleaning up the code and PRs and maybe indicating symlinks in the Web UI are also still on my TODO list.

[taminob]

Next update - I resolved the issues mentioned before (and lots of others as well). Using the modified server and client, uploading/downloading/deleting/renaming of symlinks seems to work.

I mentioned the currently known limitations in the PR for the server (they all only affect the server side), namely:

• symlinks restored from the trash bin will be re-created as regular files with the target as their content
• symlinks are not indicated in the Web UI
• symlinks are copied as regular files with the symlink target path as their content (if copied via Web UI)

Additionally, I tested nothing on Windows so far (might not even compile) - most of the implementation would actually also work for .lnk files (so "shortcuts") since Qt handles shortcuts as Windows' symlinks. If that or native NTFS symlinks should be used, might require further discussion since it will be hard to change once it is released with either option.

@basos9 if you're still interested in trying out the changes, I'd appreciate any feedback - I am sure that I missed a lot of bugs since my testing did only cover the most basic cases so far.

[f1d094]

@taminob: First - thank you for your hard work!

Second: "symlink target as their content" sounds a lot like dereferencing the symlinks. I'm gathering that you mean a file that is a special on the serverside, and has the client-side link name as the filename and the contents server-side are just the text of the path of the actual symlink, or something similar? You may want to clarify in your comments, "symlink target" has specific the connotation of being the actual file that the symlink points to.

Also a quick thought: When saving the symlinks, be sure to use relative vs full-path symlinks as they were created on the source system.

Forgive me if this was clarified previously. I've not had the bandwidth to follow the developments and conversation over the past few weeks.

[taminob]

Second: "symlink target as their content" sounds a lot like dereferencing the symlinks.

@f1d094 thank you for pointing out the ambiguity. "symlink target path" is maybe more accurate, the symlinks are not dereferenced - so e.g. broken symlinks can be synchronized as well.
Also, the "raw" symlink is uploaded, so if it is relative, it remains relative and if it's absolute, it remains absolute. The value returned by "readlink" is used.

[AkechiShiro]

What's currently stucking this feature from rolling out ? There are two PRs right now #6205 (client) nextcloud/server#41321 (server side)

[AkechiShiro]

Given this was discussed since 2018, such a feature missing is a bit sad.

[AkechiShiro]

I feel like the client Nextcloud has lacks a lot of polished feature, this does not feel like there is a company behind.

Testing of feature such as file syncing seems to not even be implemented or properly handled. As files do not sync with the latest client from upstream.

[f1d094]

@AkechiShiro Well, it appears the team just doens't give a shit that their project breaks a core function of the Linux and/or Mac operating systems. As @camilasan stated above, they seem to think that very few users use this...so to hell with the rest of us. @taminob put a ton of effort into a PR for which he's getting the middle finger because symlinks don't sync across to windows systems easily and nobody can get their heads around the idea that to make this an easy implementation all you need to do is NOT SYNC SYMLINKS TO WINDOWS SYSTEMS. jfc.

@joshtrichards has closed the other issue altogether. First noting it as a dupe of here, but listing the reason for closing the thread as "not planned" (#5509 (comment)). That's not a good sign.

You would think that someone on this project would be an actual linux user. The linux client should never have been made/released without support for this very-basic-function. It's just embarrassing...and frustrating...and infuriating.

Does anyone know of another sync-tool that actually knows how to support linux properly? Owncloud? Syncthing? Other-whatever?

[joshtrichards]

@joshtrichards has closed the other issue altogether. First noting it as a dupe of here, but listing the reason for closing the thread as "not planned" (#5509 (comment)). That's not a good sign.

For the record, the "not planned" is GitHub shorthand for any issue closed for any reason other than "completed".

You would think that someone on this project would be an actual linux user. The linux client should never have been made/released without support for this very-basic-function. It's just embarrassing...and frustrating...and infuriating.

I'm a Linux user and I don't need this feature. That doesn't mean you or someone else doesn't.

[ferdiga]

Does anyone know of another sync-tool that actually knows how to support linux properly? Owncloud? Syncthing? Other-whatever?

IMO - rsync (on linux) has all the options - do not know if rsync on windows supports all these
--links, -l copy symlinks as symlinks
--copy-links, -L transform symlink into referent file/dir
--copy-unsafe-links only "unsafe" symlinks are transformed
--safe-links ignore symlinks that point outside the tree
--munge-links munge symlinks to make them safe & unusable
--copy-dirlinks, -k transform symlink to dir into referent dir
--keep-dirlinks, -K treat symlinked dir on receiver as dir
--hard-links, -H preserve hard links
--omit-link-times, -J omit symlinks from --times
--link-dest=DIR hardlink to files in DIR when unchanged

[f1d094]

@ferdiga Appreciated. I am well-familiar with rsync and use it daily. It isn't the tool for the job however...otherwise nextcloud/owncloud/syncthing/etc wouldn't even exist. ;) A centralized setup with many-to-one client/server setup with mobile access via webdav is a pretty rich setup...or it would be, if it supported-instead-of-broke symlinks.

Much like Nextcloud was unknown to me until a couple of years ago, I was hoping a competing-but-similar project might be out there that is more linux-friendly.

[f1d094]

I'm a Linux user and I don't need this feature. That doesn't mean you or someone else doesn't.

@joshtrichards I don't understand how you can be a linux user and not use symlinks on the regular...but to each his own. Either way, what seems to be the holdup on this? Symlinks are a core file type for OSX and Linux and somehow this core function has been ignored for YEARS. It is a very small lift compared to many things. I came here a year or so ago to do my own PR because it is just ridiculous that it isn't implemented but then found that @taminob already had one in process...and it was being more-or-less ignored.

As linux users we have to put up with all-manner of garbage that doesn't do anything on our OS that is pushed over from windows (or OSX) systems...but a feature that should be very minor to implement somehow sets everyone's hair on fire because it won't translate to windows. Who cares? Their OS doesn't support symlinks so simply don't sync them there. They miss out on precisely nothing and those of us who use them don't get their filesystems interfered with.

It isn't a big ask. There is a developer who is willing to do the work. Why won't anyone work with him?

[olaulau]

AFAIK, there are some things that approximate to symlink on windows, such as NTFS hardlink, shortcuts ... If we would, we could translate symlink to something else. Or maybe copy the files, or ignore them (user could choose behavior in settings).
Also, we could have different behavior concerning the destination of symlinks, depending on they pointing inside the mirrored directory or outside.
But I totally agree, symlinks should be supported, especially on an open-source project which is so close to the linux community !

[f1d094]

Does anyone know of another sync-tool that actually knows how to support linux properly? Owncloud? Syncthing? Other-whatever?

To answer my own question and for anyone else who is curious: Syncthing (https://syncthing.net/) appears to support symlinks but with some caveats. Unfortunately, there is no webdav access which is a non-starter for me. But if you sync only between linux systems and your phone and don't care about webdav or the extra junk in Nextcloud, Syncthing might be the project for you.

I have not tried/tested since webdav access is higher priority for us but would love to hear from any linux users out there who have tried it.

[taminob]

I don't think the discussion here will reach the server team who rejected it because of inter-OS compatibility (see nextcloud/server#41321 (comment)). I'm actually also fine with having this feature as a server app so that you manually have to enable it - as long as it can be used, I'll be happy (Linux users are usually used to things not working out-of-box :| ).

I'm currently stuck because I'm waiting for feedback on the desktop client PR (see #6205 (comment)) if it would be possible to include it even though it's "just" an app (similar to the E2E encryption which is also an app and included in the desktop code base).

[f1d094]

I'm actually also fine with having this feature as a server app so that you manually have to enable it - as long as it can be used

I don't care if I have to stand on 1 foot, pat my head, and rub my tummy while whistling the Star-Spangled Banner… whatever it takes. All of our systems are highly customized. One more mod would be a raindrop in the ocean.

Good luck getting anyone to actually work with you.

[jellyterra]

If it is not compatible with Windows, JUST make it available on Mac and Linux. 😅
Don't throw the baby out with the bathwater.

[taminob]

If it is not compatible with Windows, JUST make it available on Mac and Linux. 😅 Don't throw the baby out with the bathwater.

That argument came from the server maintainers and I don't think they read issues in this repository (if any Nextcloud team member looks into this issue at all anymore).

But I'll probably be too busy for this for the next 12 months anyway, so don't expect any progress in the near future.
If anyone else wants to continue the work in the meantime, feel free to reach out to me and I'll provide help with the existing changes as far as I can (not that I recommend the contribution experience without getting paid :) ).

[doctorscott]

Supporting symbolic links should be incredibly easy, even on Windows! It is just a file that has a resolvable path. Well, it is a slightly bit more than that, but still. Right now, it converts working directory symbolic links to hard links or just an empty directory. Again, this should be easy! On windows, create a symbolic link with: mklink /d "link" "target". Try this and sync and look on another computer that is synced. It is not even close! Honestly, this is not really a feature request, but a bug fix request.

[RuizuKun-Dev]

would be nice to have support for this, non of the Cloud Storage software do unfortunately

[ikcalB]

+1

all OS support links.
option to use OS defaults, make plain text file, file with original link path / etc. are all feasible - imho the only question is about a (sensible) default?

[SchroedersKater]

There is also another good reason for symbolic links here.

[taminob]

Here is a short summary for everyone who doesn't follow the PRs and also doesn't want to read this entire issue discussion:

• I created a proposal for this feature in the PRs Feature/sync unix symlinks as links #6205 and #41321 (nextcloud-server)
• The PR for the server was rejected by the maintainers, but it can be easily modified to work as an app
• Since the desktop app does not support apps, it needs to merged into the main code base
  -> this approach got the approval of the desktop app maintainers quite recently

So we are on a good way to actually get this done, but the remaining development efforts will be:

• Transforming the server code into an app
• Finalizing the desktop PR (adding automated tests, resolving merge conflicts)

I personally won't have enough time for this for the next couple (~10-12) of months, but will try to get this done as soon as I find the time to spare. If anyone else wants to give it a try in the meantime, feel free to write me so that I can give you some more context for the current state of the PR changes.