Syncthing: Add support for syncing ownership of files

21

It would be nice to add support for syncing file ownership with uid and gid. This already supports permissions.


2019 note: this issue was originally titled "...syncing uid:gid of files" and there's initial discussion of that, but that's too constricted hence I've retitled it. //@calmh

crosbymichael picture crosbymichael  ·  5 Feb 2015

Most helpful comment

6

Dumb thought, but why exactly does this have to involve the UID/GID/SID values directly for synchronization? Why not just store the user and group names in the metadata, and have a local, per-device mapping that translates that name to a UID/GID/SID for Syncthing? Then, as a fallback, if there's no local mapping for the name, just look it up through the usual OS channels and use the result there (if possible). That would provide a quick and easy way to sync cross-platform, eliminate the requirement for some kind of centralized user database, and probably make it work right out of the box for people who aren't syncing cross-platform without any special setup being required.

Further, you could easily just keep the particular data that isn't translatable to the local system in the database, and provide an RPC in the REST API for modifying that directly (I don't think it's likely that most users will have to modify such data on a regular basis, so I don't personally see much value to allowing users to edit it outside Syncthing), which would solve the handling of groups on Windows and ownership in general on non-rooted Android devices.

Ferroin picture Ferroin  ·  25 Jan 2018

All comments

0

Wouldn't that require running Syncthing as root?

kozec picture kozec  ·  5 Feb 2015
0

Yep that would require root.

jpjp picture jpjp  ·  5 Feb 2015
0

I am pretty sure this is unlikely to happen as uids:gids can make no sense on windows or on a device which doesn't support users (android) or even have them.

AudriusButkevicius picture AudriusButkevicius  ·  5 Feb 2015
0

You can solve the same problem by running multiple instances though.

AudriusButkevicius picture AudriusButkevicius  ·  5 Feb 2015
0

I am syncing the /Users directory between two mac, so I run syncthing as
root. It runs great.

I run into the annoyance of having uid out of sync a few times.

As a reference, I use crashplan for backups. They preserve the uid:gid
metadata and they do their best effort to preserve this attributes if you
restore on a different platform.

So adding this values to the syncthing db and trying to apply if the OS is
homogeneous is doable.

Unfortunately I am not a programmer, so I can't contribute a patch, but I
think many people will greatly appreciate this feature.
On Feb 5, 2015 3:25 AM, "Audrius Butkevicius" [email protected]
wrote:

You can solve the same problem by running multiple instances though.


Reply to this email directly or view it on GitHub
https://github.com/syncthing/syncthing/issues/1329#issuecomment-73010531
.

facastagnini picture facastagnini  ·  5 Feb 2015
0

@AudriusButkevicius I don't think windows or android is a good reason not to support this. This is the same idea as mtime and permissions.

It would not matter if this is the default or not but a configurable option to support this would go along way for use cases when using this as a deployment and configuration system for servers and not just desktop use.

crosbymichael picture crosbymichael  ·  5 Feb 2015
0

if you sync between 2 linux servers or more with SAMBA on top can UID and GID be synched then?
this would solve my problem because all samba instances would have same UID and GID on all locations for me for an example. if both systems are same linux then file ownership should be synched at least....
if same but it should note that same UID and GID should be on both sides the same maybe?

id testuser

uid 1000 gid 1000 on both locations....
one could have 1000 and the other 501 ... which are not same user....

api984 picture api984  ·  5 Feb 2015
0

@crosbymichael feel free to make a pull request.
Given it's implemented sensibly, I am pretty sure @calmh will consider merging it.

Saying Windows and Android is not a good reason not to support is not true (at least what I believe in).
Turning the code into an unmanageable mess just for the sake of it working on Unix, is a good enough reason for me. I just sadly cannot comprehend a sensible solution which works across the board.

By the end of the day, we (or perhaps that is just my personal view) are trying to make a tool which works on multiple platforms, and we try to make sure that all features work on all platforms (more or less).

I guess you could do uid:gid mapping based on the user account names across Unix and Windows, but then perhaps Unix people would expect it to work based on integers... Atleast that is the sort of behaviour I would expect syncing stuff from Unix to Windows... Not sure... gids don't even make sense on Windows.

AudriusButkevicius picture AudriusButkevicius  ·  5 Feb 2015
0

@AudriusButkevicius lets try to keep the discussion professional and not suggest that someone's contribution for a feature that they care about would be "Turning the code into an unmanageable mess just for the sake of it working on Unix"

crosbymichael picture crosbymichael  ·  5 Feb 2015
0

I am not saying that.

What I am saying, is that the only solution which comes to my mind now, would make an unmanageable mess. I guess I am a bit of an advocate of Windows here, and I don't want Windows to be left out, hence why it would be a mess.

As I said, given it's done sensibly, I am sure it could become a feature, even if an optional one.

AudriusButkevicius picture AudriusButkevicius  ·  5 Feb 2015
0

Ok thanks. I opened this issue first instead of a PR so that this could be discussed with the maintainers. I know what it's like to have feature requests implemented in a PR without any discussion.

I'm more than happy to look for an elegant solution for this. If it turns out to add more complexity than what it would be worth, then I have no problem closing this issue.

crosbymichael picture crosbymichael  ·  5 Feb 2015
0

The boss is back in a few days, hence you can perhaps propose a potential solution to the problem which then could be discussed.

There is some history of this available here (which might have some sensible ideas):
https://discourse.syncthing.net/t/sync-file-ownership-and-permissions/143

Though I feel you cannot get a way with some sort of optional remapping config

AudriusButkevicius picture AudriusButkevicius  ·  5 Feb 2015
0

Yeah, the relevant discussion is linked above. Solving this as described above (just copy uid+gid) isn't rocket science. Unfortunately it's also a half baked solution, as it doesn't work on non-Unixy systems, and will anyway not solve the other aspects that will pop up if this is implemented (sync by username and not uid, map some uids to others, etc)... As such, it would need to be _really_ neat and unobtrusive to be worth it, in my opinion.

calmh picture calmh  ·  9 Feb 2015
0

Solving this issue via copy uid+gid is good step and already will be usable via most wanted users on system with mutual user list (for example, ldap database or active directory). But in other cases uid for same user on different machines may be different, so when we sync numeric uid - on one computer user with uid 1002 may be Bob, on second - Peter.
So better solution is sync user login (name) and group name. They can be same on different systems (Linux, Windows, Mac) and this way of sync will work on mixed sync process.

MurzNN picture MurzNN  ·  13 May 2015
1

I appreciate this is a complex area. It's challenging even to work out what the best approach is, before you try and implement it. However- It's been done before, by Samba. In fact Samba goes way beyond the simple case to deal with ID mapping across LDAP domains etc., which is hopefully beyond what most syncthing users would need.

It would seem sensible to look at some reduced subset of what Samba can do as an initial design goal,

The benefits would be significant-

  1. Sync files for mutiple users beween 2 or more machines with a single shared installation and config on each machine.
  2. Maintain (or at least do minimum, controllable damage to) file ownership, permissions, creation times stamps etc., especially if files are replicated across to a machine and then "round tripped" back again to the source machine for any reason.
incansvl picture incansvl  ·  8 Nov 2017
1

I think that in order for anything to happen in this area someone needs to step up with (first) a design proposal of what the relevant subset is, detailing which situations it handles and which situations it does not, and (second) an actual implementation of that proposal. The person taking on that challenge should certainly look at prior art in the area and count on this being a several months long project to drive home.

I don't think it's helpful, in itself, to note that there are solutions that solve a different but similar problem after probably hundreds of person hours of development. That's clearly the case, but it doesn't move this issue forward. It's also self evident that there would be benefits to syncing ownership and advanced permissions.

calmh picture calmh  ·  8 Nov 2017
0

If this gets implemented, it should ideally provide some means of mapping ID's and names. People don't use the same user name everywhere (especially if they've got different operating systems involved), and there are cases where neither syncing based on numeric ID or based on name will work correctly (in fact, by numeric ID won't work most of the time when multiple users are involved, and is pretty much guaranteed not to work if syncing between multiple operating systems).

Ferroin picture Ferroin  ·  8 Nov 2017
1

It's not syncthing's task to do uid/gid name mapping, there is efficient robust dedicated tools to do that out there (like idmap itself) and it's a required service of any correctly designed domain to provide consistent uid/gid mapping. For example a samba-based domain can provide consistent uid/gid numbers based on ldap's sid accross all servers joined to the domain. If not, it means the network is misconfigured and it's not syncthing's fault.

Providing uid/gid support is only making syncthing able to take care of two metadata numbers, no less no more.

If someone have [ug]id↔name mapping problem it's a problem in his setup, it's not a syncthing problem. And I don't expect syncthing to be able to workaround user's misconfiguration problems (and please don't implement such things !).

Tools like rsync or unison just take care of uid/gid number, it's up to the sysadmin to install an id mapping service, and syncthing does not have to do more (and please do not do more than just replicating the uid/gid numbers).

I see this issue is open since almost three years now, it looks like the only real blocker is the fact the thread derivated from the very simple "syncthing needs uid/gid replication" topic (which is not much complex than permission or mtime replication) to the very complex "syncthing must be an id mapping service" topic.

Let's not expect syncthing to do things it's not up to syncthing to do or uid/gid replication will never be there…

illwieckz picture illwieckz  ·  15 Jan 2018
0

In this issue - users want to sync files with different owners, so syncthing need to have solution for this needs - keep and sync file owners (no matter what way: uid/gid, or username, group, acl, other ways), not only file permissions.
This will be very useful when you sync via syncthing as root, for example, whole /home folder, as multi-target alternative of rsync tool. This way will be much easier instead of running separate syncthing daemon for each user.

MurzNN picture MurzNN  ·  16 Jan 2018
0

For example, our company have 50 distributed computers (with slow internet connection) with about 300 users in /home folder, and I try to find the way to sync each user home folder between all computers in background via one syncthing daemon.

At now this is implemented via rsync to central server at each night, but we want online sync.

Starting 300 syncthing daemons is not so good way.

Starting syncthing as user after login will slow down login process, so user will must wait when sync process will finished after each login.

MurzNN picture MurzNN  ·  16 Jan 2018
0

Here I found a workaround for sync owners and acl permissions via Syncthing https://github.com/idistech/syncthing-acl

MurzNN picture MurzNN  ·  16 Jan 2018
0

For example, our company have 50 distributed computers (with slow internet connection) with about 300 users in /home folder, and I try to find the way to sync each user home folder between all computers in background via one syncthing daemon.

I have a similar need, badly I not only have to sync user home folders (otherwise I would just setup a crontask to rewrite the uid/gid on a per-user basis), I have to sync shared folders for working groups, it means for each shared folder everyfile has a unique gid (easy to rewrite) but also a per-user uid that belongs to the user who created the file, and _that_ metadata can't be inferred from nothing: it has to be kept, it can't be recreated with some algorithm… If the uid information is lost, it's definitely lost which is not good.

User and group ids are already shared among the whole network, so my lone need is to have syncthing syncing uid/gid numbers, nothing else.

Here I found a workaround for sync owners and acl permissions via Syncthing https://github.com/idistech/syncthing-acl

That workaround looks interesting.

illwieckz picture illwieckz  ·  16 Jan 2018
0

The workaround with syncthing-acl is interesting but doesn't work.

We've been trying for a while but it simply is too complex to function properly. UID/GID should be supported natively by syncthing.

eolix picture eolix  ·  16 Jan 2018
0

The big issue here seems to be the fact that uid/gid numbers (even numbers, let alone actual usernames) can't be stored on system that don't support it.
Consider the following usecase:
machines A and B are some unix-kind, machine C is Windows,
file X is created on A while B is offline, gets synced to C, but C has no idea of uid/gid, since Windows doesn't do that;
A goes offline, then B goes online;
B gets file X from C, but no uid/gid (obviously)
Technically, at this point X is synced across all devices, but A and B have different uid/gid for it (A has the original values, B has something default, presumably those of user Syncthing is running as. What should happen after A goes online while B is online? Sync conflict?
Resolving this properly would require storing uid/gid in index (???) even if the system doesn't support it.

nekr0z picture nekr0z  ·  24 Jan 2018
0

Resolving this properly would require storing uid/gid in index (???) even if the system doesn't support it.

Which we can do. But what if the file then gets renamed or copied on such a system? A and B each grab it, have no uid/gid values sent from C, and choose their own. But they are out of sync with each other? etc.

calmh picture calmh  ·  24 Jan 2018
0

Which we can do. But what if the file then gets renamed or copied on such a system? A and B each grab it, have no uid/gid values sent from C, and choose their own. But they are out of sync with each other? etc.

The whole point of C having uid/gid in index is in telling B the desired uid/gid at the time B gets the file from C so that B can attempt to set uid/gid right even if C can't. This way by the time A is back online B has it all figured out properly and there's no potential conflict.
As far as I understand, this would require some major overhaul in the way file properties are synced (i.e. have them synced based on index, not on the actual file). The desired logic may work something like this:

  1. node gets a file and some information about the desired permissions/uid/gid from index
  2. node adjusts the downloaded file's permissions/uid/gid information as close as it can
  3. node stores all the metadata in index so as to pass it on to the next node that asks for this file

Now, in step 2 there may be several reasons to fallback (node is a Windows system that can't store some permission information, or uid/gid; node is a Linux system running as unprivileged user that cat store permissions but not uid, etc.) but in any such case the important thing is a) for node to be aware of its limitations (which is a question, too: say node is Linux but the filesystem the repository is on is FAT - a rare but possible scenario) and b) for node to be able to pass correct premissions/uid/gid information further on (based on index, not the stored file).
Again, if I understand correctly, this would require a major overhaul. The question is whether it's worth it.

nekr0z picture nekr0z  ·  24 Jan 2018
0

That's what happens now: All syncing goes through index, not "actual files" (whatever that is), i.e. devices only exchange file metadata by index and raw block data. No problem there.

The problem of undefined uid/gid for files originating on windows could be solved by having a mandatory default uid/gid when syncing ownership as a folder parameter that must be the same everywhere.

imsodin picture imsodin  ·  24 Jan 2018
0

The problem of undefined uid/gid for files originating on windows could be solved by having a mandatory default uid/gid when syncing ownership as a folder parameter that must be the same everywhere.

Or even having something like "UID=-1" in index, so that all the nodes understand that this particular file was created on Windows and it's ok to use the fallback uid for it (until uid is explicitly changed on some unix-like node).

nekr0z picture nekr0z  ·  24 Jan 2018
2

What's the difference between a fallback uid/gid and what I called a default uid/gid?

But I think all this discussion is pretty moot: First, someone must step up who is actually interested in implementing (not having) this and then we can talk about what features and how it should be implemented. Otherwise this is just idle talk.

imsodin picture imsodin  ·  24 Jan 2018
0

How about ignoring windows machines? We're talking about UID/GID because we want to use syncthing under *nix. Make it clear that if you add a windows machine your ownership will be screwed and let's get on with it.

Really, literally, the only people caring about syncing ownership will only use syncthing with *nix.

That said, I agree with imsodin. Anyone wants to start implementation?

eolix picture eolix  ·  24 Jan 2018
2

How about just ignoring all machines and not implementing this?

AudriusButkevicius picture AudriusButkevicius  ·  24 Jan 2018
0

How about copying files manually?

If such thread exists is for a reason.

eolix picture eolix  ·  24 Jan 2018
1

Sorry, I am way past by daily beer allowance, hence the poor jokes, but I don't think "if you run windows, fuck you" is a meaningful answer. Files can originate from windows and what happens to these files in terms of ownership can't just be ignored and undefined. Windows is a crippled first class citizen and this should be handled somehow.

AudriusButkevicius picture AudriusButkevicius  ·  24 Jan 2018
0

That's alright :) My point wasn't really dissing windows machines completely. But since the file permissions sync is already in place, and already ignores them since windows doesn't have these flags, I thought it would be far easier to sync ownership in the same way, without the need of implementing a complex multi-platform solution that includes an index.

I admit hypocrisy since I haven't personally looked at the code yet, but I will this weekend and see what can be done with a path of least resistance. The long-term solution would of course be supporting all OSs, but if something can be done in the right direction right away it already is better than nothing.

eolix picture eolix  ·  25 Jan 2018
0

There is also a philosophical aspect here in that syncing ownership encourages running as root, which is something we actively discourage today.

calmh picture calmh  ·  25 Jan 2018
0

Sorry, I am way past by daily beer allowance, hence the poor jokes, but I don't think "if you run windows, fuck you" is a meaningful answer. Files can originate from windows and what happens to these files in terms of ownership can't just be ignored and undefined. Windows is a crippled first class citizen and this should be handled somehow.

@AudriusButkevicius Having the ability to sync file ownership requires a working user management system, having a working user management system is a requirement, not something to implement: it's not up to syncthing to do the user management work, syncthing just have to sync the metadata related to it.

Once you got the replicated naming system (for example an ActiveDirectory domain implemented with Samba), once you got the reproducible user-uid compute system (for example using winbind the RID way) and once you got the filesystem compatible with that (for example ext4 or btrfs), you can use any tool sharing uid/gid numbers, or any tool just have to implement uid/gid numbers support.

I only have Windows users without exception and I'm not afraid of syncthing not being able to sync uid/gid on Windows because my file servers are built to be compatible with uid/gid mechanism and the user management system is already built to be distributed and replicated.

If you want to be able to share Linux uid on Windows, you can look for a third party Windows tool to do the Linux uid→Windows SID conversion the exact opposite way winbind do (and then syncthing would be able to rely on it). What you're looking for is not syncthing, but a reverse-winbind running on Windows.

To implement uid/gid replication syncthing just have to replicate 2 integers, no less no more. If someone wants to implement a windows reverse-winbind service, why not, but it’s not a syncthing job… And it's for sure another issue. The need for a windows service implementing a reverse-winbind mapping is another issue which is not uid/gid replication.

illwieckz picture illwieckz  ·  25 Jan 2018
2

The metadata is not available outside of syncthing, how do expect any external system to do anything if it can't access it?

You can't just wedge in a random solution that works while you have no windows hosts, and whenever you have files originating from windows state it's undefined behaviour.

Also, what about ownership between windows to windows? Why is this less important that unix to unix?
I will personally try to stand in the way of getting anything merged that leaves windows feature deprived for no good reason (and so far I haven't seen any).

I don't expect to support cross os support mapping win usernames to gids and stuff like that, but if this is ever implemented I'd expect it to work the same way between unix to unix as it does on windows to windows.

Instead of trying to quantify whats syncthings problem or whats not, someone should sit down and write a proposal on how will this work. What happens if files originates on windows and goes to unix, what happens vica versa, what happens if the win user or uid/gid doesn't exist, how will gid information be exposed on windows for other tools to manage gid to windows username mapping, and vica versa for linux. What are existing tools that do that that we can recommend to our users, etc, etc.

Once we all agree and settle on how the implementation should look like, all we need then, is a superhero who cares enough about this to go and implement it.

AudriusButkevicius picture AudriusButkevicius  ·  25 Jan 2018
6

Dumb thought, but why exactly does this have to involve the UID/GID/SID values directly for synchronization? Why not just store the user and group names in the metadata, and have a local, per-device mapping that translates that name to a UID/GID/SID for Syncthing? Then, as a fallback, if there's no local mapping for the name, just look it up through the usual OS channels and use the result there (if possible). That would provide a quick and easy way to sync cross-platform, eliminate the requirement for some kind of centralized user database, and probably make it work right out of the box for people who aren't syncing cross-platform without any special setup being required.

Further, you could easily just keep the particular data that isn't translatable to the local system in the database, and provide an RPC in the REST API for modifying that directly (I don't think it's likely that most users will have to modify such data on a regular basis, so I don't personally see much value to allowing users to edit it outside Syncthing), which would solve the handling of groups on Windows and ownership in general on non-rooted Android devices.

Ferroin picture Ferroin  ·  25 Jan 2018
0

@Ferroin Dumb thought, but why exactly does this have to involve the UID/GID/SID values directly for synchronization? Why not just store the user and group names in the metadata, and have a local, per-device mapping that translates that name to a UID/GID/SID for Syncthing?

  1. There is no need to re-implement yet another id mapping mechanism when you already have one in place (basically, samba servers with trust relationship).
  2. There is less things to implement to just transfer numbers and apply them without dealing with them at all.
  3. This issue is about syncing uid:gid of files, not about implementing an id mapping mechanism.

I sometime feel this issue is somewhat held hostage by the desire for an overkill solution when there is real need for simple number syncing, and for sure this issue is hijacked by another one.

That would provide a quick and easy way to sync cross-platform, eliminate the requirement for some kind of centralized user database

People already have this in place and working. Microsoft Windows and Samba on free Unices already provide such features and organizations already have it working correctly. Tools like rsync and unison already integrate well in that scenario, but rsync is unidirectional and unison is for two distant storages only.

I'm not saying making uid/gid mapping possible without relying on a strong infrastructure may be cool to have, but it's yet another issue. See #6105 for that another issue.

illwieckz picture illwieckz  ·  24 Oct 2019
0

@calmh: I think that in order for anything to happen in this area someone needs to step up with (first) a design proposal of what the relevant subset is, detailing which situations it handles and which situations it does not

So, the expressed need there is about transferring metadata _without any transformation_, the same way a cp -a as root on Linux would do (time, uid, gid, permissions, acl…).

If some feature is not supported on some destination, a simple warning would be loggable, the same way it happens when on Linux you copy some files from ext4 to vfat filesystems and lose some metadata like permissions.

This issue is only about that, _no more_.

illwieckz picture illwieckz  ·  24 Oct 2019
0

@illwieckz You need a mapping layer, period. You can't just throw raw SID's at a non-Windows system and expect it to work, and you can't throw *NIX UID's at a Windows system and expect it to work (and I'm not even sure _how_ Android would handle either, though I expect the answer to be 'not at all').

At minimum, we need to translate the local user ID format (SID on Windows, UID/GID on others) to a name, and then sync the _name_ and not the raw ID. This is just a case of making sure it works on all systems we can make it work on, and should be trivial to implement. Hell, even between two Linux systems we should be doing this, because 99% of the time it's the user name that matters, not the exact UID. Note that most users who are not in enterprise settings don't actually have centralized authentication, especially if they're running a mix of different platforms.

If people want to use custom name mappings on specific nodes from that name to something local, then _that_ is an independent issue, though i almost guarantee that _someone_ will want this functionality. The assumption that we would be having that as part of this issue is the whole reason I suggested just storing raw values and mapping on a per-node basis, as that would be more efficient on multiple levels than having to do name translation.

Ferroin picture Ferroin  ·  24 Oct 2019
0

@Ferroin I'm not saying there is no need for a mapping layer, I'm saying the need for a mapping layer is another issue, see #6105 for that other issue.

illwieckz picture illwieckz  ·  24 Oct 2019
1

I think this issue is mistitled, which derails discussion. The issue should be "Add support for syncing ownership" or something more generic.

It was established above some time back in 2015 that just syncing uid & gid doesn't meet the bar of being a good feature in Syncthing as it doesn't work on Windows and will break in general in a cross platform setup.

So you all can argue that syncing just uid:gid without translation is easy, and it is, but that's not the point.

calmh picture calmh  ·  24 Oct 2019
0

If people want to use custom name mappings on specific nodes from that name to something local

Don't you even get me started! A simplest usecase there can be: I sync a folder from my home machine to machine at my office. At home the username is my first name (and other members of the family use theirs), at the office - my last name (and colleagues use theirs). Syncthing works perfectly as it is, and you're proposing to either break this or make it a lot of hassle to set up. And then we have a folder shared between colleagues, on both their office and home machines, and you want that to become an even bigger mess.

Not to mention that on Unix-likes changing the ownership will generally require the Syncthing process to run as root, which is opening yet another canful of worms.

nekr0z picture nekr0z  ·  24 Oct 2019
0

So you all can argue that syncing just uid:gid without translation is easy, and it is, but that's not the point.

It's the _need_.

The person taking on that challenge should certainly look at prior art in the area

Known prior art:

illwieckz picture illwieckz  ·  24 Oct 2019
0

It's a need. It's not the only need in ownership syncing, as the prior art clearly illustrates.

calmh picture calmh  ·  24 Oct 2019
0

If people want to use custom name mappings on specific nodes from that name to something local

Don't you even get me started! A simplest usecase there can be: I sync a folder from my home machine to machine at my office. At home the username is my first name (and other members of the family use theirs), at the office - my last name (and colleagues use theirs). Syncthing works perfectly as it is, and you're proposing to either break this or make it a lot of hassle to set up. And then we have a folder shared between colleagues, on both their office and home machines, and you want that to become an even bigger mess.

Or, you know, we could just default to not syncing ownership, and leave it as something that users have to enable themselves...

Then no existing installs suddenly break, setup for the common case doesn't change at all, and people who want to sync ownership in simple cases can do so.

Note that I'm not saying that this functionality isn't something we should have if we can sync ownership, just that it's kind of orthogonal to implementing the _ability_ to sync ownership (especially if we sync by name or some internal ID, then all the stuff for this can just be put in the same code that maps from the internal representation to the local ID).

Not to mention that on Unix-likes changing the ownership will generally require the Syncthing process to run as root, which is opening yet another canful of worms.

Or you run with appropriate capabilities. Or you have a dedicated helper program that only changes the name which is either SUID or has the appropriate capabilities. This is such a solved issue it's not even funny, it's just that Syncthing hasn't had any need for this type of thing, so there's no functionality for it.

Ferroin picture Ferroin  ·  24 Oct 2019
0

Yes it's _a_ need, that's why uid/gid syncing and uid/gid mapping are _two different_ features.

This issue was not mistitled. If this was mistitled, then a new issue named “_Add support for syncing uid:gid of files_” has to be created right now.

The original issue comment was not ambiguous:

It would be nice to add support for syncing file ownership with uid and gid. This already supports permissions.

It was all about syncing uid and gid the same way permissions are.

We know that simple uid/guid mapping would only work in homogeneous environment, and it's OK because it's only needed by people having built homogeneous environment. The option to enable such feature just not has to be enabled by default, the same way -a is not enabled by default when you run cp.

I still don't understand why this issue is hijacked with another one, why the title is even rewritten a way the expressed need is somewhat hidden (not being explicitly expressed anymore), and why some comments are written to encourage people to misread the expressed need not the way it was expressed (implying reading it right would be a mistake).

illwieckz picture illwieckz  ·  24 Oct 2019
0

@illwieckz Had you considered the fact that what's being discussed would also implement the requested feature, just in a way that's much more useful to a wider group of people?

I mean, yes, someone (patches are welcome I assume) could go and implement exactly what was requested. It would have to be completely rewritten however the moment someone wanted to sync by name or do sane translation of ID's, both of which would likely be requested in very short order once the original request was implemented.

On top of that, _this is going to require a protocol change_ regardless of what is getting synced. The Index messages in the block exchange protocol will need to be extended to add an entry to the FileInfo structure to transfer the ID. No sane developer makes a breaking protocol change without being absolutely certain that they won't need to make another one in the immediate future, so at minimum it needs to be determined what the format on-the-wire will be for this before it could sanely be implemented.

Ferroin picture Ferroin  ·  24 Oct 2019
0

I'm also interested in this feature and I'm also willing to contribute to the project.

Having read this issue and #6105 I learned that there is some argument about whether only uid/gid syncing is sufficient or in addition an ID mapping is needed.

Here are my thoughts for an implementation:

  • As @Ferroin correctly pointed out there has to be an protocol extension similar to what has been done in #6216. I think this should go along with that PR and repesent the unique uid/gid of the gobal state.
  • All other requirements (e.g. the setting if/not sync ownership and mapping of local uid/gid to the global state) should be fulfilled as folder setting and hence locally within the client.

I however did not really follow if this feature is wanted or not by the syncthing maintainers.
Just to ensure I'm not going to invest work for nothing: @AudriusButkevicius @calmh and others - would you consider merging a PR if I'll work on this?

aawsome picture aawsome  ·  13 Apr 2020
0

I'm leaning heavily towards "no" for simple uid/gid syncing, for several reasons but mainly that it will break in the presence of Windows or devices not configured to do the same ownership sync. The reason it will break is that when a file changes on a non-uid-syncing device it will get either zero uid/gid (Windows) or wrong uid/gid (non-uid/gid-syncing Unix device). It will also break between systems with differing uid/gid-mapping, but this is obvious and inherent to the feature...

Syncing names instead of integers is more general. It will regardless have many similar failure modes. Anyone looking to implementing this needs to consider and discuss these failure modes.

calmh picture calmh  ·  13 Apr 2020
0

I think it's a sensible feature to have, but as I said earlier I think it's a can of worms in terms of implementation, because some people will want to sync based on account/group names, some will want based on uid/guid, I think windows only has names.

So I think the first step of this would be to write an implementation proposal of how all of these pain points would be addressed.
Once we are happy with how the proposed implementation would work and what it would deliver, then I think we'd be happy to accept a PR.

Also, just to note, this has moved on a bit since some of the last posts. There is now an advanced feature flag which copies the ownership from the directory the file is being created in, which might be sufficient. This obviously requires special capabilities/root.

AudriusButkevicius picture AudriusButkevicius  ·  13 Apr 2020
1

All right, I'll start preparing a proposal :+1:

My thoughts about the pain points are as follows:

  • Each client needs to define a mapping from local user/group to the global uid/gid and another mapping from global uid/gid to local user/group (for each synthing folder). The local user/group can be (depending on the OS) the POSIX uid/gid, a string or whatever is needed for this specific OS.
  • The mapping from local user/group is defined by a given mapping table. If a local entry is not in the mapping table, make an option to either map to a (given) constant ID or (on POSIX clients) map the global ID to the identical local ID. Default (and current behavior) would be an empty mapping table and mapping to the constant ID 0.
  • The mapping to the local user/group is defined by the same mapping table as above. If a local entry is not in the mapping table, make an option to either map to a (given) constant user/group or (on POSIX clients) map the global ID to the identical local ID. Default (and current behavior) would be an empty mapping table and mapping to the user that executed syncthing.
  • Filling the mapping table is a GUI topic. I'm thinking about a possibility to import csv or JSON files..

I hope that this should cover the pain points - I there something I missed?

aawsome picture aawsome  ·  13 Apr 2020
0

I think it's best if you start a forum thread. Because you'll hear tons of opinions here with objections, demands and what not here, so I'd rather keep that to the minimum.

AudriusButkevicius picture AudriusButkevicius  ·  13 Apr 2020
0

@aawsome there is two feature requests that are messing up in that whole thread:

  1. Implementing uid/gid sync, i.e. syncing of uid and gid integers. No more. Users would have to rely on third party id mapping like Winbind. It would be cool to extend the syncing to ACL too. That was the initial, simple, feature request. We all know this would not be useful for everyone, but that's a requirement for 2. anyway.

  2. Implementing a complete alternative of Winbind or other id mapping architecture to be shipped within Syncthing, so users of 1. would not have to rely on third party tools. This second feature request requires 1. to be implemented first.

This second feature request, which is legit but can be done as a second step, requires a far much amount of work. Sadly, this second feature request added so much noise to this thread it entirely took over this thread.

So, five years later 1. is not implemented because of 2, but 2. requires 1. to be implemented first. That's part of why the thread is stalled.

While I fully understand why feature request 2 would be very cool and I may use it myself (I know how hard it is to get a complete setup like the one using Winbind), it's possible to wait for 2. to be implemented after 1. is implemented, by relying on existing third-party id mapping tools. In any way, 1. is required for 2.

I attempted to separate the issues again and created #6105 to track feature request 1. while this one would be for feature request 2, in an attempt to unblock the situation. I'm not sure the feature request 2. is going somewhere before 1. is fixed. It would probably better to reopen #6105 before spending time on this one if you worry about this one to be still open in the next five years.


_Edit: I forgot there is a third feature request messing with this thread:_

  1. _The need for a name translation infrastructure, for when user names and group names differ and tools like Winbind or feature 2. can't do the job. But that would make sense to implement 1. then 2. then 3._
illwieckz picture illwieckz  ·  14 Apr 2020
1

I don't think just option 1 would ever be accepted, because it works on one platform only, which is a no-go for a cross platform tool.

AudriusButkevicius picture AudriusButkevicius  ·  14 Apr 2020
0

@AudriusButkevicius That's not correct. All my Linux boxes have Samba and Winbind installed...so even if I were syncing between a Windows box and a Linux box, all domain accounts are available in both places. But at least in my case I'm syncing between ~24 Linux boxes which all have the same account information available.

I see two systems that can be implemented.

The first is implementing ownership/group info. IIRC Windows doesn't have the concept of file ownership--just permissions. This would cover Linux-like systems.

  1. An option to sync/not sync file ownership info.
  2. If ownership syncing is enabled, have the option to strictly sync based on the underlying user ID or SID (Linux uses UID, Windows uses SID) meaning account identifiers must be the same on both ends. This would be useful for domain-joined systems or Linux boxes that have winbind or synced /etc/passwd files.
  3. Have another option to strictly sync based on the user name lookup. i.e. if a file on one box gives 'aaron' permissions, ignore the UID/SID and look up 'aaron' on the target box and give it the same ownership.
  4. Have some sort of translation map:
    'aaron': 'asmith'
    'joe': 'jjohnson'
    etc...

The second would deal with permissions.

Similar to the first system, permission options could be something along the lines of "don't sync", "sync by UID/GID/SID", "sync by name", and "override user/group"

In my case I would choose to sync ownership and permissions either by UID/GID/SID or by lookup.

A bit of pseudo code:

if folder.sync_ownership == "disabled":
  # Do nothing

elif folder.sync_ownership == "by-id":
  set_owner_group(incoming_file_path, incoming_file_owner_uid, incoming_file_gid)

elif folder.sync_ownership == "by-lookup":
  user = lookup_uid_by_name(incoming_file_owner_name)
  group = lookup_gid_by_name(incoming_file_group_name)
  if user is None or group is None:
    # Ignore it?  Maybe an option to sync unknown users/groups to a specific user/group?
    # Maybe log the path to the file, and what user/group couldn't be resolved for easier troubleshooting
  else:
    set_owner_group(incoming_file_path, user, group)

Permissions could follow a similar format.
Attempt to resolve the names or uid/gid and have a mapping for compatibility between different operating systems. i.e. read/write/execute on a folder in Linux gets converted into 'full control' on Windows. read/execute gets converted to read/list folder contents (or something like that).

If Linux has extended ACL support enabled, have an option to sync the extended ACLs instead of the standard ACLs.

Just a thought.

darkpixel picture darkpixel  ·  16 Jun 2020
2

"That is not correct. All my Linux boxes have Samba and Winbind installed"

This is "not correct" only in your local context. I don't think every Linux machine ever has winbind installed and pre-configured waiting for syncthing to do it's thing. Also, winbind means nothing on Windows.

I don't think your comment disagrees with what needs doing, the actual implementation details may vary hence I am not going into details.

I am mostly arguing that whatever implementation lands, it should land with ability to remap/lookup/whatever you want to call it stuff straight away, not to deprive/hinder clusters with mixed operating systems.

AudriusButkevicius picture AudriusButkevicius  ·  16 Jun 2020
0

Sure.

What's the use case for "I want permissions synced" along with "...but the user accounts aren't the same on both (or all) machines"?

If I want to sync with some internet rando with permissions (and/or ownership) to another internet rando...but they don't have accounts that are in sync? I just can't see that use-case. If I were syncing family photos to my mom's computer, I don't care about permissions or ownership on the files. If I'm distributing media files or a podcast, I don't care about permissions.

But if I'm in a corporate environment where permissions matter, I already have the infrastructure set up to ensure usernames and passwords exist on the boxes....or why would I need syncing in the first place? In my case, my Linux boxes are DFS targets, as well as folder redirection targets for user profiles...so they already communicate with active directory.

Maybe on a home network where one machine has 'owner' and the other machine has 'aaron'? A simple mapping would handle that...either a list of users to map to other users, or simply "map all permissions to "aaron"....but I think that's a rare case as opposed to just letting syncthing run as "aaron" and sync while ignoring permissions.

Maybe I'm missing something?

darkpixel picture darkpixel  ·  16 Jun 2020
0

You are looking at it through a corporate prism, which is probably a minority of our users, hence I don't think we explicitly try to cater for them, but try to cater for the little guy, and I think the little guy will need this.

You yourself pointed out a case in a home network.
There are tons of other cases, such as accounts that were created before "oh I should make sure the uid's or the names match".
People don't run ADs in their house. I want to be able to map uid 1000 to my name on windows, and uid 1001 to my sons account in Windows. My uid on my mac and my linux box might not match up.

There are various appliances (synology comes to mind), that run an app under user X, but requires the files to be owned by user Y for some other app on the appliacne to be able to read/use those files. Stuff inside of docker containers falls under this as well I think.

Nobody is saying that we need complicated mappings, it would simple mappings as you yourself pointed out.

I am just saying that "just syncs uids, one to one with no remappings" as someone proposed above, is not good enough.

AudriusButkevicius picture AudriusButkevicius  ·  16 Jun 2020
0

You are looking at it through a corporate prism, which is probably a minority of our users, hence I don't think we explicitly try to cater for them, but try to cater for the little guy, and I think the little guy will need this.

I pointed out a home network case and showed that users probably wouldn't make use of the feature. What home would:

  1. set up some sort of authentication system beyond what's on the local Windows box for home users
  2. Not set up a domain
  3. Manually keep those users in sync and expect permissions to sync

Probably a lot less than the 'minority' of corporate users you are talking about.

Most home users just want files synced and working (like syncthing already does now).

Permissions are almost exclusively a corporate or power-user thing.

People don't run ADs in their house. I want to be able to map uid 1000 to my name on windows, and uid 1001 to my sons account in Windows. My uid on my mac and my linux box might not match up.

Sure--so two ways to do that in this hypothetical situation:

  1. Make sure your usernames are consistent across the boxes and syncthing could resolve names to IDs.
  2. Set up a mapping. (mac aaron) <-> (windows aaroniscool)

But like I said, I doubt that's the majority. How many home users want to do weird permission mapping stuff? I'd bet a majority of corporate users want to handle permissions. In fact, that's why we still use Windows DFSR and other hacky solutions in our corporate environment instead of syncthing. Syncthing only manages stuff where I don't care about permissions--like globally accessible folders where everyone has full control or globally accessible folders where everyone gets read-only access.

I am just saying that "just syncs uids, one to one with no remappings" as someone proposed above, is not good enough.

I agree. Maybe boil it down to four 'permission options' for a folder:

  1. do nothing (current and default behavior)
  2. sync IDs (syncs based on raw UID/GUID/SID/wahtever)
  3. sync names (look up 'aaron' on one box, find 'aaron' on the other box) with an option for what to do when a user isn't found (use default permissions, or use the syncthing user)
  4. map names (provide a list of 'remote' IDs and what 'local' ID it gets mapped to possibly with a special mapping for "remote ID isn't listed" gets mapped to a specific local ID or the syncthing user)
darkpixel picture darkpixel  ·  16 Jun 2020
0

The reason this is issue exists is not because there fundamental disagreement of what needs to be done here
I think (and I might be wrong here) the maintainers agree on the scope of work here.

The reason this is open is because nobody needs this badly enough to work on it.

There were a number of suggestions to reduce the scope "just to get something out the door", which as you say, would only be useful to people running ADs, which I am personally against, because it leaves other platforms (thinking about the specific proposal) and the little guys deprived.

AudriusButkevicius picture AudriusButkevicius  ·  16 Jun 2020
0

Agreed, however starting simple to "just get something out the door" to support the (probably) biggest use-case (permissions sync in a corp environment) wouldn't "leave other platforms deprived".

You already don't have it. You're already deprived. ;)

The reason this is open is because nobody needs this badly enough to work on it.

I'd totally dive in and work on it....if Syncthing were written in Python or nodejs...Go has been on my bucket list for a while, but I just haven't had the time...

darkpixel picture darkpixel  ·  17 Jun 2020
1

If corporations need it, they can go and sponsor someone to work on this., yet I'd expect the full implementation.

AudriusButkevicius picture AudriusButkevicius  ·  17 Jun 2020
0

Who can I sponsor and how much do they need?

darkpixel picture darkpixel  ·  17 Jun 2020
0

kastelo.net I guess is the official "commercial" body behind syncthing, but I guess you can hire someone who knows a bit of Go and html to do this.

AudriusButkevicius picture AudriusButkevicius  ·  17 Jun 2020
3

The reason this is open is because nobody needs this badly enough to work on it.

Maybe I can tell about my story: I started implementing it (just minor change when syncing just uid and gid 1:1 and also with option to sync/not sync ownership). This however needed a DB schema update and also a rescan of all folders on all devices to update the DB.

So I tested it and used it on my devices (all linux, all identical uid/gid) - works like a charm!

Then I thought about the next steps to create a PR:

  • think about mapping
  • owners in Windows -> ugly!
  • how to make a database migration -> ugly!
  • change the Web-UI to let the users enter mappings, etc.
  • what to do if only ownership of a file is changed??

So, finally I left with just using my patched version which is still working like a charm.. :wink:

Maybe if I find more time, I'll tackle the above mentioned points and prepare a PR.

Or, if wished, I can simply share my modifications, maybe it is enough to opt in/opt out and someone can tackle the DB migration topic such that we'll only get an optional ownership syncing that only works for linux but doesn't harm anything on Windows.

aawsome picture aawsome  ·  17 Jun 2020
0

@aawsome I'd love to get a copy of the patch and buy you a beer or something from an Amazon wishlist... I only need 1:1 uid/gid mapping on Linux since all my NAS boxen have identical IDs through winbind.

Or if you want to try getting a PR submitted or something, I might be able to do some sort of sponsorship through my company or one of my clients that needs it.

darkpixel picture darkpixel  ·  17 Jun 2020
0

@aawsome Could I get a copy of your changes please? I have 2 linux boxes I'd like to sync. I would value what you have even if the corner cases you outlined have not been addresses. Cheers!
@darkpixel Did you ever get a copy of the uid/gid syncing patch? If so, could I get a copy?

hiveai picture hiveai  ·  29 Aug 2020
1

You can find the patch here : https://github.com/aawsome/syncthing/tree/sync-owner
I will soon test it on a linux multi-device configuration (if I reach the end of a custom build on docker), I will keep you informed!

Striffly picture Striffly  ·  25 Nov 2020
1

Sorry for not answering. This is the patch I'm still using, however it is based on syncthing 1.5.0 and there have been quite some changes in the master code since then.
I wanted to have a look how to rebase my patch but so far I didn't find the time for it. (and, honestly, after some time forgot about this topic).
Appreciate any help with rebasing this!

aawsome picture aawsome  ·  25 Nov 2020