Cool, I'll do it on weekend :)

FYI, I'm not sure we want this behavior, we will need to discuss first

I'm relatively new to Docker and this had confused me at first as well. Every time I have to add files that should belong to a non-root user, I have to do RUN chown after the ADD (keeping it all above the USER line since chown needs root permission).

I had just assumed it was expected behavior and there was a good reason for it. The documentation for ADD does make it sound like expected behavior:

All new files and directories are created with mode 0755, uid and gid 0.

Also it was in original issue about chowning, that have been done by @creack. But I was too lazy to do it and @creack also didn't do it by some reasons.

2684 - original issue

I don't think making ADD set file ownership to whatever user the USER instruction has set above is a good idea. USER was initally added to specify the user which the commands executed by RUN or CMD should be run under.

@unclejack Why do you think this isn't a good idea? I think it's what the user expects: All following operations will be executed as the given user.

I just got bitten by this and gotta say it's a surprise that ADD does not respect the preceding USER declaration. If someone wants to add as root, simply run ADD before the USER is set to something else, as one would do w/ RUN.

+1 for this

Keep in mind we're recommend that people run their services as a non-privileged user. Inconveniences like this behavior will prevent people from following that.

Are there any issues running a chown after you add things?

Ref: https://github.com/crosbymichael/influxdb-docker/blob/master/Dockerfile#L9

Doesn't AUFS have issues with tracking ownership/mode changes?

@tianon yes

@crosbymichael yeah, this is very inconvenient. for example I have postgres image, there is already USER postgres in it, so I need to do:

USER root
RUN chown
USER postgres

Also I must know about USER postgres in base image.

Proposal: a second form of the ADD command. I'd love to be able to do:

ADD ["filename", "owner", "group", "755"]

I open a proposal a few days ago that addresses this issue: #7537. Essentially, after talking on IRC, it was decided that only COPY should have this particular behaviour (and ADD will be backward compatible until it is deprecated and removed).

Hi guys, any update on this issue?

Being forced to ADD and then RUN chown has a very nasty side-effect of artificially bloating docker images. For example:

FROM busybox:latest

RUN adduser -D bob
ADD 30MB_file /30MB_file
RUN chown bob /30MB_file

I ran build on this once without the chown and once with:

➔ docker images | grep bob
test    with_chown          a27913861eca        6 days ago          65.35 MB
test    without_chown       7ab2bbe29b50        6 days ago          33.89 MB

And we can see what is happening via docker history:

 ➔ docker history test:with_chown
IMAGE         CREATED        CREATED BY                                      SIZE
a27913861eca  6 days ago     /bin/sh -c chown bob /30MB_file                 31.46 MB
7ab2bbe29b50  6 days ago     /bin/sh -c #(nop) ADD file:9474e987e88bd93248   31.46 MB
e9413ccecaa1  6 days ago     /bin/sh -c adduser -D bob                       3.955 kB
4986bf8c1536  2 weeks ago    /bin/sh -c #(nop) CMD [/bin/sh]                 0 B
ea13149945cb  2 weeks ago    /bin/sh -c #(nop) ADD file:8cf517d90fe79547c4   2.433 MB
df7546f9f060  3 months ago   /bin/sh -c #(nop) MAINTAINER Jérôme Petazzo     0 B
511136ea3c5a  19 months ago                                                  0 B

When we consider that a large application might have several hundred megabytes or more, this chown becomes very expensive.

9934 has been opened to fix this issue (my patches were rejected).

This behavior really, really needs to be changed to make sure ADD uses current USER. Current way of implementing it with chown:

1. Created 3 (sic!) unnecessary layers
2. Makes Dockerfile ugly
3. Is totally not intuitive.

@goldmann agree on all points!

This topic was discussed by the maintainers and, while they agreed that having USER specify the owner of files added by ADD / COPY would have been a better choice, changing this behavior _now_ would be a breaking change, which would be too much of a risk.

For this reason, https://github.com/docker/docker/pull/9934 was created to come with alternatives. An implementation of this is created in https://github.com/docker/docker/pull/10775, which is currently under review.

For those interested, please follow the review/discussion on https://github.com/docker/docker/pull/9934 to check progress.

@kostickm is working on a proposal for this.

I think fix this problem will help to decrease images size. Specify users when ADD files is a great idea!

For example, if I ADD a binary like Hadoop, then chown it to hadoop user, the image size will increase 100+MB!

That's why I decided to install hadoop under root user:(

+1
I have an issue with permissions.
I build images for arm architecture (Raspberry) with buildroot (to generate my rootfs) and i can't use chown command since the command is compiled for arm.
ADD keywork honoring USER would be great.
Also like the proposal : ADD ["filename", "owner", "group", "755"]
Or something similar. In many case, being able to build image without the RUN command would be great for many cases, extending possibilities to multi architecture for Dockerfile.

how about a ADDUSER and COPYUSER commands which would do an implicit chown?

Hi!

I just got bitten by this too. Is there still discussion about this? What are the pros/cons?

There's currently a proposal for implementing this using command flags. The current status is best described in this comment; https://github.com/docker/docker/pull/13600#issuecomment-115913292

Hello!
We are no longer accepting patches to the Dockerfile syntax as you can read about here: https://github.com/docker/docker/blob/master/ROADMAP.md#22-dockerfile-syntax

Mainly:

Allowing the Builder to be implemented as a separate utility consuming the Engine's API will open the door for many possibilities, such as offering alternate syntaxes or DSL for existing languages without cluttering the Engine's codebase

Then from there, patches/features like this can be re-thought. Hope you can understand.

Am i doing something complete the wrong way?

RUN useradd -d /home/hybris -u 1000 -m -s /bin/bash hybris
VOLUME /home/hybris
ADD hybris /home/hybris
RUN chown -R hybris /home/hybris

Still all files belong to root.

@jfrazelle This issue is resulting in a doubling of the size cost of every chown'd file in an image.

Given myfile of 1gb size:

...
ADD myfile /tmp
RUN chown bob:group /tmp/myfile

Result:
ADD: 1 layer @ 1gb
RUN: 1 layer @ 1gb

Total: 2gb

I beg you to reconsider just closing this issue off as 'no longer accepting patches'. It's begun causing a real headache as our project has grown.

@stefanleh @lukebarton yes, it's a PITA, but please, also read the linked issues/PR for some background.

There's active work in progress to move the builder out from the daemon and once that's arrived (it's looks to be on target for 1.10), discussions around changes in the Dockerfile syntax/behavior, or alternatives can be reopened.

@lukebarton the growing image size is a real pain here
@thaJeztah any updates here (as 1.10 is out)?

@typekpb Same here, and I was expecting 1.10 to have a fix for this issue.

I currently work around the problem with RUN in conjunction with SimpleHTTPServer or git daemon, like this:
On the host: python -m SimpleHTTPServer
Inside Dockerfile: RUN wget http://${host:-172.17.0.1}/somefile.tar.gz && tar xzf somefile.tar.gz
or
On the host: git daemon --base-path=$(pwd) $(pwd)/repo
In Dockerfile: RUN git clone --single-branch myrepo

@thaJeztah I'm happy to revive my 2-year-old PR that fixes this issue, as long as I get some guarantee that it won't be blocked for "backwards compatibility issues".

@cyphar sorry, can't give you guarantees (you know I was +1 for it a the time)

Heh, yeah. It's just that there's a very strong split personality regarding backwards compatibility issues. For example, my PR fixing this issue (which everyone agrees is better than the current setup) was rejected for not being "backwards compatible" -- an argument I do not understand. On the other hand, even with API versioning, clients aren't backwards compatible with old servers (something that has actually bit us at SUSE).

I wish we'd all agree on what does and doesn't make sense to complain about.

There was, once, a suggestion to allow people to specify the user via a flag on the ADD/COPY command but that was rejected at all - despite it being backwards compatible (e.g. ADD --user=foo src tgt).

about backwards compatibility, just update the major version and call it a day.

Is there any solution? It seems this topic was discussed in many issues and lots of proposals were described, but I can't really figure out if anything better than ADD + RUN chown exists. I don't really care how, but would really love to remove this extra layer which created after ADD and completely replaced by layer after RUN chown

/me will work on a PR for this this week. Hopefully it'll get merged this time.

That would be great, thanks!

Well, not sure if doable together, but I would love to see this solved for
COPY as well.
On Mar 9, 2016 09:51, "Marc" [email protected] wrote:

That would be great, thanks!

—
Reply to this email directly or view it on GitHub
https://github.com/docker/docker/issues/6119#issuecomment-194187573.

@typekpb It's likely that it'll only be trivially solvable for COPY (ADD has several misfeatures that make this unreasonably hard -- namely the magical .tar unpacking).

EDIT: I spoke too soon, we can solve both at the same time (although decompressing archives might have some odd consequences).

I've created a PR that implements this (again): #21088.

+1 to this guy. :)

:) agreed

Shot down again. Boo.

Hi Team,
Kindly advise what is the status of this issue , because it does not make sense having this unexpected behavior of ADD

Current behavior:

ADD source_1gb.file destination_directory

RUN chown -R 775 destination_directory

Expected: 1gb
Actual:2gb

suggested solution:

ADD source_1gb.file destination_directory --USER=non_root

ADD source_1gb.file destination_directory --REQ_PERMISSION=755

just the way we have RUN command which follows below
USER non_root
RUN echo "hello i am runninga s non root"

or give us the option to run command "ADD" like "RUN" in 2 modes "root and non root "

i am planning to open new issue to have flexibility to run "ADD" command in 2 modes "root and non root "

Please fix the issue of ADD , its really expensive because its increasing size of image by adding extra layer

also we understand we can avoid this issue by maintaining proper permission of file or directory to be added , but that's extra overhead and not user friendly

Thanks
Ripunjay

+1

I too would like to see this issue resolved. The current behavior really does not make sense.

I thought I _had_ to be misunderstanding something. Is there some story that's not being told here? There's no logic in having a USER command which continues to act as root!

Read it all and still can't figure out if any of those PRs were merged or accepted.

Any news?

Cheers

I too am intensely curious whether any sane solution was implemented. I can't afford doubling the size of the stuff we copy in.

Out of curiosity, why was the idea of simply adding a COPYUSER and ADDUSER command rejected? Simply duplicates COPY and ADD but respects previous USER directives. No backwards compatibility issues, and should be fairly straightforward to implement and document. It doesn't have the same flexibility as adding --user and --group flags, but it would cover 98% of use cases and be much safer to implement.

As an alternative I've been looking at Rocker, which implements this feature, and I think it fits the bill perfectly. Rocker is a tool that builds Docker containers, extending the spec of Dockerfiles. Using it in a CI environment should be absolutely straight forward.

@Sodki beat me to suggesting Rocker.

Probably many problem with permissions and image bloat stem from two antipatterns (IMHO):

1. the same container is used to build and to run the code. Split the two step and you can keep things clean. That's where rocker comes handy.
2. apps that write in the same directories where their code is. The unix way is that program files are owned by root and processes run as nobody or another less privileged user.
   Rocker can help here as well to preserve user ownership, but it probably is still bad for security and at least error prone (user ids must be made to coincide).

Hey, after reading all the comments. I'm not sure whether there any of the solution has been merged. Still facing the same issue in docker version 1.12.3.

Anyone able to help here? The only problem is of the big image size in executing the chown command.

@kiwenlau @ripun @david-mccullars This is to comment on the additional layer size introduced by the chown command. I tested a more drastic situation on my Ubuntu system (in Oracle VirtualBox) where more than 1 chown instructions are placed in the Dockerfile, and indeed the image size gets larger and larger with additional chown instructions in the Dockerfile. But strangely such issue does not occur on our Jenkins server, which is a RHEL.

Eventually I found out that this is because the storage driver used by the RHEL server is _devicemapper_, whereas in my Ubuntu system it's _aufs_ by default. I forcibly changed the storage driver in my Ubuntu system to _devicemapper_ and the additional chown command does not introduce the new layer disk usage any more.

docker history image_size_chown

With _devicemapper_ the output is
IMAGE CREATED CREATED_BY SIZE
a87cb09fe91f About an hour ago /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/bin/ 0 B
e1f41b263f7c About an hour ago /bin/sh -c chown -R martian /home/martian/ 0 B
f3169ab0f1f5 About an hour ago /bin/sh -c chown -R martian /home/martian 0 B
e68d44c49296 About an hour ago /bin/sh -c mv /home/martian/test_images/som 59.77 MB

With _aufs_ the output is
IMAGE CREATED CREATED_BY SIZE
a87cb09fe91f About an hour ago /bin/sh -c #(nop) CMD ["/bin/sh" "-c" "/bin/ 0 B
e1f41b263f7c About an hour ago /bin/sh -c chown -R martian /home/martian/ 117.5 MB
f3169ab0f1f5 About an hour ago /bin/sh -c chown -R martian /home/martian 117.5 MB
e68d44c49296 About an hour ago /bin/sh -c mv /home/martian/test_images/som 59.77 MB

And the second image is exactly larger by 235 MB in size, introduced by the two chown commands.

Docker has documented on storage driver selection here.

The default storage driver selection can be checked via sudo docker info.
To configure docker daemon to use a different storage driver, add below line to _/etc/default/docker_ and restart docker service. Note that images created under the original storage driver will NOT be available under the new one. They can be seen again when the storage driver option is restored.
DOCKER_OPTS="--storage-driver=devicemapper"

I am still seeing this behaviour in Docker version 1.13.1, build 092cba3, and it was very surprising to me.

@robhaswell correct, there has been a lot of discussion around this, but changing it would be a breaking change. see https://github.com/docker/docker/issues/30110 for the current follow-up discussion.

Hello. I am new to docker and just discovered a bug. If I use ADD after USER statement then docker still uses root to add files. Then I tried to use COPY and still the same problem. I think this thread is about this. Is this resolved?

baaaah, you must be kidding me, it's several years now... fixing this would be a breaking change?! hordes of people get bitten and annoyed by this every single day (me included), and you are afraid of a breaking change? when the "feature" that would be "broken" is pretty much just a bug...

@attila-lendvai yes; fixing "bugs" can be a breaking change.

I'm not sure why it isn't acceptable to just make a new directive, say
COPY2 which behaves as expected and can be optionally used by people
that care without breaking anyone using the unintuitive COPY.
Everyone gets a solution, nothing breaks.

On 04/26/2017 09:20 PM, Sebastiaan van Stijn wrote:
>

@attila-lendvai https://github.com/attila-lendvai yes; fixing "bugs"
can be a breaking change.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub
https://github.com/moby/moby/issues/6119#issuecomment-297584736, or
mute the thread
https://github.com/notifications/unsubscribe-auth/AAXeDrdOfjpjbpjTHOuaJHMGIAlo5PUYks5rz-1jgaJpZM4B_uID.

@thaJeztah Yes, fixing "bugs" is a breaking change, but one has to consider how many people are relying specifically on this "feature"/"bug" and how many people have been confused by this bug or are unknowingly affected by it.

The behavior of USER brings to mind this quote:

In order to discover if this [unexpected behavior] was actually correctly used by any application program or anything really depended on it, Andrew Tridgell, the original author of Samba once hacked the kernel on his Linux laptop to write a kernel debug message if ever this condition occurred. After a week of continuous use he found one message logged. When he investigated it turned out be be a bug in the "exportfs" NFS file exporting command, where a library routine was opening and closing the /etc/exports file that had been opened and locked by the main exportfs code. Obviously the authors didn't expect [that unexpected behavior] either.
(from https://www.samba.org/samba/news/articles/low_point/tale_two_stds_os2.html)

I suspect that no one is actually relying on this "feature"/"bug" of USER because whether or not a developer actually realizes that USER doesn't affect ADD/COPY, they will be more inclined to write

ADD file/as/root
USER foo
RUN command_as_foo

than

USER foo
ADD file/as/root
RUN command_as_foo

Writing the latter is irresponsible, even if the developer was aware of how USER worked.

However, this is merely a hypothesis. Perhaps we could hold a vote or analyze public Github data to see just how many people are relying on this "feature"/"bug".

Also, if someone wants to change the owner and group owner of the files added inside the docker image via COPY or ADD, that actually increases the size of the image by the size of the files added.

It would be great to have COPY2 so that the USER with whom we're actually copying the files inside the docker image becomes the owner and groupowner.

I'm subscribed on this issue since maybe two years, and I'm really puzzled why no _solution_ has been found, whatever this looks like

One of the first arguments was that the Dockerfile format is frozen. That might have been true at that time, but does not hold anymore as there as been some changes since then (like FROM ... AS or adding HEALTHCHECK)

Since the last release the option --from has been added to COPY. So there is also no reason anymore against adding options to Dockerfile keywords.

So what are the arguments against a COPY --chown <uid>:<gid> these days?

@thaJeztah i tried to be ironic about the developers lamenting that fixing this bug would be a breaking change. no shit, who would have thought!

just the amount of discussion that happens in the numerous threads that i encountered is a lot of time wasted, both by the developers and by the users. and all that for what? trying to protect some hypothetical users who knowingly relied on the behavior that COPY, rather surprisingly, ignores USER?

what are major version numbers for? and release notes?

if there actually exists anyone who relies on this bug then they should not upgrade to a new major release, or read the release notes.

and let's say their code is broken because they didn't read the release notes... what about the tons of users who get bitten by this every single day? maybe one outweighs the other...

So, this is considered to be a breaking change while renaming the whole project is not?

This was commited a few months ago: https://github.com/moby/moby/commit/07250457df03396ea61dbf3782b5f8cf1094e7e0. I'll leave it just here.

Well, just to vent my frustration - this just caught me out (and I've been using Docker for a year now). How is this still a problem?

Wow. I'm a complete Docker newb and just ran into this one. From my virgin perspective, I absolutely expected ADD and COPY to respect USER. That they don't was a source of considerable astonishment that increased exponentially when I saw just how long ago this issue was raised and how simple the apparent fix is. I don't generally post to zombie threads, but felt obliged to add my insignificant voice to the general chorus of bafflement expressed here.

This issue was closed because ... ?

As a workaround using Alpine for me it was:

• COPY xfile location/xfile
• RUN adduser -S xuser
• RUN chown xuser -R location/xfile
• USER xuser

@ehernandez-xk Hey, Looks like you didn't read the previous comments. Though this seems to be a workaround, but this is going to increase your image size by the size of your xfile.

what if incase the files I'm copying inside the docker image is of 1Gb. I can't afford to have my image size increased by 1GB.

@italomaia Not sure, why this issue was closed?

@thaJeztah This is not fixed.

what if incase the files I'm copying inside the docker image is of 1Gb. I can't afford to have my image size increased by 1GB.

It also may take an unacceptably long amount of time.

Hey @jessfraz is the reason why this issue was closed still valid?

Perhaps there is some way to figure out how much storage space is being used on AWS to store docker FS layers that do nothing but change ownership of a previously-copied resource, and offer a bounty based upon the cost of such storage... Who knows, maybe Amazon would like a few terabytes of storage back...

On Jun 8, 2017, at 5:24 PM, David Antliff notifications@github.com wrote:

As with other issues, round and round we go. The fact that no Docker developer has reopened this speaks volumes about the way the Docker project is run. This is a common criticism of Docker's developers and it's a red flag for anyone thinking of building production infrastructure on Docker. I recommend looking at other options, as this situation doesn't look like it's going to improve.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/moby/moby/issues/6119#issuecomment-307231503, or mute the thread https://github.com/notifications/unsubscribe-auth/AYDy42P0kzpN4ALmiOSlagXjrfvm64JPks5sCGadgaJpZM4B_uID.

@ehernandez-xk @borntorock it seems we could use --squash as a workaround simultaneously with chown

+1

Personally, my biggest problem with the status quo is the amount of time the separate chown operation takes, as is seems to be copying the files as it changes ownership. Adding squash would presumably just make this worse...

On Jun 12, 2017, at 2:11 PM, Kirill Beresnev notifications@github.com wrote:

@ehernandez-xk https://github.com/ehernandez-xk @borntorock https://github.com/borntorock it seems we could use --squash as a workaround simultaneously with chown

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/moby/moby/issues/6119#issuecomment-307770992, or mute the thread https://github.com/notifications/unsubscribe-auth/AYDy42WPKw3o-dksFpTnMulbqyASLmtqks5sDSsHgaJpZM4B_uID.

TL;DR non-intuitive behavior; use chown -R to set / enforce permissions after ADD or COPY if you want them to be other than root; USER has no effect on file permissions whatsoever.

@thiagowfx Your tl;dr summarizes a little too aggressively - it bears mentioning that this will double the size of your docker image.

The additional build time is also a concern. For me at least, when adding a large archive, the chown -R seems to take longer than the original ADD operation.

On Jul 11, 2017, at 4:32 PM, Paul Phillips notifications@github.com wrote:

@thiagowfx https://github.com/thiagowfx Your tl;dr summarizes a little too aggressively - it bears mentioning that this will double the size of your docker image.

—
You are receiving this because you commented.
Reply to this email directly, view it on GitHub https://github.com/moby/moby/issues/6119#issuecomment-314562581, or mute the thread https://github.com/notifications/unsubscribe-auth/AYDy45yLHw__ewUuVJez78OrPBT5dYJbks5sM9vggaJpZM4B_uID.

It's typically worse than just a chown. Because you're likely to be "running" as a non-root user (which is why you probably encountered this issue in the first place), you need to change the root user first, and then switch back to your non-root user afterwards. It typically results in four commands:

COPY ...
USER root
RUN chown -R ...
USER <original>

It can be improved by allowing your user to run sudo /bin/chown in /etc/sudoers (one command) but that has a bad smell to it in my opinion (not to mention a security hole if you are locking the container down in other ways), so you add another line to remove it later (and then the next layer adds it back again to do the same thing...)

Surely something like the following is better for everyone?

COPY --owner user.group

Can the Docker Engine plugin interface be used to implement new commands such as this?

There is already a PR for this particular problem, but it is closed, waiting for some discussions to end. https://github.com/moby/moby/pull/28499

This is extremely painful when having to do an npm install with COPY followed by chown, and combined with the Mac filesystem performance issues! A simple COPY with chown would make a significant difference.

To save those who are interested in seeing this fixed a lot of time: here is the current attempt to implement a fix.

Wow. I'm a complete Docker newb and just ran into this one. From my virgin perspective, I absolutely expected ADD and COPY to respect USER.

Agree, this is not intuitive. USER was added to set the context for RUN and CMD instructions, and so one might assume it's more or less like an su USER and should also apply to COPY/ADD.

However, under the hood, docker is running itself and the build as root? So if root is the process doing the copy, then the resulting file is also owned by root. E.g. In an ordinary shell, when a cp command is executed, the UNIX permissions of the process running the copy command is used by default (unless special options are used). So docker implemented a pseudo user context switch, but only for some instructions and not others? This boils down to being somewhat inconsistent. Its confusing to have a user context only imply half the time...

This feature has now been implemented through an optional --chown flag on ADD and COPY through pull request: https://github.com/moby/moby/pull/34263

The feature is included in Docker 17.09 and up, and allows you to do;

ADD --chown=someuser:somegroup /foo /bar
COPY --chown=someuser:somegroup /foo /bar

Or other combinations of user/group name (or ID);

--chown=someuser:123
--chown=anyuser:anygroup
--chown=1001:1002
--chown=333:agroupname

There are two additional feature requests in this area;

• https://github.com/moby/moby/issues/35018 ("Allow COPY command's --chown to be dynamically populated via ENV or ARG")
• https://github.com/moby/moby/issues/34819 ("Proposal: Add --chmod flag to ADD/COPY commands (analogous to --chown)")

Given that the original problem was addressed (the implementation is different from the one proposed in this issue, you can read the discussion in https://github.com/moby/moby/pull/9934 and https://github.com/moby/moby/pull/9934#issuecomment-73937587 for more background on that), I'm locking the discussion on this issue to prevent it from collecting other (possibly unrelated) comments.

If you arrive on this issue because this feature is not working as expected, or you think other enhancements can be made; please open a new issue instead so that it can be tracked separately (feel free to add a link to this issue if you think it's relevant).