Yarn: Yarn install does not install missing modules in node_modules

120

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

Missing module directories in node_modules are not installed when running yarn install after you get the message "success Already up-to-date". Please note that a work around is to always run yarn install --force, however that slows down yarn install if no new modules are needed. Also everything seems to work with yarn install if the node_modules directory is deleted.

If the current behavior is a bug, please provide the steps to reproduce.

  1. Add a module (i.e. gulp-concat) through yarn -> "yarn add gulp-concat"
  2. Run "yarn install" until the quick return with "Already up-to-date"
  3. rm -rf node_modules/gulp-concat
  4. Run "yarn install"
  5. It will quick return with the message "Already up-to-date" without reinstalling gulp-concat

What is the expected behavior?

If a module is missing from node_modules (i.e. directory is not there) yarn should install the missing module and not report everything is up to date.

Think of a multi developer environment where one developer installs a module and updates the source control with the updated yarn.lock file and/or offline cache but not the node_modules directory. Another developer updates and then runs yarn install, which should not report back that everything is up to date.

Please mention your node.js, yarn and operating system version.

yarn 0.17.10
node.js 6.9.2
os CentOS 6.7 (I also tested on Windows)

classicspam picture classicspam  ·  13 Dec 2016

Most helpful comment

201
pribilinskiy picture pribilinskiy  ·  11 May 2017

All comments

0

Any news on that? Thx.

ValYouW picture ValYouW  ·  8 Jan 2017
7

Interestingly, I found that this doesn't affect new packages:

# in a git repo with node_modules tracked (just to undo install)
yarn add left-pad
git add package.json yarn.lock
git commit -m save
git clean -df && git checkout -- node_modules/
# this will install left-pad
yarn install

My guess is that node_modules/.yarn-integrity is doing something here, such that if you update yarn.lock it'll notice things are out of sync. It does mean you can't be fidgeting around with node_modules, though.

joshma picture joshma  ·  19 Jan 2017
5

Same issue here.

I would expect that yarn installs packages missing from my node_modules folder when I use the yarn install command.

BjornRombaut picture BjornRombaut  ·  31 Jan 2017
18

Yep, this is definitely something to do with the integrity check. If you run yarn check it correctly notes the missing dependency. But when you simply yarn / yarn install, it assumes all is well. Delete yarn.integrity and it obviously rebuilds.

This is potentially problematic also if you switch a package from dependencies to devDependencies and attempt to re-yarn install. It claims everything is already up to date. You can yarn --force, but that ignores caches, which isn't quite what we need.

What we need is for yarn install to automatically do an integrity check. If the check fails, it should rebuild.

glebec picture glebec  ·  6 Mar 2017
0

Any updates on this? This is causing lots of wasted time here at Tableau.

mattyork picture mattyork  ·  6 Apr 2017
37

To add to this, if you rm -rf node_modules/*, then try to yarn install again, it says success Already up-to-date. If you run yarn check it notices all the missing modules, but another yarn install will still not install them. You have to actually delete the node_modules folder itself to reinstall.

chlab picture chlab  ·  10 Apr 2017
0

Thanks for the tip @chlab, that was driving me nuts.

omarkhan picture omarkhan  ·  17 Apr 2017
201
pribilinskiy picture pribilinskiy  ·  11 May 2017
3
raido picture raido  ·  15 May 2017
3

@pribilinskiy @raido I've found yarn --force or yarn install --force does the trick as well, what's the difference?

luchillo17 picture luchillo17  ·  30 Jun 2017
1

@luchillo17 the --force flag probably should be used as the last resort. It will also make requests to the server, ignoring the cache, all packages will be downloaded again.

pribilinskiy picture pribilinskiy  ·  11 Jul 2017
0

@pribilinskiy Indeed its a nuke solution.

luchillo17 picture luchillo17  ·  11 Jul 2017
38

Tried everything here, then realized I was in the wrong directory in my cli... 30 mins later 😭 😆

ricklove picture ricklove  ·  12 Jul 2017
7

yarn install --skip-integrity-check seems to actually do the trick

palfrey picture palfrey  ·  7 Nov 2017
25

I had what seemed like the same problem, but my issue ended up being the NODE_ENV var was accidentally set to production on my system and I could not install devDependencies. This is not a bug and is not specific to Yarn, but just in case this is your issue you can do yarn install --production=false

jasonmorita picture jasonmorita  ·  14 Nov 2017
1

Experienced the same when we pulled a devDependency from a remote repository.

We had to manually execute yarn add <package name> --dev to download it. Luckily, yarn.lock did not change at all after execution.

jhnferraris picture jhnferraris  ·  14 Feb 2018
0

i tried everything in here , but still facing this error . i mentioned script to start server and

`npm start

[email protected] start /home/vamshi/myApps/node/first
nodemon index.js

module.js:540
throw err;
^

Error: Cannot find module './node.js'
at Function.Module._resolveFilename (module.js:538:15)
at Function.Module._load (module.js:468:25)
at Module.require (module.js:587:17)
at require (internal/module.js:11:18)
at Object. (/home/vamshi/myApps/node/first/node_modules/nodemon/node_modules/debug/src/index.js:9:20)
at Module._compile (module.js:643:30)
at Object.Module._extensions..js (module.js:654:10)
at Module.load (module.js:556:32)
at tryModuleLoad (module.js:499:12)
at Function.Module._load (module.js:491:3)
npm ERR! code ELIFECYCLE
npm ERR! errno 1
npm ERR! [email protected] start: nodemon index.js
npm ERR! Exit status 1
npm ERR!
npm ERR! Failed at the [email protected] start script.
npm ERR! This is probably not a problem with npm. There is likely additional logging output above.

npm ERR! A complete log of this run can be found in:
npm ERR! /home/vamshi/.npm/_logs/2018-02-28T15_45_22_939Z-debug.log
and when i tried to install add dependencies . yarn install v1.3.2
[1/4] Resolving packages...
success Already up-to-date.
Done in 0.86s.
`

vamshi9666 picture vamshi9666  ·  28 Feb 2018
0

@vamshi9666, you're using npm, and this issue is about yarn. If you installed packages with yarn I guess you should use yarn run start.

This throws err; thing is a sure sign that you're running two different programs that pick up a different list of packages, so you need to fix that.

squadette picture squadette  ·  28 Feb 2018
0

even yarn start gives same error in log @squadette

vamshi9666 picture vamshi9666  ·  1 Mar 2018
1

I had this issue but turns out it was not wit yarn but with Webstorm.
I did yarn install several times and Webstorm kept reporting that packages were not installed.
Then I deleted node_modules folder altogether and did yarn install. And Webstorm didn't even show me node_modules in the project tree! But in reality it was present in my project folder.
Turns out it's a bug with Webstorm and to remedy it you have to File->Synchronise to make it see updated file structure.

avalanche1 picture avalanche1  ·  23 Apr 2018
0

Think of a multi developer environment where one developer installs a module and updates the source control with the updated yarn.lock file and/or offline cache but not the node_modules directory. Another developer updates and then runs yarn install, which should not report back that everything is up to date.

I don't think this scenario is actually broken. The reproduction steps you mentioned don't match this scenario. When you run yarn, it creates a "node_modules/.yarn-integrity" file listing out everything it installed in node_modules if that file isn't already present. If it is present, yarn compares that file (and not the rest of node_modules) against yarn.lock to see what needs to be changed.

So if one developer commits some changes to package.json and yarn.lock (because they run yarn add foo on their machine and committed the changes while node_modules was .gitignore-d.), and then a different developer pulls those changes and runs yarn, their yarn will correctly see that foo is in their yarn.lock but not in node_modules/.yarn-integrity, and it will add foo to node_modules.

Macil picture Macil  ·  28 Sep 2018
2

I think this happened to me in a docker container, when package versions have been upgraded. The container runs Alpine linux, which only has yarn 1.7.0 in its stable repository. Installing yarn 1.12.3 from the edge repositories fixed the problem, but here's what happens anyway in case other people are in the same boat:

  • package.json and yarn.lock updated on development machine with yarn upgrade
  • package.json and yarn.lock are updated in container, node_modules is not
  • Container basically does yarn check --integrity && yarn check --verify-tree || yarn install

    • yarn check --integrity fails

    • yarn install appears to work

  • Subsequent yarn check --integrity succeeds
  • Subsequent yarn check --verify-tree fails!

    • Packages in node_modules are still at their old versions

Here's the relevant output of the commands:

$ yarn check --integrity

success Folder in sync.
Done in 0.23s.

$ yarn check --verify-tree

success Folder in sync.
Done in 6.00s.

$ rsync -av --delete --exclude 'node_modules' /project/src/front-end/ /project/front-end

sending incremental file list
./
package.json
yarn.lock

sent 239,342 bytes  received 76 bytes  478,836.00 bytes/sec
total size is 425,735  speedup is 1.78

$ yarn check --integrity

warning Integrity check: Top level patterns don't match
error Integrity check failed
error Found 1 errors.

$ yarn check --verify-tree

error "@angular/animations" is wrong version: expected "^7.1.1", got "7.1.0"
...
error Found 16 errors.

$ yarn install --production=false

[1/4] Resolving packages...
[2/4] Fetching packages...
[3/4] Linking dependencies...
[4/4] Building fresh packages...
Done in 38.73s.

$ yarn check --integrity

success Folder in sync.
Done in 0.24s.

$ yarn check --verify-tree

error "@angular/animations" is wrong version: expected "^7.1.1", got "7.1.0"
...
error Found 22 errors.

But here's some information from yarn install --production=false --verbose:

verbose 3.462 Selecting "@angular/[email protected]" at level 0 as the peer dependency of "@angular/[email protected]".
...
verbose 4.8 Skipping copying of file "/home/project/.cache/yarn/v1/[email protected]/animations-7.1.1-8fecbd19417364946a9ea40c8fdf32462110232f/package.json" as the file at "/project/front-end/node_modules/@angular/animations/package.json" is the same size (1295) and mtime (499162500000).

Sure enough, it skipped a ton of stuff:

$ head -n 3 node_modules/@angular/animations/package.json

{
  "name": "@angular/animations",
  "version": "7.1.0",

The only thing that works without upgrading is blowing away the node_modules folder before running install, so I have to choose between that and installing a bunch of stuff from edge repositories :(

dakotahawkins picture dakotahawkins  ·  1 Dec 2018
2

I've run into this issue. I solve this problem with:

rm -rf node_modules yarn.lock
npm install
yarn import
yarn install --check-files

Looks like it worked!

kserjey picture kserjey  ·  20 Aug 2019
1

I had the same issue,
1st => remove existing node modules(Might not mandatory in some cases ),
2nd run 'yarn cache clean && yarn install'

thats it.

Sahan-srt picture Sahan-srt  ·  30 Sep 2019
1

Out of all comments, the only thing that helps in my case is: yarn install --production=false.

In all other cases, after deleting node_modules and running yarn install (or other suggestions), I only get .yarn-integrity file in node_modules.

EDIT: also, after each yarn add, I need to run yarn install --production=false as it deletes all devDependencies.

mfolnovic picture mfolnovic  ·  9 Oct 2019
1

EDIT: also, after each yarn add, I need to run yarn install --production=false as it deletes all devDependencies.

I also get this. yarn 1.17.3

kg-currenxie picture kg-currenxie  ·  4 Mar 2020
0

Out of all comments, the only thing that helps in my case is: yarn install --production=false.

In all other cases, after deleting node_modules and running yarn install (or other suggestions), I only get .yarn-integrity file in node_modules.

EDIT: also, after each yarn add, I need to run yarn install --production=false as it deletes all devDependencies.

This saved me, thank you. I am working with mixed project (java, kotlin, javascript and typescript) in a monorepo. I was thinking that the problem was on the frontend-maven-plugin, but after using your workaround it works perfectly.

For people that may stumble upon this issue: I am using workspaces and maven multimodules, maven coordinates build hierarchy, workspaces coordinates yarn dependencies (it looks complicated but it is actually very simple, I am surprised that there are no tutorials of it around the web, maybe it is just to niiche).

There is a section on the plugin called environmentVariables, mine looks like this:

<environmentVariables>
  <NODE_ENV>production</NODE_ENV>
  <BABEL_ENV>build</BABEL_ENV>
  <TS_NODE_PROJECT>\"./tsconfig.eslint.json\"</TS_NODE_PROJECT>
</environmentVariables>

Now, for some god knows why reason, if I remove this variables, the build works, but I need to prefix every script with cross-env NODE_ENV=production BABEL_E.... etc.

With --production=false flag it works perfectly, now I am guessing that the problem lies in the interaction that yarn has with NODE_ENV.

That concludes my random rambling. I hope this turns out to be useful for someone. Thank you.

EDIT: https://classic.yarnpkg.com/en/docs/cli/install/#toc-yarn-install-production-true-false yep. --production=true or NODE_ENV=production makes yarn ignores devDependencies.

LouizFC picture LouizFC  ·  24 Jul 2020