What are lock files?

A lock file describes the entire dependency tree as it is resolved when created including nested dependencies with specific versions. In npm these are called package-lock.json and in yarn they are called yarn.lock. In both npm and yarn they are placed alongside the package.json file.

A package-lock.json looks similar to this:

{
  "name": "lockfile-demo",
  "version": "1.0.0",
  "lockfileVersion": 1,
  "requires": true,
  "dependencies": {
    "ansi-styles": {
      "version": "3.2.1",
      "resolved": "https://registry.npmjs.org/ansi-styles/-/ansi-styles-3.2.1.tgz",
      "integrity": "sha512-VT0ZI6kZRdTh8YyJw3SMbYm/u+NqfsAxEpWO0Pf9sq8/e94WxxOpPKx9FR1FlyCtOVDNOQ+8ntlqFxiRc+r5qA==",
      "requires": {
        "color-convert": "^1.9.0"
      }
    },
    "chalk": {
      "version": "2.4.2",
      "resolved": "https://registry.npmjs.org/chalk/-/chalk-2.4.2.tgz",
      "integrity": "sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ==",
      "requires": {
        "ansi-styles": "^3.2.1",
        "escape-string-regexp": "^1.0.5",
        "supports-color": "^5.3.0"
      }
    }
  }
}

A yarn.lock is formatted differently but contains similar information:

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


ansi-styles@^3.2.1:
  version "3.2.1"
  resolved "https://registry.yarnpkg.com/ansi-styles/-/ansi-styles-3.2.1.tgz#41fbb20243e50b12be0f04b8dedbf07520ce841d"
  integrity sha512-VT0ZI6kZRdTh8YyJw3SMbYm/u+NqfsAxEpWO0Pf9sq8/e94WxxOpPKx9FR1FlyCtOVDNOQ+8ntlqFxiRc+r5qA==
  dependencies:
        color-convert "^1.9.0"

chalk@^2.4.2:
  version "2.4.2"
  resolved "https://registry.yarnpkg.com/chalk/-/chalk-2.4.2.tgz#cd42541677a54333cf541a49108c1432b44c9424"
  integrity sha512-Mti+f9lpJNcwF4tWV8/OrTTtF1gZi+f8FqlyAdouralcFWFQWF2+NgCHShjkCb+IFBLq9buZwE1xckQU4peSuQ==
  dependencies:
        ansi-styles "^3.2.1"
        escape-string-regexp "^1.0.5"
        supports-color "^5.3.0"

Both of these contain some important pieces of information:

  • The actual version of every dependency installed
  • The dependencies of every dependency
  • The resolved package including a checksum to verify the integrity of the package

Why do we need lock files?

Lock files are intended to pin down (i.e., lock) all versions for the entire dependency tree at the time that the lock file is created. Why is it important to use a package lock file and lock package versions?

Without a package lock file, a package manager such as Yarn or npm will resolve the most current version of a package in real-time during the dependencies install of a package, rather than the version that was originally intended for the specific package.

For example, if a project depends on package dummy-pkg: ^1.0.0 then two separate installs executed at different times could retrieve different versions of dummy-pkg. This can happen if a user installs dummy-pkg, which retrieved version 1.0.0 and then that package releases a new version 1.0.1 several minutes later. Thereafter, a second user running an install in the project would end up retrieving dummy-pkg version 1.0.1 instead of version 1.0.0.

Using lock files ensures that installations remain identical and reproducible throughout their entire dependency tree, across users, such as team members working together, and across systems, such as when running a CI build.

Why is this better than simply locking down all your dependencies in the package.json file?

For example, just specify:

"express": "4.14.0"

In this example, npm will install express package, version 4.14.0. But the thing that you won’t think about is “what if express published new version with the same version number?” or “what if express didn’t lock their package versions?”. These are the questions you can’t know an answer for.

What about npm ci?

npm ci bypasses a package’s package.json to install modules from a package’s lockfile. This ensures reproducible builds—you are getting exactly what you expect on every install.

This command is similar to npm install, except it’s meant to be used in automated environments such as test platforms, continuous integration, and deployment – or any situation where you want to make sure you’re doing a clean install of your dependencies.

In short, the main differences between using npm install and npm ci are:

  • The project must have an existing package-lock.json or npm-shrinkwrap.json.
  • If dependencies in the package lock do not match those in package.json, npm ci will exit with an error, instead of updating the package lock.
  • npm ci can only install entire projects at a time: individual dependencies cannot be added with this command.
  • If a node_modules is already present, it will be automatically removed before npm ci begins its install.
  • It will never write to package.json or any of the package-locks: installs are essentially frozen.
npm ci # will install exactly what's in the package-lock.json
yarn install --frozen-lock-file # will install exactly what's in yarn.lock without updating it

References:

What about transitive dependencies?

Both yarn and npm will never take into account lock files for transient dependencies; these are completely ignored by package managers. Only the top-level project, where an install action is performed, is looked up for its entire dependency tree through a lock file to which the package manager refers as the dependencies manifest.

References

What about publishing your package?

Contrary to what some people believe, the content that you publish to npm is not always the same as what’s on GitHub or overall in your project. The way that a module is published is that npm will determine the files that should be published by checking for a files key in your package.json and a .npmignore file or if none is present the .gitignore file. There are also some files that are always included and some that will always be excluded. You can find the entire list of those files on the npm page. For example the .git folder will always be ignored.

Afterwards npm will take the list of files and will package them all up together into a tarball using npm pack. If you want to check out what files are packaged you can run in a project npm pack --dry-run and you’ll see the output with all of the files.

That tarball will then be uploaded to the npm registry. One thing you might notice when you run this command is that if you already have a package-lock.json it is actually not being bundled. This is because package-lock.json will always be ignored as specified by the list in the npm docs (and mentioned above).

Subsequently this means that if another developer installs your published package (rather than say a developer cloning your git repo), they’ll never download your package-lock.json and therefore it will be completely ignored during the installation.

This might cause the “works on my machine” effect by accident since your CI and developer environment might pick up a different version of dependencies than your users.

References

Shrinkwrap lock files

There is one case in which a special lock file exists that is taken into account even for transient dependencies. The npm-shrinkwrap.json file pins down the dependency tree like the other lock files do, but an npm publish process will also commit this file to the registry. More importantly, when end users consume the library in a typical application and run an npm install – the library’s shrinkwrap file will decide on which versions to pull instead of the semver resolution, which happens during installation.

What is a shrinkwrap file good for? Well, it allows library maintainers to pin down and curate their own library dependencies in order to ship known versions. It does, however, require careful maintenance for the entire dependency tree. Also, if a shrinkwrap file is used, there is no need for any other lock file to exist in the source control. A good example of a library that takes this approach is the well-known hapijs project.

The files have exactly the same content, but there are a handful of differences in how npm handles them, most of which are noted on the docs pages for package-lock.json and npm-shrinkwrap.json:

  • package-lock.json is never published to npm, whereas npm-shrinkwrap is by default
  • package-lock.json files that are not in the top-level package are ignored, but shrinkwrap files belonging to dependencies are respected
  • npm-shrinkwrap.json is backwards-compatible with npm versions 2, 3, and 4, whereas package-lock.json is only recognized by npm 5+

You can convert an existing package-lock.json to an npm-shrinkwrap.json by running npm shrinkwrap.

If you are publishing your package to npm, you have a choice between:

  1. using a package-lock.json to record exactly which versions of dependencies you installed, but allowing people installing your package to use any version of the dependencies that is compatible with the version ranges dictated by your package.json, or

  2. using an npm-shrinkwrap.json to guarantee that everyone who installs your package gets exactly the same version of all dependencies

The official view described in the docs is that option 1 should be used for libraries (presumably in order to reduce the amount of package duplication caused when lots of a package’s dependencies all depend on slightly different versions of the same secondary dependency), but that option 2 might be reasonable for executables that are going to be installed globally.

Issues with shrinkwrap files

One issue is that the shrinkwrap.json file, which contains sensitive information about your dependencies, may be included when publishing a package. Another is that Shrinkwrap makes it more difficult to add new dependencies. There is also a security risk inherent in shrinkwrap, which is vulnerable to remote code execution attack if HTTPS URLs are not used.

By using a shrinkwrap file you’ll pin the exact version which can be great but it can also block people from getting critical patch fixes that would otherwise be installed automatically. npm strongly discourages the use case of shrinkwrap for libraries and recommends it more for CLIs or similar.

Open Question: I wonder what the consequences are with using shrinkwrap files? For example, how often are they actually used in practice? How much dependency bloat is caused by using them? Would findings related to “technical lag” and “pinning dependencies” be similar?

References

So which should I use? package-lock.json or npm-shrinkwrap.json?

When working on a library

If you’re working on a library (as in a package onto which others will depend on), you should use package-lock.json. An alternative is to use shrinkwrap but make sure it never gets published with the package (the new lock file is never published automatically). Why not publish the shrinkwrap? It’s because NPM respects shrinkwraps it finds within packages and since a shrinkwrap always points to a specific version of individual packages, you would not take advantage of the fact that NPM can use the same package to satisfy requirements from multiple packages if the semver range allows it. In other words, by not forcing NPM to install specific versions, you allow NPM to better reuse packages across the dependency tree and make the result smaller and faster to assemble.

There’s one caveat to this however. When you’re working on your library, you get the exact same dependencies every time because either package-lock.json or npm-shrinkwrap.json is present in the repository. The same goes for your continuous integration server where you check out the same code. Now imagine your package.json specifies a dependency on some package as ^1.0.0 and that also happens to be the version specified in the lock file and installed every time. Everything works. Now what happens if a new version of the dependency is published, accidentally breaks semver and your package breaks because of it?

Unfortunately, you might not be able to notice that until a bug report comes in. Without any lock files in the repository, your build would fail at least on the CI because it would always install the latest versions of the dependencies and thus run the tests with the new broken version (provided that the build is run periodically, not just for PRs). With the lock in place however, it will always install the working locked version.

There’s a couple of solutions to this problem however. First, you could sacrifice the exact reproducibility and not add the lock file to your version control system. Second, you could make a separate build configuration which would run npm update prior running the tests. Third, you simply delete the lock before running the tests in the special build. How to actually deal with the broken dependency once discovered is another topic on its own mainly because semver as implemented by NPM doesn’t have a concept of allowing a wide range but also blacklisting specific versions.

This of course begs the question whether it’s actually worth it to add the lock file into the version control when working on libraries. A thing to keep in mind however is that the lock file contains not only dependencies but also dev dependencies. In that sense working on a library is similar to working on an application (see the next section) and having the exact same dev dependencies over time and across multiple machines is an advantage.

When working on an application

Alright, what about packages used by the end users in the terminal or bundled executables in general? In this case, the package is the final result, the application, and you want to make sure that the end users always get the exact dependencies you had while publishing it. This is where you want to use shrinkwrap and make sure to also publish it with the package so that it’s respected by NPM during install. Remember, you can always see what the package would look like if published using npm pack.

Note that pointing to a specific version of a dependency in package.json is not good enough because you want to make sure the end users get the exact same dependency tree including all of its sub-dependencies. A specific version in package.json guarantees the version only at the top level.

What about other types of applications, for example projects you start from within their repository? It doesn’t really matter that much in this case. All that matters is that the correct dependencies are installed and both of the locks can satisfy that. Your choice.

Other References