The simple magic of package manifests and lockfiles

If you aren’t using open source components to build your apps, you’re not living in 2019. Our research suggests 92% of professional applications are built using open source. But what’s the modern best practice for pulling in and using these libraries?

In the past, the best that we could do was to actually pull the source code for those libraries into the source tree of our applications. This practice, commonly referred to as either vendoring or bundling, has fallen out of favor for a variety of reasons: 

• For one, it becomes very difficult to know exactly what software you’re using and what versions are present. 
• Also, engineers tend to see the code in the source tree and start making changes—which makes it extremely difficult to move to a newer version. 
• Finally, in this world, updating libraries was very difficult as you had to manually find and download the new version and include it in your source tree.

Luckily, we now have a better answer for almost all common programming languages and ecosystems. Simply use a manifest file to describe all of your direct dependencies, then generate and store a “lockfile” describing all of your direct and transitive dependencies, including the specific versions of each. 

The term lockfile was coined to describe how to “lock” the specific version of every package that you depend on. 🔒 Once you’ve written the manifest file and generated a lockfile, the best practice is to store them in your source control repository. This allows you to share them with your teammates and your continuous integration environment, as well as deploy to production.

How this works varies from package manager to package manager, but the general concept is the same. I’ll walk through an example here in JavaScript using the popular combo of npm and yarn. 

Let’s say that you’ve written a simple node.js server using the Express framework to fetch some information from Libraries.io. You are using Express as your web framework, but you are also making calls to the Libraries.io API using librarian-api. (which you know because you directly went through a process of choosing them). 

Next, since you’re using npm, create a simple package.json for your web app. Clearly, since you wrote it, you want to commit it and add it to your version control history so that you can update it over time. Those paying attention will see that you’ve specified a version of Express but are leaving the version of librarian-api far less specified.

Now you can install the packages that your app depends on using yarn. Running yarn install will install your packages, but it also generates a file yarn.lock (which is the lock file used by yarn). Looking at  that, you can see that, wow, it lists a lot more packages—almost two hundred in fact! 

This is because all of the packages you’re using each have their own things they depend on which depend on other things and so on. If you add and commit this to source control, now when you run ‘yarn install’, you’ll get the exact same versions of Express, librarian-api, and all of its dependencies installed for your application. 

So by using a lockfile, you are able to avoid unintended changes creeping into your application and know that the version of all of the open source code you’re using locally, in your continuous integration tests, and the version that is deployed are the same and thus will work the same. If not using a lockfile, then you could end up with a new version being deployed to your production systems that hasn’t been through your continuous integration tests—introducing new bugs or problems.

An added bonus of having a lockfile in addition to a manifest file like package.json is that static analysis tools can understand all of the open source software you use to make recommendations. This means that the Tidelift Subscription recommendations about which versions of each dependency you should use become far more actionable. You’ll know that you don’t have unintended security vulnerabilities present and that you’re on a version which is still being actively maintained by the creator of the software.

Package manifests and lock files aren’t just for JavaScript either. They’re also commonly supported by bundler (Rubygems), pip (python), packagist (php), and more!

Package manifests and lockfiles are all you need to get started using the Tidelift Subscription, too. We never look at any of your code—we just scan your lock files to see what projects you're using. Interested in learning more about your app's code health? You can start a 14-day free trial today.