JavaScript, NPM, and the rise of granular software

Keenan Szulik
by Keenan Szulik
on January 4, 2018

In the last month, we’ve alluded to the relationship between package managers and small, modular packages on a couple of occasions, without explicitly diving in to what that relationship looks like.

Package managers have a significant impact on the way that software is produced and consumed.  JavaScript, and its most prevalent package manager NPM, are an illustrative case.  (In future posts, I’ll examine how other package managers influence their respective software communities.)

Is JavaScript is eating the world?

There are many possible ways to measure the popularity of a programming language—and it’s easy to find countless opinions on the most popular languages with a quick search.  But by most metrics, JavaScript consistently ranks among the most popular languages.  For example, according to Redmonk’s language rankings from June 2017, JavaScript is the most popular language in the world; GitHub, too, shows that there are more JavaScript repos than in any other language.

Looking at rankings, however, fails to convey just how many packages make up the JavaScript community, and its huge scale—by package count—compared to other languages.

Using the libraries.io open data release, I looked at the most commonly used languages, as well as the most popular package managers, and the result is clear: JavaScript and NPM are huge ecosystems, operating with two to three times the number of packages of any other open source community.

javascript_NPM_size

Aside from their general popularity, however, there must be an explanation for why JavaScript and NPM are both so much larger than all other ecosystems—at least by total package count.

The answer is simple: the granularity of software projects in the JavaScript and NPM community.

JavaScript embodies the shift towards modular software, and NPM serves as the primary distribution mechanism for those small, lightweight packages.

Modularity and the NPM ecosystem

I’ve written before about the trend towards granular software packaging and what that means for developing and shipping software: smaller packages are designed for simpler tasks, and tend to be updated more frequently.  But does this paradigm hold true in the NPM ecosystem?

The answer to that is an emphatic yes.  The median GitHub repository size for a JavaScript package is just 126 KB (and 108 KB across all packages in NPM).  This is smaller than the median size for most other major languages (such as Python, Ruby, Java, and C#), with PHP being the only language ecosystem with more than 25,000 packages to have a smaller median size.

Similarly, JavaScript and NPM also see more updates than other language communities, with the median NPM package having three unique versions, and the mean package having 6.9 versions.

However, there are downsides to this modular world: it’s more difficult to wrangle and distribute the proper packages, and the reality is that there is a mostly forgotten and unused tail of open source.

The long tail of JavaScript

It’s generally accepted that usage of open source has a long tail, meaning that a small number of packages account for most usage, and most packages receive little activity (following the Pareto principle).

The JavaScript ecosystem is an extreme example of the long tail phenomenon: only 28.6% of NPM packages have at least one dependent repository (or, repository relying on its code), meaning that over 450,000 NPM packages are unused by other NPM packages.  

This doesn’t necessarily mean that these packages are ignored by end users—many of whom host their code privately, where usage isn’t publicly visible—but it does make managing and distributing all this software more cumbersome.  

Package managers such as NPM exist to try to alleviate that difficulty, but they primarily help with the distribution and installation process.  Once a user is running tens or hundreds or thousands of such granular packages, a new challenge is born: keeping track of rapid changes and updates to all of your open source dependencies.

In the coming weeks, I’ll explore the extent to which this wave of granular packaging is increasingly prevalent in software communities beyond JavaScript and NPM, but, for now, if you are interested in learning more, consider signing up for updates or following us on Twitter.

Lifter guide