Foursquare Intersections logo
Explore
Subscribe
Engineering

Introducing Pants: a build system for large-scale codebases like Foursquare's.

Written by 4SQ ENG on Sep 16, 2014 - Read time: 2 min - Read Later

Foursquare and Swarm are written predominantly in Scala on the server side. But as we've grown, so have the size, complexity and diversity of our codebase:

  • We currently have around 700,000 lines of handwritten code in 6500 .scala files.
  • We generate about 1.9 million lines of Scala code from 1400 .thrift files using Spindle, our homegrown data model code generator.
  • We generate UI code from Closure Templates.
  • We compile CSS using Less.
  • We have a significant amount of Python code for running our deploy toolchain, data processing pipelines and other offline systems.
  • Like all large codebases, we also have little bits of other things here and there: some C, some Java, some Ruby, some Bash scripts, and so on.

Naturally there is a complex web of dependencies between different parts of the codebase. In fact our code dependency graph has about 2500 vertices and tens of thousands of edges.

We needed a build toolchain that would work well with this complexity, and the result is Pants, an open source build system we developed together with engineers from Twitter, Square and elsewhere.

Pants was designed to support large-scale, heterogeneous codebases. It uses fine-grained dependency management to build only what you need, keeping build times from getting unnecessarily long (a must when using Scala, with its slow compilation speeds). Pants also makes it straightforward to plug in custom build tools, such as code generators, and it supports every part of the build lifecycle: codegen, external dependency resolution, compilation, linting, test running, bundling deployable artifacts and more.

You can read more about Pants here, including the etymology of the name. If your codebase is growing beyond your toolchain's ability to scale, you might want to give Pants a try. And of course we're always looking for contributors to the project!

Subscribe

Follow 4SQ ENG

Introducing Pants: a build system for large-scale codebases like Foursquare's.

Read Later

Pardot response heading