Month: September 2018

I recently came across a neat article in LinkedIn’s blog. It was written by their Site Reliability Engineering group (SRE) which covered some interesting Command Line Interface (i.e., CLI) utility libraries related to Python.

I can see how the power of Python would be fantastic for CLI utilities. The problem is Python’s dependency management. The dependency issue is even more compounded for CLI utilities since a devops engineer would need to be able to bring over a utility to any server (assuming a consistent version of the operating system) without worrying about the utility’s dependencies. Such an issue would prevent a system admin from doing the needed work (whether emergency or planned). In SRE’s article they explain this eloquently:

At LinkedIn, we ship hundreds of command-line utilities to every machine in our data centers and to all of our employees’ workstations. The vast majority of these utilities are written in Python.

In addition to developing these command-line utilities, we have hundreds of supporting libraries that are constantly being iterated on, with new versions published daily. Because of the inherent problems present when dealing with such a huge and ever-changing dependency graph, we need to package the executables individually to avoid dependency conflicts. Initially, we took advantage of the great open source tool PEX. PEX elegantly solved the isolated packaging requirement we had by including all of a tool’s dependencies inside a single binary file that we could then distribute.

PEX was not sufficient for SRE’s usage, so they created their own dependency packager called shiv. It’s an interesting name choice. I initially thought it was a prison-made knife, but apparently the name was based on the word “shiver” although the documentation has a knife emoji. In their words:

Why shiv?

>The tool freezes a Python environment, so you can think of shiv as a shorter way of saying “shiver.”

PEX and Shiv are interesting utilities that bring the power of Python to the command line through one executable utility.

Thoughts? Feedback? Let me know: @eli4d on Twitter