The eli4d Gazette – Issue 060: LinkedIn’s Site Reliability Engineering Group’s approach to Python CLI based Utilities through PEX and Shiv

I recently came across a neat article in LinkedIn’s blog. It was written by their Site Reliability Engineering group (SRE) which covered some interesting Command Line Interface (i.e., CLI) utility libraries related to Python.

I can see how the power of Python would be fantastic for CLI utilities. The problem is Python’s dependency management. The dependency issue is even more compounded for CLI utilities since a devops engineer would need to be able to bring over a utility to any server (assuming a consistent version of the operating system) without worrying about the utility’s dependencies. Such an issue would prevent a system admin from doing the needed work (whether emergency or planned). In SRE’s article they explain this eloquently:

At LinkedIn, we ship hundreds of command-line utilities to every machine in our data centers and to all of our employees’ workstations. The vast majority of these utilities are written in Python.

In addition to developing these command-line utilities, we have hundreds of supporting libraries that are constantly being iterated on, with new versions published daily. Because of the inherent problems present when dealing with such a huge and ever-changing dependency graph, we need to package the executables individually to avoid dependency conflicts. Initially, we took advantage of the great open source tool PEX. PEX elegantly solved the isolated packaging requirement we had by including all of a tool’s dependencies inside a single binary file that we could then distribute.

PEX was not sufficient for SRE’s usage, so they created their own dependency packager called shiv. It’s an interesting name choice. I initially thought it was a prison-made knife, but apparently the name was based on the word “shiver” although the documentation has a knife emoji. In their words:

Why shiv?

>The tool freezes a Python environment, so you can think of shiv as a shorter way of saying “shiver.”

PEX and Shiv are interesting utilities that bring the power of Python to the command line through one executable utility.


Thoughts? Feedback? Let me know: @eli4d on Twitter


The eli4d Gazette – Issue 056: Python’s former BDFL and the Apollo 11 Moon Landing

Python’s former BDFL

Human languages take millennia to develop, whereas programming languages can take less than a month for initial design. Human languages are evolutionary, and programming languages take on the preferences, principles, and beliefs of their designer.

If the programming language is successful in terms of adoption and usage, then the language moves from its (initial) one person design to evolution that is based on the opinions and preferences of many people in conjunction with the emergence of current trends and fads (including competing languages).

Many times, the language designer is throned as its BDFL – benevolent dictator for life. He or she gets to have final say on the evolution of the language…until they have enough.

Python was created by Guido Van Rossum in 1991. It’s been evolving for 27 years with Guido as its BDFL until last week.

Twenty-seven years is a very long time for a programming language to both survive and thrive. It’s also a long time to wrestle and dispute the various proposals for Python’s evolution.

Apollo 11 Old Style Re-broadcast

On a lighter note, I came across this charming page about the Apollo 11 moon landing (mentioned in a recent Studio Neat issue). While the date/time re-broadcast is over, you can see the Apollo 11 landing through some links to YouTube.


Thoughts? Feedback? Let me know: @eli4d on Twitter


The eli4d Gazette – Issue 054: Dependency and Package Managers in Python and PHP

Python Environment and Package Installation

I saw a recent XKCD comic that describes the current state of setting up a Python development environment:

Python Environment

The excellent Explain XKCD site covers the point of the cartoon. The issue is the (ongoing) difficult transition from Python 2.x to Python 3.x and the reality that there is no defacto package installation tool. The messiness of Python’s development environment deeply contrasts with the cleanliness and beauty of the actual language.

One interesting and significant effort in approaching the above issue can be found through the PyPi package manager. A great FLOSS Weekly episode had an interview with one of the maintainers of the new PyPi covering its pros and cons.

PHP’s Composer and Package System

In the PHP world – Packgist site is the definite destination for third-party PHP packages. In conjunction with these packages, Composer is the defacto package/dependency management tool. It’s more than a package manager because it focuses on managing packages on a per project basis.

At its core Composer focuses on dependency management and the secondary result is that it also does package management. One problem that arises is the usage of Composer as a pure package installer across a system rather than just a project. As Composer’s documentation states:

By default it does not install anything globally. Thus, it is a dependency manager. however support a “global” project for convenience via the global command.

Composer’s “global” command installs packages in a common central project and this, in turn, can cause major headaches with projects that have packages installed in different locations.

Pantheon’s blog describes this issue (and its solution) really well:

One of the most commonly documented ways for a PHP command line tool to be installed is via the composer global require command. This command is easy to document and easy to run, which explains its popularity. Unfortunately, this convenient function has a darker side that can cause some pretty big problems. The root of the problem is that Composer, by design, manages dependencies on a per-project basis; however, the global command installs everything into a common central project. The upshot of this is that two distinct projects that were never intended to be combined must suddenly share dependencies. In this configuration, it is all-too-common to encounter dependency conflicts that never would have been observed had these applications been installed independently.

The current solution to this is a Pantheon created tool called Cgr. The blog article provides details on its usage and when to use Composer global.

Note: many thanks to the folks on the Poststatus Slack for discussing Composer’s issues and pointing to Cgr.

The eli4d Gazette – Issue 052: The Ship of Theseus & Building cross-platform apps in Python

The Ship of Theseus and Story Loops

Imaginary Worlds is one of my favorite podcasts. The most recent episode was amazing in how it related the question of identity through the Ship of Theseus, Westworld, and Star Trek.

There is much that I could say about it, but I think that it would be best to listen to it. So let me leave you with a quote from the podcast that was great:

It (i.e. Westworld) taps into the fact that we’re all on story loops. Some of these story loops are created by us, sometimes they’re created by society, but either way, after a while you just tend to go through the motions (i.e. just like the hosts on Westworld). And every so often it’s good to stop and reflect if this is the version of you that you want to be. It’s easy – all you have to do is ‘freeze all motor functions’.

Building cross-platform apps in Python

I’m not sure if cross platform development is like a unicorn in the software field. Whether it was Java’s “write once, run anywhere” promise or Electron’s pitch of “build cross platform desktop apps with JavaScript, HTML, and CSS”.

I came across an interesting project whose goal is to provide cross-platform development for Python. The BeeWare Project “aims to take the power of Python as a language, and use it to enable users of all skill levels to develop applications with native user interfaces.” This may be an interesting project for those focusing on learning Python.


Thoughts? Feedback? Let me know: @eli4d on Twitter


The eli4d Gazette – Issue 48: Learning Django 2 and the 2018 Stack Overflow Developer Survey

If you’re interested in learning Django 2

As a teacher and a learner, I’m on the lookout for great folks that create courses. If you’re interested in learning the Django framework (this may especially be of interest to my former Python students) then you might be interested in this kickstarter campaign by Tracy Osborn. Tracy approaches programming from a web designer point-of-view, and this is quite valuable from a “beginner’s mind” approach.

Check out the 2018 Stack Overflow Developer Survey

Stack Overflow is an amazing question and answer site that covers a tremendous amount of software development issues. It started originally as a response to Experts Exchange and has far exceeded its competitor’s breadth, depth, and utility.

Stack Overflow’s yearly development surveys have become an industry staple for software development trends. It’s an in-depth survey that has become better and better over the years. The 2018 survey is quite deep and extensive and is a well worth read.

If you’re looking for a concise summary of the 2018 survey, then check out Quincy Larson‘s summary.


Thoughts? Feedback? Let me know: @eli4d on Twitter


The eli4d Gazette – Issue 032

Tech Pick

I’ve had Python on my brain lately due to heavy duty preparation for my upcoming online Python course. So coming across Instagram’s Python technology stack was interesting. The article begins with:

Each day, over 95 million photos and videos are shared on Instagram. The unstoppable photo-centric social media platform has over 600 million registered users — 400 million of whom are active every day. Talk about operating at scale: Instagram kills it at levels most companies can barely even dream about.

Even more impressive, though, is the fact that Instagram serves this incredible amount of traffic, reliably and steadily so, by running Python (with a little help from Django) under the hood. Yes, that Python — the easy to learn, jack-of-all-trades general purpose programming language. The one everybody in the industry dismisses as, “Yeah, Python is great in so many ways, too bad it’s not really scalable.”

I thought that this was a great example to mention to my students – “well if Instagram uses Python at their scale, Python must be a good thing to learn.” And yet this didn’t sit right with me, and I remembered reading an excellent article that clarified my unease: “You Are Not Google”.

Python is a great language for many reasons (easy to learn, lots of built in libraries, great scientific/numeric support (SciPy, Pandas, iPython, NumPy), etc..) but Instagram’s use of it is not one of them. I’m not Instagram, and it’s very likely the case that you aren’t either.

Media Pick

I’ve had “American Gods”aal on my Kindle for what seems like forever. It was recommended by a friend back in 2004 and I got the Kindle version on sale a few months ago (via Bookbub).

Where to begin? The story is like a fractal of a pick-up stick game. You think “really Neal – a pick-up stick game?” And there’s Gaiman laughing manically. So you play the game and read the book 1 bit at a time…and just when you think you understand, repeating themes and whispers of previous chapters slap you across the face.

My Kindle highlights are filled with too many sentences from this book, and I can hardly pick a favorite. Here’s an example:

“The house smelled musty and damp, and a little sweet, as if it were haunted by the ghosts of long-dead cookies.”

aal = Amazon affiliate link

Random

I’m a big OmniFocus, but recent (XML) corruptions have been worrisome (on the positive side OmniFocus support is top notch). One database corruption happened in March, and another just happened about a week ago. My favorite data format is plain text. But I haven’t come across a plain text task management system that implements GTD and spans mobile and Mac OS X as seamlessly as OmniFocus.


Thoughts? Feedback? Let me know: @eli4d on Twitter