Tag: Python

The eli4d Gazette – Issue 073: Python-Based Scientific Computing the Browser through Pyodide and The State of UX Survey

Firefox Pyodide – Scientific Computing in the Browser?

The Mozilla Hacks Blog had a really neat post about an experimental project involving a full Python-based data science stack on the browser.

WebAssembly is the technology that makes this possible. Per the WebAssembly link:

WebAssembly is a new type of code that can be run in modern web browsers — it is a low-level assembly-like language with a compact binary format that runs with near-native performance and provides languages such as C/C++ and Rust with a compilation target so that they can run on the web. It is also designed to run alongside JavaScript, allowing both to work together.

Pyodide is a really neat experiment that combines the power of WebAssembly with some amazing scientific libraries in Python (i.e. NumPy, Pandas, Matplotlib, parts of SciPy, and NetworkX).

If browser-based web applications can run at native speeds, then how necessary are native apps? (and especially mobile native apps)

The State of UX Survey

I came across an interesting UX survey – The State of UX in 2019. It’s an interesting view into the design side of things. I don’t know how accurate or true it is but it is definitely visually compelling (as you would expect for this kind of report).

You can find the newsletter archive at https://eli4d.com/category/newsletter/ and the blog at eli4d.com.

Thoughts? Feedback? Let me know: @eli4d on Twitter

The eli4d Gazette – Issue 069: Results from the 2018 Python Developers Survey and REST API Testing Resources

Interesting results from a 2018 Python Developers Survey

JetBrains, the makers of many IDEs teamed up with Python Software Foundation to run developer survey for 2018 (this is similar to the State of JavaScript survey for JS developers).

The 2018 survey can be found here: https://www.jetbrains.com/research/python-developers-survey-2018/ (There’s also a 2017 survey too)

Some interesting survey points:

  • Python 3 adoption is inching closer to 100 percent. Considering that Python 3 was released in 2008, the adoption curve has been very very slow.
  • The key take aways is a nice summary of the current state of Python development. It’s interesting to note that data science has overtaken web development among Python users. This is not completely surprising considering the availability of Jupyter notebook in conjunction with excellent tooling available through data science frameworks and libraries like NumPy, Pandas, Matplotlib, and SciPy.

Neat REST API Testing Resources

I came across these interesting REST testing resources:

  • https://reqres.in/: a hosted REST API that is ready to go
    • advantages: no setup needed
    • disadvantages: no guarantees of where your data resides (privacy, etc…)

  • JSON Server: an easy way to locally deploy a fake REST server
    • advantages: you can put whatever data you want since it’s on your machine (so no worries about privacy)
    • disadvantages: need to set it up (though the setup seems very easy using Node.js)

You can find the newsletter archive at https://eli4d.com/category/newsletter/ and the blog at eli4d.com.

Thoughts? Feedback? Let me know: @eli4d on Twitter

The eli4d Gazette – Issue 063: A Thanksgiving Feast of Developer Flash Cards, Black Friday Deals, and the YNAB Book

Flash Cards for Developers

Spaced repetition via flashcards is a great way to learn new information. I came across flashcardsfordevelopers.com while listening to a recent Syntax.fm podcast (one of my current favorites when it comes to tech podcasts).

There are tons of different flash card collections including:

It’s a neat resource for both learning and review.

Winter is Coming…wait no…Black Friday is coming

I’ve been lax in past years about Black Friday deals. However, this year, I’m going to try to be more proactive about having a list of items for Black Friday shopping. I’ve found the Wirecutter to be a great review site that helps me answer the “what are two best choices for xyz product?”.

The Wirecutter has an excellent article on how to prepare for all the Black Friday and Cyber Monday deals. I especially like the emergency preparation section.

Just Finished Reading

I just finished the “You Need A Budget” book. I had purchased it from Audible during a sale before Audible decided to send deals to only Audible subscribers (makes sense I suppose…but still annoying). This book is excellent. Jesse Mecham has the perfect voice for audio (this is extremely unusual for an author) and he comes across in a frank and personable way. While I haven’t yet tried the YNAB approach to budgeting, I certainly intend to give it a whirl. An important point about this book is it does not push the YNAB service/product. The only place any information shows up is in the appendix. Base on the title it’s easy to assume that the book exists to market the service (since so many other books do exactly that), but Jesse is sincere in conveying his belief that a budget is a way to freedom rather than restriction.

One other thing I need to mention is chapter 9. Chapter 9 describes his approach to giving allowance to his kids. It’s a very concrete action based approach based on the book “The Opposite of Spoiled: Raising Kids Who Are Grounded, Generous, and Smart About Money”. I’m definitely going to explore this approach.

I highly recommend this book in audio format!

Thoughts? Feedback? Let me know: @eli4d on Twitter

The eli4d Gazette – Issue 062: Fun Data Sets and Falling for Fake News

Interesting Data Sets

With the upcoming elections, there are lots of articles spouting lots of data. These are some interesting data sets that can be used with something like Python’s Pandas library and various visualization libraries like Plotly and D3.

Falling for Fake News

I came across a really interesting Washington Post article about fake news. The article talks about our human tendency to fall for fake news. While the article traces the origin of a fake video on Facebook, the larger point is how easily we all can fall for fake news/photos/videos. The problem is that we are not very good at telling fake media from real media, while the technology to produce fake information has significantly increased in sophistication and ease of use. This particular paragraph from the article is quite telling:

Even after decades of Photoshop and CG films, most of us are still not very good about challenging the authenticity of images — or telling the real from the fake. That includes me: In an online test made by software maker Autodesk called Fake or Foto, I correctly identified the authenticity of just 22 percent of their images. (You can test yourself here.)

I took the Autodesk fake photo test, and I also got 22 percent of the images correct. I’m stunned by this – I got one out of every four pictures correctly. How does this apply to read articles and posts on the web (not just Facebook but all other sources of information on the web)?

I’ve been struggling to figure out how to deal with this. With the upcoming U.S. midterm elections on November 6th, I wonder about this even more. The only thing I can come up with is the Russian proverb “Trust but verify”. If I want to trust a particularly new story, then I am obligated to verify its authenticity by checking other news sources such as actual newspaper sites with differing viewpoints (right, left, and middle). The obligation is on me to put the effort and investigate a news item that on first glance seems super-compelling and utterly true. In fact, the truer a news item feels, the more suspicious I need to become and the more verification it requires.

Yellow journalism is nothing new, but the rapid application of technology to make the news seem authentic is the biggest difference from the past.

Currently Reading

“The Second Ship (The Rho Agenda Book 1)”: I got this Richard Phillips recommendation from Steve Gibson’s science fiction reading guide (see the top link on https://www.grc.com/linkfarm.htm). It initially felt a little bit too YA, but then it took off like a rocket. It’s fun well-written scifi. A fun sentence from the book:

Fresh snow was fun and could sometimes get you an extra day off. Old snow made you feel as gray and dirty as it was.

Thoughts? Feedback? Let me know: @eli4d on Twitter

The eli4d Gazette – Issue 060: LinkedIn’s Site Reliability Engineering Group’s approach to Python CLI based Utilities through PEX and Shiv

I recently came across a neat article in LinkedIn’s blog. It was written by their Site Reliability Engineering group (SRE) which covered some interesting Command Line Interface (i.e., CLI) utility libraries related to Python.

I can see how the power of Python would be fantastic for CLI utilities. The problem is Python’s dependency management. The dependency issue is even more compounded for CLI utilities since a devops engineer would need to be able to bring over a utility to any server (assuming a consistent version of the operating system) without worrying about the utility’s dependencies. Such an issue would prevent a system admin from doing the needed work (whether emergency or planned). In SRE’s article they explain this eloquently:

At LinkedIn, we ship hundreds of command-line utilities to every machine in our data centers and to all of our employees’ workstations. The vast majority of these utilities are written in Python.

In addition to developing these command-line utilities, we have hundreds of supporting libraries that are constantly being iterated on, with new versions published daily. Because of the inherent problems present when dealing with such a huge and ever-changing dependency graph, we need to package the executables individually to avoid dependency conflicts. Initially, we took advantage of the great open source tool PEX. PEX elegantly solved the isolated packaging requirement we had by including all of a tool’s dependencies inside a single binary file that we could then distribute.

PEX was not sufficient for SRE’s usage, so they created their own dependency packager called shiv. It’s an interesting name choice. I initially thought it was a prison-made knife, but apparently the name was based on the word “shiver” although the documentation has a knife emoji. In their words:

Why shiv?

>The tool freezes a Python environment, so you can think of shiv as a shorter way of saying “shiver.”

PEX and Shiv are interesting utilities that bring the power of Python to the command line through one executable utility.

Thoughts? Feedback? Let me know: @eli4d on Twitter

The eli4d Gazette – Issue 056: Python’s former BDFL and the Apollo 11 Moon Landing

Python’s former BDFL

Human languages take millennia to develop, whereas programming languages can take less than a month for initial design. Human languages are evolutionary, and programming languages take on the preferences, principles, and beliefs of their designer.

If the programming language is successful in terms of adoption and usage, then the language moves from its (initial) one person design to evolution that is based on the opinions and preferences of many people in conjunction with the emergence of current trends and fads (including competing languages).

Many times, the language designer is throned as its BDFL – benevolent dictator for life. He or she gets to have final say on the evolution of the language…until they have enough.

Python was created by Guido Van Rossum in 1991. It’s been evolving for 27 years with Guido as its BDFL until last week.

Twenty-seven years is a very long time for a programming language to both survive and thrive. It’s also a long time to wrestle and dispute the various proposals for Python’s evolution.

Apollo 11 Old Style Re-broadcast

On a lighter note, I came across this charming page about the Apollo 11 moon landing (mentioned in a recent Studio Neat issue). While the date/time re-broadcast is over, you can see the Apollo 11 landing through some links to YouTube.

Thoughts? Feedback? Let me know: @eli4d on Twitter

The eli4d Gazette – Issue 054: Dependency and Package Managers in Python and PHP

Python Environment and Package Installation

I saw a recent XKCD comic that describes the current state of setting up a Python development environment:

Python Environment

The excellent Explain XKCD site covers the point of the cartoon. The issue is the (ongoing) difficult transition from Python 2.x to Python 3.x and the reality that there is no defacto package installation tool. The messiness of Python’s development environment deeply contrasts with the cleanliness and beauty of the actual language.

One interesting and significant effort in approaching the above issue can be found through the PyPi package manager. A great FLOSS Weekly episode had an interview with one of the maintainers of the new PyPi covering its pros and cons.

PHP’s Composer and Package System

In the PHP world – Packgist site is the definite destination for third-party PHP packages. In conjunction with these packages, Composer is the defacto package/dependency management tool. It’s more than a package manager because it focuses on managing packages on a per project basis.

At its core Composer focuses on dependency management and the secondary result is that it also does package management. One problem that arises is the usage of Composer as a pure package installer across a system rather than just a project. As Composer’s documentation states:

By default it does not install anything globally. Thus, it is a dependency manager. however support a “global” project for convenience via the global command.

Composer’s “global” command installs packages in a common central project and this, in turn, can cause major headaches with projects that have packages installed in different locations.

Pantheon’s blog describes this issue (and its solution) really well:

One of the most commonly documented ways for a PHP command line tool to be installed is via the composer global require command. This command is easy to document and easy to run, which explains its popularity. Unfortunately, this convenient function has a darker side that can cause some pretty big problems. The root of the problem is that Composer, by design, manages dependencies on a per-project basis; however, the global command installs everything into a common central project. The upshot of this is that two distinct projects that were never intended to be combined must suddenly share dependencies. In this configuration, it is all-too-common to encounter dependency conflicts that never would have been observed had these applications been installed independently.

The current solution to this is a Pantheon created tool called Cgr. The blog article provides details on its usage and when to use Composer global.

Note: many thanks to the folks on the Poststatus Slack for discussing Composer’s issues and pointing to Cgr.

The eli4d Gazette – Issue 052: The Ship of Theseus & Building cross-platform apps in Python

The Ship of Theseus and Story Loops

Imaginary Worlds is one of my favorite podcasts. The most recent episode was amazing in how it related the question of identity through the Ship of Theseus, Westworld, and Star Trek.

There is much that I could say about it, but I think that it would be best to listen to it. So let me leave you with a quote from the podcast that was great:

It (i.e. Westworld) taps into the fact that we’re all on story loops. Some of these story loops are created by us, sometimes they’re created by society, but either way, after a while you just tend to go through the motions (i.e. just like the hosts on Westworld). And every so often it’s good to stop and reflect if this is the version of you that you want to be. It’s easy – all you have to do is ‘freeze all motor functions’.

Building cross-platform apps in Python

I’m not sure if cross platform development is like a unicorn in the software field. Whether it was Java’s “write once, run anywhere” promise or Electron’s pitch of “build cross platform desktop apps with JavaScript, HTML, and CSS”.

I came across an interesting project whose goal is to provide cross-platform development for Python. The BeeWare Project “aims to take the power of Python as a language, and use it to enable users of all skill levels to develop applications with native user interfaces.” This may be an interesting project for those focusing on learning Python.

Thoughts? Feedback? Let me know: @eli4d on Twitter

The eli4d Gazette – Issue 48: Learning Django 2 and the 2018 Stack Overflow Developer Survey

If you’re interested in learning Django 2

As a teacher and a learner, I’m on the lookout for great folks that create courses. If you’re interested in learning the Django framework (this may especially be of interest to my former Python students) then you might be interested in this kickstarter campaign by Tracy Osborn. Tracy approaches programming from a web designer point-of-view, and this is quite valuable from a “beginner’s mind” approach.

Check out the 2018 Stack Overflow Developer Survey

Stack Overflow is an amazing question and answer site that covers a tremendous amount of software development issues. It started originally as a response to Experts Exchange and has far exceeded its competitor’s breadth, depth, and utility.

Stack Overflow’s yearly development surveys have become an industry staple for software development trends. It’s an in-depth survey that has become better and better over the years. The 2018 survey is quite deep and extensive and is a well worth read.

If you’re looking for a concise summary of the 2018 survey, then check out Quincy Larson‘s summary.

Thoughts? Feedback? Let me know: @eli4d on Twitter

The eli4d Gazette – Issue 032

Tech Pick

I’ve had Python on my brain lately due to heavy duty preparation for my upcoming online Python course. So coming across Instagram’s Python technology stack was interesting. The article begins with:

Each day, over 95 million photos and videos are shared on Instagram. The unstoppable photo-centric social media platform has over 600 million registered users — 400 million of whom are active every day. Talk about operating at scale: Instagram kills it at levels most companies can barely even dream about.

Even more impressive, though, is the fact that Instagram serves this incredible amount of traffic, reliably and steadily so, by running Python (with a little help from Django) under the hood. Yes, that Python — the easy to learn, jack-of-all-trades general purpose programming language. The one everybody in the industry dismisses as, “Yeah, Python is great in so many ways, too bad it’s not really scalable.”

I thought that this was a great example to mention to my students – “well if Instagram uses Python at their scale, Python must be a good thing to learn.” And yet this didn’t sit right with me, and I remembered reading an excellent article that clarified my unease: “You Are Not Google”.

Python is a great language for many reasons (easy to learn, lots of built in libraries, great scientific/numeric support (SciPy, Pandas, iPython, NumPy), etc..) but Instagram’s use of it is not one of them. I’m not Instagram, and it’s very likely the case that you aren’t either.

Media Pick

I’ve had “American Gods”aal on my Kindle for what seems like forever. It was recommended by a friend back in 2004 and I got the Kindle version on sale a few months ago (via Bookbub).

Where to begin? The story is like a fractal of a pick-up stick game. You think “really Neal – a pick-up stick game?” And there’s Gaiman laughing manically. So you play the game and read the book 1 bit at a time…and just when you think you understand, repeating themes and whispers of previous chapters slap you across the face.

My Kindle highlights are filled with too many sentences from this book, and I can hardly pick a favorite. Here’s an example:

“The house smelled musty and damp, and a little sweet, as if it were haunted by the ghosts of long-dead cookies.”

aal = Amazon affiliate link

Random

I’m a big OmniFocus, but recent (XML) corruptions have been worrisome (on the positive side OmniFocus support is top notch). One database corruption happened in March, and another just happened about a week ago. My favorite data format is plain text. But I haven’t come across a plain text task management system that implements GTD and spans mobile and Mac OS X as seamlessly as OmniFocus.

Thoughts? Feedback? Let me know: @eli4d on Twitter