Experiments With Python Packages

Maybe I’m missing something here but Python’s system for creating, importing, publishing, and installing modules and packages and distributables seems… complicated. And a bit arbitrary. And more than a little counter-intuitive. And distinctly voodoo. Of course, it may well be that it only seems complicated to me because I haven’t got a clue what I’m doing. Or, perhaps it’s wonderfully elegant and efficient and powerful and I just don’t appreciate those features of it yet because I haven’t got a clue what I’m doing. Or, maybe I’m just a bit of a noob who hasn’t got a clue what he’s doing. Given that there appears to be a common element associated with my lack of clue, it’s time to have an explore. Experiment time!

Todo List

A few things have appeared on my radar during my flounderings so I’ve compiled a list of things to dig into:

• What are the limitations on where I can import from?
• What are the pros and cons of relative and absolute imports?
• What defines a module?
• What defines a package?
• Do we still need __init__.py files in a post-Python3.3 world?
• How does all this fit into virtual environments?
• How does all this integrate with tools like pip and uv?
• How can I integrate PyTest into my projects for unit testing?
• Why do I get so many module not found errors?
• When do we need a .toml file?
• How can I create a library of ‘utility’ routines that I can share across my projects?
• How can I publish a project to PyPi? Or should I just throw it up on Github?

Given the complexity of the whole thing and the number of rabbit holes that have distracted me from my well intentioned learning path to the point that I’ve forgotten, on several occasions, what exactly I’ve been trying to achieve, I think I need to take a methodical approach. With that in mind, I’m going to start with the very basics.

Caveat: some of what I’m going to try here is stupid, poor practice, obviously going to fail etc. etc. I’m working on the principle that you need to understand what things don’t work in order to understand why the things that do work, work. For example, putting my ‘utility’ code in a file called utils.py (as I’m about to do) will turn out to be a bad idea because there’s already a package in PyPi with that name. Don’t get me started on why that little wrinkle is already starting to piss me off.

Experiment 0

I’m starting with everything in a single folder experiment0:

utils_experiments/
└── experiment0
    ├── project.py
    └── utils.py

The ‘utility’ code lives in utils.py:

def a():
  print("a")

and it gets exercised by code in project.py

import utils

utils.a()

As expected that works just fine:

• When called from experiment0/ with python3 project.py it prints out a.
• When called from utils_experiments/ with python3 experiment0/project.py it also prints out a.