Python modules and packages

Source of this lab is hosted at: https://gitlab.com/jans-workshops/modules-packages-1

In this lab we will create simple python module and package. Purpose of this lab is to show how you can structure your project and learn the basics of import logic. The lab is expecting you to be familiar with your terminal and do basic operations.

Tasks will take you through several levels of working with modules and packages.

  • Level 1 - Create simple python module
  • Level 2 - Create simple package and use it in external module
  • Level 3 - Create installable package and use it within virtual environment

Every level should be in it’s own directory where our root directory will be code. And our lab structure will look like this:

code
├── level_1
├── level_2
├── level_3
└── level_4..5...etc

Modules and package we crate will contain simple caesar cipher encoder and decoder. Principle of Caesar Cipher is to replace every letter in message with another letter at given position n.

Example:

Message: alphabet
Shift: 3
Cipher: doskdehw

Note: Caesar could use only alphabet during his time, however with python we are using space of entire UTF-8, so you can shift by many more characters and produce non alphanumeric text. On the top of that python is case sensitive which means that position of A is not the same as positon of a.

ord('A')
# 65

ord('a')
# 97

You can find all codes in directory resources/code

Level 1

Python module is a single .py file. Module can be executed as a script or called from another module.

We are going to create simple python module caesar.py and execute it.

Final structure of the code

level_1
├── caesar.py
└── use_it.py

  1. Create new directory: mkdir level_1

  2. Go to the directory cd level_1

  3. Create new python file: caesar.py

  4. Open the file in your favorite text editor

  5. Function for encoding into Caesar cipher looks like this

    • You can copy paste the code into the file
    def encode(shift: int, text: str) -> str:
    """
    Encodes text with caesar cipher.

    Parameters
    ----------
    shift
        Number of characters to shift.

    text
        Text to encode

    Returns
    -------
    str
        Text shifted by shift characters.

    """

    shifted_numbers = map(lambda char: ord(char) + shift, text)
    return ''.join(map(lambda num: chr(num), shifted_numbers))

  6. To test your code simply add following few lines under the function

    if __name__ == '__main__':
    message = "Your secret text."
    cipher = encode(4, message)
    print(cipher)

  7. Save the file

  8. And run the code with command python caesar.py

    • if everything went well the output should look like this.
    ]syv$wigvix$xi|x2

  9. Let’s add decode function to our code and test if it works.

    def encode(shift: int, text: str) -> str:
    """
    Encodes text with caesar cipher.

    Parameters
    ----------
    shift
        Number of characters to shift.

    text
        Text to encode

    Returns
    -------
    str
        Text shifted by shift characters.

    """

    shifted_numbers = map(lambda char: ord(char) + shift, text)
    return ''.join(map(lambda num: chr(num), shifted_numbers))


def decode(shift, text):
    """
    Decodes caesar cipher.

    Parameters
    ----------
    shift
        Number of characters to shift.

    text
        Text to decode

    Returns
    -------
    str
        Decoded text.

    """
    shifted_numbers = map(lambda char: ord(char) - shift, text)
    return ''.join(map(lambda num: chr(num), shifted_numbers))


if __name__ == '__main__':
    message = "Your secret text."
    cipher = encode(4, message)
    # use encoded message as input
    decoded = decode(4, cipher)

    print(f"Oridignal message: {message}")
    print(f"Encoded message: {cipher}")
    print(f"Decoded message: {decoded}")

  10. Save the file and run our code again python caesar.py

    • The output should looks like this:
    Oridignal message: Your secret text.
Encoded message: ]syv$wigvix$xi|x2
Decoded message: Your secret text.

We have successfully created executable python module.

Let’s try to use our module from another module

  1. Create new file: use_it.py in the same directory as caesar.py

  2. Insert the following code:

    • To use another module, use the the keyword import <module_name>.
    • Then you can use the functions with dot notation module.function()
    import caesar

if __name__ == '__main__':
    message = "We call the caesar module!"
    cipher = caesar.encode(4, message)
    # use encoded message as input
    decoded = caesar.decode(4, cipher)

    print(f"Oridignal message: {message}")
    print(f"Encoded message: {cipher}")
    print(f"Decoded message: {decoded}")

  3. Save the file and run it python use_it.py

    • Output should look like this:
    Oridignal message: We call the caesar module!
Encoded message: [i$gepp$xli$geiwev$qshypi%
Decoded message: We call the caesar module!

You may noticed that did not execute all code from caesar.py in use_it.py.

This is because of the condition if __name__ == '__main__', to check the python documentation to learn more about __main__.

Level 2

In this level we will reuse our caesar.py from previous task to create python and create python package. The package usually contains multiple modules, so we will create our package named ciphers.

Once we have our package in place we can start importing modules from it. We are going to try very simple case with just one module, so you can add more modules later on if you wish.

Don’t forget to check the python import system documentation for more info.

Final structure of the code for level_2 task.

level_2
├── ciphers
│   ├── __init__.py
│   └── caesar.py
├── try_me
│   └── try_me.py
└── use_it.py

Let’s prepare our pacakge.

  1. Create directory mkdir level_2
  2. Go to directory cd level_2
  3. Create another directory mkdir ciphers
  4. Go to the directory cd ciphers
  5. Copy our caesar.py from level_1 into directory ciphers (you can use command below):
    • MacLinux: cp ../../level_1/caesar.py .
    • Windows: cp ..\..\level_1\caesar.py .
  6. Create new file __init__.py
  • You can leave the file empty
  • __init__.py will tell python to understand the directory as a module
  • and our caesar.py will become submodule

Now that our package is ready to use, let’s use it!

  1. Go back to directory level_2: cd ..

  2. Create new file use_it.py

    • We are going to use our package from here

  3. Open the file use_it.py in your favorite text editor

  4. And insert following code and save the file

    from ciphers import caesar

if __name__ == "__main__":

    message = "I'm using ciphers package"

    encoded = caesar.encode(4, message)

    # use encoded message as input
    decoded = caesar.decode(4, encoded)

    print(f"Oridignal message: {message}")
    print(f"Encoded message: {encoded}")
    print(f"Decoded message: {decoded}")

  5. Run the code: python use_it.py

    • Expected output:
    Oridignal message: I'm using ciphers package
Encoded message: M+q$ywmrk$gmtlivw$tegoeki
Decoded message: I'm using ciphers package

Now imagine we would like to use our from within another project. But it won’t be that easy as it looks like. Try it with the steps below.

  1. Create new directory mkdir try_me

  2. Go to the directory cd try_me

  3. Create new file try_me.py

  4. Insert the similar code as before:

    from ciphers import caesar

if __name__ == "__main__":

    message = "I'm trying to use ciphers package"

    encoded = caesar.encode(4, message)

    # use encoded message as input
    decoded = caesar.decode(4, encoded)

    print(f"Oridignal message: {message}")
    print(f"Encoded message: {encoded}")
    print(f"Decoded message: {decoded}")
    • Instead of printing the output, we got an error
    Traceback (most recent call last):
  File "try_me.py", line 1, in <module>
    from ciphers import caesar
ModuleNotFoundError: No module named 'ciphers'
    • As you can see python throws ModuleNotFoundError exception. Python is searching for modules in PYTHONPATH and it does not containe path to the directory level_2/ciphers.
    • The easiest way to fix this is to make your module installable in level_3 task.
    • Check documentation for more information about PYTHONPATH and sys.path

Level 3

This task will take us to completely different level where we create installable package.

This lab will be similar to official packaging guide. Once your package is finished you can distribute it to PyPI (currently out of scope of this lab)

We will need to modify our structure a bit so in the end it will look like this:

level_3
├── app_1
│   ├── Pipfile
│   ├── Pipfile.lock
│   └── main.py
└── ciphers_project
    ├── MANIFEST.in
    ├── README.rst
    ├── ciphers
    │   ├── __init__.py
    │   ├── __version__.py
    │   └── caesar.py
    └── setup.py

Let’s start:

  1. Create a directory mkdir level_3

  2. Go to the directory cd level_3

  3. Create another directory mkdir ciphers_project

    • This is an umbrella directory for entire project, it will contain our package + necessary files to produce installable package

  4. Go to the project directory cd ciphers_project

  5. Copy our package from level_2: cp -r ../../level_2/ciphers .

    • If you do not have the code from previous level, go to the resources/level2 and copy ciphers directory.

  6. Let’s create setup.py file

    • setup.py file is an entrypoint where we specify package metadata and other requirements
    • For more information check references and setuptools documentation.

  7. Open setup.py in your favorite editor and write following code:

    • Adjust valuies of EMAIL and AUTHOR to your name and email.
    from setuptools import setup, find_packages
import os

NAME = "ciphers"
DESCRIPTION = "Example package providing ciphers."
URL = ""
EMAIL = "1oglop1@gmail.com"
AUTHOR = "Jan Gazda"
REQUIRES_PYTHON = ">=3.6.0"
VERSION = "0.1.0"

# What packages are required for this module to be executed?
REQUIRED = [
    # 'requests', 'maya', 'records',
]

# The rest you shouldn't have to touch too much :)
# ------------------------------------------------
# Except, perhaps the License and Trove Classifiers!
# If you do change the License, remember to change the Trove Classifier for that!

here = os.path.abspath(os.path.dirname(__file__))

# Import the README and use it as the long-description.
# Note: this will only work if 'README.rst' is present in your MANIFEST.in file!
with open(os.path.join(here, "README.rst"), encoding="utf-8") as f:
    long_description = "\n" + f.read()

setup(
    name=NAME,
    version=VERSION,
    description=DESCRIPTION,
    long_description=LONG_DESCRIPTION,
    author=AUTHOR,
    author_email=EMAIL,
    python_requires=REQUIRES_PYTHON,
    url=URL,
    packages=find_packages( exclude=("tests",)),
    # entry_points={
    #     'console_scripts': [
    #         # 'mycli=mymodule:cli',
    #     ],
    # },
    install_requires=REQUIRED,
    include_package_data=True,
    license='MIT',
    classifiers=[
        # Trove classifiers
        # Full list: https://pypi.python.org/pypi?%3Aaction=list_classifiers
        "License :: OSI Approved :: MIT License",
        "Programming Language :: Python",
        "Programming Language :: Python :: 3",
        "Programming Language :: Python :: 3.6",
        "Programming Language :: Python :: Implementation :: CPython",
        "Programming Language :: Python :: Implementation :: PyPy",
    ],
)

  8. Create another file README.RST and open it in your favorite editor.

    • README.rst - All projects should contain a readme file that covers the goal of the project.
    • Feel free to use my example:
    ===============
Ciphers package
===============


This is an awesome package providing various ciphers.

  9. Save the file README.rst

  10. Create another file MANIFEST.in

    • MANIFEST.in - needed when you need to package additional files that are not automatically included in a source distribution.
    • type following into the file and save it.
    include README.rst

We have just made necessary preparations to create installable package. The structure of the project should look like this:

ciphers_project
├── MANIFEST.in
├── README.rst
├── ciphers
│   ├── __init__.py
│   ├── __version__.py
│   └── caesar.py
└── setup.py

We are going to do two things:

  • install our package into separate virtual environment.
  • Build portable installer wheel

Installing the package in virtual environment.

  1. Go to the task directory level_3

    • if you have followed all steps you should be able to use cd ..

  2. Create new directory mkdir app_1

  3. Go to the directory cd app_1

  4. Install our package using pipenv: pipenv install ../ciphers_projectpip

  5. Verify package was correctly installed: pipenv graph

    ciphers==0.1.0

Let’s use the package we have just installed.

  1. Create new file app_1.py and open it in your favorite text editor.

  2. Insert following code:

    import ciphers
from ciphers import caesar

if __name__ == "__main__":

    print("We are using package ciphers loaded from", ciphers)

    message = "Hello World!"

    encoded = caesar.encode(4, message)

    # use encoded message as input
    decoded = caesar.decode(4, encoded)

    print(f"Oridignal message: {message}")
    print(f"Encoded message: {encoded}")
    print(f"Decoded message: {decoded}")

  3. Run the code: pipenv run python app_1.py

    • Desired output:
    We are using package ciphers located at: <module 'ciphers' from '/Users/user/.virtualenvs/app_1-2aSkwHkG/lib/python3.6/site-packages/ciphers/__init__.py'>
Oridignal message: Hello World!
Encoded message: Lipps$[svph%
Decoded message: Hello World!

Okay we have learned that we can install package from it’s directory. But we can also build an installer and share it with your friends.

Building wheel

Wheel A wheel is a ZIP-format archive with a specially formatted file name and the .whl extension. You can then distribute this archive and share your python packages. PEP 427 – The Wheel Binary Package Format 1.0

We are going to use setup.py to build the wheel.

  1. Go to the project folder cd ciphers_project

  2. And type: python setup.py bdist_wheel

    • If everything went well, command created new folders in our project
    ciphers_project
├── build
│   ├── bdist.macosx-10.6-intel
│   └── lib
│       └── ciphers
│           ├── __init__.py
│           └── caesar.py
├── ciphers.egg-info
│   ├── PKG-INFO
│   ├── SOURCES.txt
│   ├── dependency_links.txt
│   └── top_level.txt
├── dist
│   └── ciphers-0.1.0-py3-none-any.whl
└── setup.py
    • build - This is were our package is being build
    • ciphers.egg-info - egg-info directory contains Python egg metadata, regenerated from source files by setuptools.
    • dist - Setuptools distribution folder. Is the directory where we can find our wheel.
    • All data from directories above were automatically generated and you can safely remove and exclude them from VCS.
    • For more info about building also check: https://docs.python.org/3.6/install/#how-building-works

You can now distribute this wheel to your friends, customer or publish it to PyPI.