Dev Blog

How to upload Python code to PyPi

2020-04-28T00:00:00+05:30

Today in this post I will be talking on how to upload your Python code to Python Package Index (PyPi) so that other developers around the world can benefit from the code you wrote.

PyPi is a software repository for Python programming language. This a similar to Git repositories, except that PyPi is exclusively for Python packages. Packages uploaded here are installed using pip a Python package installer. So, let’s move ahead and let me tell you how you can let other developers make use of the code you have written.

So, let’s get started…

Creating the package

I am assuming that you already have some code files ready to be distributed. Apart from the code files we also need some specific files at specific locations. Following is the suitable file structure:

package_name
├── module_1
│  ├── __init__.py
│  └── other_py_files.py
├── module_2
│  ├── __init__.py
│  └── other_py_files.py
├── README.md
├── LICENSE
└── setup.py

In the above structure:

__init__.py: these can be empty file, but they need to be present in all the directories that you wish to be imported in python code, using import module_1. Hence, module_1 and module_2 can be imported.
LICENSE: it is advisable that you include a license. If you are not sure as to which license to include you can have a look Choose an open source license.
README.md: do add a readme, giving enough info related to your package. This help others figure out if a package is suitable for their needs and also an idea as to how to use the code.
setup.py: this is a build script required by setuptools, for now create this file, we will add content in the following section.

The setup file

setup.py is a build script, that contains all the necessary information needed for setuptools for installing the package. Open the setup.py file and add the following content:

with open("README.md", "r") as fh:
   long_description = fh.read()

setuptools.setup(
    name="package_name", # Replace with your own username
    version="0.0.1",
    author="Example Author",
    author_email="author@example.com",
    description="A small example package",
    long_description=long_description,
    long_description_content_type="text/markdown",
    url="https://github.com/pypa/sampleproject",
    packages=setuptools.find_packages(),
    classifiers=[
        "Programming Language :: Python :: 3",
        "License :: OSI Approved :: MIT License",
        "Operating System :: OS Independent",
    ],
    python_requires='>=3.6',
)

Let me explain the above content.

The setuptools.setup() function takes several arguments, some of the basic one are mentioned here.

name: this will be the name of the package. Make sure that it contains only alphabets, numbers, hyphen (“-“) and underscore (“_”). Also that it should be unique and not be taken up by other packages on PyPi.
version: this will the version number for your package. Have a look at, PEP 440 – Version Identification and Dependency Specification for more details on version numbers.
author: name of the author of the package.
author_email: contact email of the author of the package.
description: one liner description of the package.
long_description: detailed description of the package. This will be displayed on the package description page on PyPi. Generally this is loaded from a file. The following lines form above snippet are used to load the content from a file:
```
  with open("README.md", "r") as fh:
      long_description = fh.read()
```
Most popular file types used are markdown (md), reStructured text (rst) or plain text (txt).
long_description_content_type: type of the file/content provided to long_description. Options are text/plain, text/x-rst, or text/markdown for plain text, restructured or markdown files respectively.
url: url to project’s homepage
packages: list of all other Python packages required by your package. To make your life easier, you don’t need to list them manually, use the setuptools.find_packages() function, this will list all the packages and sub-packages used.
classifiers: provides pip and PyPi some metadata related to our package. In the above example it states that it requires Python 3, has a MIT license and is OS independent. For full list of classifiers head off to, https://pypi.org/classifiers/.
python_requires: mentions the Python version required by your package.

The arguments mentioned here provides some basic information about the package. If you wish to know all the arguments you can have a look at What keyword arguments does setuptools.setup accept.

Generating distribution packages

Now that our project structure and setup file is ready, let’s move on and build our project into distribution packages to be uploaded.

I assume that you already have setuptools and wheels installed and updated. If not use the following code to do so:

python3 -m pip install --upgrade setuptools wheel

If you have trouble doing so, have a look at Installing Packages.

Now, navigate to the root directory of your project, that is, to the directory where we have our setup.py file, and execute the following command:

python3 setup.py sdist bdist_wheel

If you have trouble creating these packages, then you can report it by creating a Github issue at Packaging Problems.

After successful execution of this code a new directory will be creating with following files in it:

dist/
  package_name-0.0.1-py3-none-any.whl
  package_name-0.0.1.tar.gz

You will have two files in dist directory at the root of your project:

.tar.gz: source file, this is nothing but compressed raw source code ready to be distributed
.whl: distribution package, this contains metadata and all files ready to be moved to specific location on the target machine, for your package to be installed

Uploading the files

Now that we have our package ready to be distributed, let’s upload them!

Upload Test

Yes! you read it right. We will test uploading of our package first. And no, we won’t be uploading our test package on PyPi’s live servers, and make a mess. PyPi have made sure of that. Along with their live server they also maintain a test server.

So go ahead log onto their test server and register yourself.

One more thing, that we need before we can upload the files. Use the following code to install twine. Twine is the primary and official tool for developers to upload files to PyPi:

python3 -m pip install --upgrade twine

Now we have everything set-up to upload our files, using the following code:

python3 -m twine upload --repository testpypi dist/*

In the previous command, --repository is used to mention that we are uploading our package to test server and not to live server.

Type in your username and password when you will be prompted for. When you do so your file should get uploaded and you should see the following:

Uploading distributions to https://test.pypi.org/legacy/
Enter your username: [your username]
Enter your password:
Uploading package_name-0.0.1-py3-none-any.whl
100%|█████████████████████| 4.65k/4.65k [00:01<00:00, 2.88kB/s]
Uploading package_name-0.0.1.tar.gz
100%|█████████████████████| 4.25k/4.25k [00:01<00:00, 3.05kB/s] 

View at:
https://test.pypi.org/project/package_name/0.0.1/

You can now view your project on PyPi test server. Head off, to the url mentioned at the end of the previous command.

Verify the upload

Now that we have uploaded our package, let’s install this directly for PyPi server using pip and verify our upload. Use the following code to do so:

python3 -m pip install --index-url https://test.pypi.org/simple/ --no-deps package_name

Let me explain you the options used in previous command:

--index-url: to mention that we are installing from test server and not from live PyPi server
--no-deps: this tells pip to not install any dependencies (if any) for the package. And that is because, data on the test server can be purged anytime without notice, as a result, installing dependencies may fail or may install something unexpected. It is always recommended to use --no-deps while installing anything from PyPi test server.

Use should see the following output for the previous command, indicating successful download and installation of your package:

Collecting package_name
  Downloading https://test-files.pythonhosted.org/packages/.../package_name-0.0.1-py3-none-any.whl
Installing collected packages: package_name
Successfully installed package_name-0.0.1

Note: Never rely on PyPi test server, all packages along with user accounts from this server is occasionally deleted and purged.

Congratulation, you have packaged and uploaded your Python project!

Uploading to and Installing from Live Server

Now, to upload your package to live server, you have to follow same steps that you performed to upload your package to test server, but with the following important modifications:

Choose a memorable and unique name for you package.
Register an account on live PyPi server at https://pypi.org – note that live and test servers are two different servers, hence no details are shared among these two servers.
To upload the package to live server you don’t need to mention --repository. Just use the following, this will upload your package to https://pypi.org, by default:
```
  python3 -m twine upload dist/*
```
Similarly, you don’t need to mention --index-url while installing the package, just use the following:
```
  python3 -m pip install package_name
```

Remember, you also need to upload your packages on live PyPi server, even if you have uploaded your packages on test server. As the name suggest, it is a test server and is meant only for testing. All packages, including user account details are occasionally deleted. So, after testing and verifying everything on test server, upload your packages on LIVE server at PyPi.org.

Raspberry Pi Monitoring

2020-04-20T00:00:00+05:30

In this write-up, I will be explaining the steps required to setup your Raspberry Pi with Grafana, InfluxDB and Telegraf to monitor the system.

Hardware setup that I used:

Raspberry Pi 4 Model B (Raspbian installed)

So let’s get started…

Step 1: Installing Docker

Docker provides a convenient way to install latest version of Docker into your development environment, we will be using the convenience script today. Although there are other ways to install Docker on Linux destro, this is the only way to install Docker on Rasbian OS.

Note, this script requires root privileges to run successfully. Script will take care of all dependencies and recommendations, and will install the latest version of Docker. The script is available here.

Let’s download the script and execute it

 curl -fsSL https://get.docker.com -o get-docker.sh
 sudo sh get-docker.sh

If you wish to run Docker as a non-root user, add your user to docker group
```
 sudo usermod -aG docker <your-username>
```
Next, let’s install docker-compose, this enables us to define and run multiple docker containers
```
 sudo apt install -y docker-compose
```
Now restart your Pi for all changes to take effect
```
 sudo reboot
```

Step 2: Creating docker-compose file

Now as the title says, we need to setup three things:

Grafana
InflucDB
Telegraf

To make things easier, we will make use of docker-compose instead of setting up each container individually. So, create a file with the following content using nano/ vim or an editor of your choice:

version: '3'
services:
    grafana:
        container_name: grafana
        image: grafana/grafana
        ports:
            - '3000:3000'
        volumes:
            - grafana-storage:/var/lib/grafana
        links:
            - influxdb
        restart: always
    influxdb:
        container_name: influxdb
        image: influxdb
        ports:
            - "8083:8083"
            - "8086:8086"
        volumes:
            - influxdb-storage:/var/lib/influxdb
        restart: always
    telegraf:
        container_name: telegraf
        image: telegraf
        volumes:
            - /var/run/docker.sock:/var/run/docker.sock
            - /proc:/host/proc
            - ${PWD}/conf/telegraf/telegraf.conf:/etc/telegraf/telegraf.conf
            - /opt/vc:/opt/vc:ro
        devices:
            - "/dev/vchiq"
        environment:
            LD_LIBRARY_PATH: /opt/vc/lib
        network_mode: host
        restart: always
volumes:
    influxdb-storage:
    grafana-storage:

Let me explain to you a bit about the above content.

version: states docker-compose file version
services: list of services that we wish to define and run using this file
container_name: name of the container that will be created for respective services, this will be useful later to run commands like docker logs <container_name>
image: docker image used to create container
ports: port combination used by our host and container. 8080:80 states that host’s port 8080 is mapped with container’s port 80, that is, all the traffic of host’s port 8080 will be routed to container’s port 80
volumes: paths of the host’s system that are used by container to have a persistent storage, when container is killed or restarted. Here we have used a named volume influxdb-storage, this must be mentioned in the top-level volumes key as well
links: way to make the container accessible by its alias, such as http://influxdb:8063
devices: way to bind a available device on host machine to container
environment: environment variables required and used by container
network_mode: states that container should use host system’s network, this doesn’t isolate container’s and host’s network which is the default behaviour
restart: states when should container be restarted
volumes: top-level key used to list the named volumes defined and used by the services

Step3: Configuring Telegraf

Telegraf is a plugin-driven server-agent that helps you collect all the necessary stats of a system, databases and/or sensors that you wish to monitor. Not only does it collect this data, but also sends this data to a database and/or cloud server to be stored.

Let’s configure it now, instead of creating the config file from scratch we will edit the existing file, to make things easier. To do so, we need to get this file from the docker image itself. First we will create a directory to store this file as mentioned in the docker compose file ${PWD}/conf/telegraf/telegraf.conf.

mkdir -p conf/telegraf
docker run --rm telegraf telegraf config > ./config/telegraf/telegraf.cong

Now, open this file. Beware! don’t be intimidated by the lines of code this file has. It’s well over 6.5K! But, don’t worry, we are concerned with not more than 50 lines.

So, open this file in nano/vim or an editor of your choice, and search for [[outputs.influxdb]].

You will have to uncomment some lines and make some edits. So that this section looks similar to the following:

[[outputs.influxdb]]
  urls = ["http://127.0.0.1:8086"]
  database = "telegraf"
  timeout = "5s"
  username = "telegraf"
  password = "<passowrd>"
 
  ## HTTP User-Agent
  user_agent = "telegraf"
 
  ## UDP payload size is the maximum packet size to send.
  udp_payload = "512B"

Now, add the following lines to it. If you wish to maintain the consistency of the file, search for [[inputs.temp]], and add these lines below it.

# # Read cpu temperature from file
[[inputs.file]]
   files = ["/sys/class/thermal/thermal_zone0/temp"]
   name_override = "cpu_temperature"
   data_format = "value"
   data_type = "integer"
 
 # # Get gpu temperature from VidoeCore general command service
[[inputs.exec]]
   commands = [ "/opt/vc/bin/vcgencmd measure_temp" ]
   name_override = "gpu_temperature"
   data_format = "grok"
   grok_patterns = ["%{NUMBER:value:float}"]

# # Get ARM clock frequency from VideoCore general command sevice
[[inputs.exec]]
  commands = ["/opt/vc/bin/vcgencmd measure_clock arm"]
  name_override = "cpu_clock"
  data_format = "grok"
  grok_patterns = ["=%{NUMBER:value:float}"]

# # Get GPU clock frequency from VideoCore general command sevice
[[inputs.exec]]
  commands = ["/opt/vc/bin/vcgencmd measure_clock gpu"]
  name_override = "gpu_clock"
  data_format = "grok"
  grok_patterns = ["=%{NUMBER:value:float}"]

Step 4: Launch the containers

Go ahead and bring up the containers to life with docker-compose -d up.

If everything goes right you should see the following:

Creating influxdb ... done
Creating telegraf ... done
Creating grafana  ... done

Step 5: Configuring InfluxDB

We need to create a user for Telegraf to interact with InfluxDB. Make sure that password used in this command is same as the one mentioned in telegraf.conf file.

  docker exec -it influxdb influx
> use telegraf
> create user telegraf with password '<password>' with all privileges

After running first line docker exec -it influxdb influx, your prompt should turn to > indicating that the command ran successfully.

Step 6: Configuring Grafana

Now head over to your browser, and navigate to Grafana’s address, or if you are doing this on local machine, then navigate to http://localhost:3000.

Go ahead and sign in. Use username as admin and password as admin. On next page you will be asked to change the default password. It is highly recommended that you do so.

Next, go to Settings > Data Sources, and click on Add Data Sources.

You should be presented with list of databases to configure as data sources. Select InfluxDB, listed under Time series databases.

On the next page set fields as mentioned below:

URL: http://influxdb:8086
Database: telegraf
User: telegraf
Password: enter password set for user telegraf

Note: Username and password must be same as that used while creating user in InfluxDB in the earlier command create user telegraf with password '<password>' with all privileges.

Now, click on Save & Test. You should get a green pop-up saying your data source is working. Now you are all set. All you have to do is create, or import a dashboard to monitor your system.

Step 7: Adding a Dashboard

Now, let’s create your own dashboard. But wait, why reinvent the wheel! Hence, I imported a dashboard from Grafana.com. Select a dashboard that suits your needs, copy the dashboard ID to your clipboard.

Now, head to Dashboard > Manage, click on the Import button. Paste the dashboard ID in the input field labeled Grafana.com Dashboard and click anywhere outside the field and wait for a while. Now, you will be presented with some new fields to edit. Select InfluxDB for the dropdown labeled influxdb and click on Import. That’s it, you should be presented with the dashboard you chose.

You could use the one I used to start with Raspberry Pi Monitoring.