Background

pip install git+ssh://git@bitbucket.org/esologic/sample_project.git

pip install git+ssh://git@bitbucket.org/esologic/sample_project.git@master # on the master branch 
pip install git+ssh://git@bitbucket.org/esologic/sample_project.git@0.0.2 # on the version tag of 0.0.2

(venv) dev@ESO-3:/tmp$ pip install git+ssh://git@bitbucket.org/esologic/sample_project.git
Collecting git+ssh://git@bitbucket.org/esologic/sample_project.git
  Cloning ssh://git@bitbucket.org/esologic/sample_project.git to ./pip-sjec1gbh-build
git@bitbucket.org: Permission denied (publickey).
fatal: Could not read from remote repository.

Please make sure you have the correct access rights
and the repository exists.
Command "git clone -q ssh://git@bitbucket.org/esologic/sample_project.git /tmp/pip-sjec1gbh-build" failed with error code 128 in None

Step 1: Make sure your repo CAN be installed as a python package

The key here is a proper setup.py file. Here are best the best set of docs I’ve found on how to make this file.

You can also look at the test repo for this project (https://bitbucket.org/esologic/sample_project/src/master/), it contains an example setup.py. This repo will also be the standard example for this post.

To make sure things are working correctly, you can try installing the package into your local python environment, or into a virtual one like I’m doing. Using sample_project as an example, we can do this like so:

(venv) dev@ESO-3:/tmp$ pip install /mnt/c/Users/dev/Documents/misc_git/sample_project/
Processing /mnt/c/Users/dev/Documents/misc_git/sample_project
Installing collected packages: sample-project
  Running setup.py install for sample-project ... done
Successfully installed sample-project-1.0
(venv) dev@ESO-3:/tmp$ python
Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> from sample_project import print_quote
>>> print_quote()
If they can get you asking the wrong questions, they don't have to worry about answers.
>>>

If your package behaves as expected when installed like this locally, you’re all set to push the changes to your bitbucket repo and continue with the rest of the guide.

Step 2: Create SSH keys and add them to bitbucket

Note: at a few places in this step I use my own email as a reference, dev@esologic.com. Make sure whenever you see that, to substitute email address associated with your bitbucket account.

If you already have ssh keys created on your computer or whatever you’re developing on, they should be located at ~/.ssh. If you don’t see both id_rsa and id_rsa.pub files in that directory, create them with:

ssh-keygen -m PEM -t rsa -C "dev@esologic.com"

Leave passphrase blank.

Now, copy the contents of ~/.ssh/id_rsa.pub to bitbucket. The following images should walk you through the steps, make sure to give the key a memorable name.

Now, the ssh key of whatever dev environment you’re on is added to bitbucket.

Windows steps to create ssh keys

$ ssh-keygen -m PEM -t rsa -C "dev@esologic.com" -E md5
$ cd C:\Users\dev\.ssh
$ ssh-add id_rsa
$ ssh -T git@bitbucket.org

Step 3: Make sure your account can read from the private repo with your python package

(venv) dev@ESO-3:/tmp$ pip install git+ssh://git@bitbucket.org/esologic/sample_project.git
Collecting git+ssh://git@bitbucket.org/esologic/sample_project.git
  Cloning ssh://git@bitbucket.org/esologic/sample_project.git to ./pip-nkrqsxao-build
setsockopt IPV6_TCLASS 8: Operation not permitted:
Installing collected packages: sample-project
  Running setup.py install for sample-project ... done
Successfully installed sample-project-1.0
(venv) dev@ESO-3:/tmp$ python
Python 3.6.8 (default, Jan 14 2019, 11:02:34)
[GCC 8.0.1 20180414 (experimental) [trunk revision 259383]] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import sample_project
>>> sample_project.print_quote()
If they can get you asking the wrong questions, they don't have to worry about answers.
>>>

Fantastic! Remember, your pip command git+ssh://git@bitbucket.org/esologic/sample_project.git will be different for your package. It will look something like this: git+ssh://git@bitbucket.org/{your username}/{your project}.git.

Step 5: Create a “machine user” in bitbucket

Step 6: Create SSH keys and add them to your machine user’s account

On whatever you system you have been using so far, enter the following commands and remember to leave passphrase blank.

mkdir ~/.ssh/ci_bot_keys
ssh-keygen -m PEM -t rsa -C "ci_bot@example.com" -f ~/.ssh/ci_bot_keys/id_rsa

Add the contents of ~/.ssh/ci_bot_keys/id_rsa.pub to bitbucket while signed in as your machine user like we did in step 2.

Step 7: Try git+ssh key insertion locally

(Note: you can skip this step, but if things don’t work when you add the step to your CI build start looking for errors here.)

export SSH_AUTH_SOCK=none
export GIT_SSH_COMMAND='ssh -i ~/.ssh/ci_bot_keys/id_rsa'

(venv) dev@ESO-3:/tmp$ cat ~/.ssh/ci_bot_keys/id_rsa | tr "\n" "_"

Go to build settings -> Environment Variables and then set the variable like so:

Now that the variable is set, we need to change our circle config to use it.

Step 9: Add the step to your /.circleci/config.yml file

- run:
    name: Install Dependencies
    command: |

      # Give us access to private repos
      export KEY_PATH=tmp_id_rsa
      echo -e "${KEY//_/\\n}" > $KEY_PATH
      chmod 600 $KEY_PATH
      export SSH_AUTH_SOCK=none
      export GIT_SSH_COMMAND='ssh -i $KEY_PATH'

      python3 -m venv venv
      . venv/bin/activate
      pip install -r ./requirements.txtts.txt

Make sure you select a circle image that has a git version of 2.17.0 or later, or this step will fail without an explanation. I found that the python image of circleci/python:3.7-buster worked when testing.

Key Learning

Python uses a Global Interpreter Lock to make sure that  memory shared between threads isn’t corrupted. This is a design choice of the language that has it’s pros and cons. One of these cons is that in multi-threaded applications where at least one  thread applies a large load to the CPU, all other threads will slow down as well.

For multi-threaded Python applications that are at least somewhat time-sensitive, you should use Processes over Threads.

Experiment

I wrote a simple python script to show this phenomenon. Let’s take a look.

def increment(running_flag, count_value):
    c = 0
    while True:
        if not running_flag.value:
            break
        count_value.value = c  # setting a Value is atomic
        c += 1

The core is this increment function. It takes in a Value and then sets it over and over, increment each loop, until the running_flag is set to false. The value of count_value is what is graphed later on, and is the measure of how fast things are going.

The other important bit is the load function:

def load(running_flag):
    z = 10
    while True:
        if not running_flag.value:
            break
        z = z * z

Like increment,  load is the target of a thread or process. The z variable quickly becomes large and computing the loop becomes difficult quickly.

The rest of the code is just a way to have different combinations of increment and load running at the same time for varying amounts of time.

Result

The graph really tells the story. Without the load thread running, the process and thread versions of increment run at essentially the same rate. When the load thread is running, increment  in a thread grinds to a halt compared to the process which is unaffected.

That’s all! I’ve pasted the full source below so you can try the experiment yourself.

from multiprocessing import Process, Value
from threading import Thread
from time import sleep
from ctypes import c_bool, c_longdouble


def increment(running_flag, count_value):
    """
    Increment the value in count_value as quickly as possible. If running_flag is set to false, break out of the loop
    :param running_flag: a multiprocessing.Value boolean
    :param count_value: a multiprocessing.Value Long Double
    """

    c = 0

    while True:

        if not running_flag.value:
            break

        count_value.value = c  # setting a Value is atomic
        c += 1


def load(running_flag):
    """
    Apply a load to the CPU. If running_flag is set to false, break out of the loop
    :param running_flag: a multiprocessing.Value boolean
    """

    z = 10

    while True:

        if not running_flag.value:
            break

        z = z * z


def mct(target, flag, value):
    """
    Returns a lambda that can be called to get a thread to increment a increment using a thread
    """
    return lambda: Thread(target=target, args=(flag, value))


def mcp(target, flag, value):
    """
    Returns a lambda that can be called to get a thread to increment a increment using a process
    """
    return lambda: Process(target=target, args=(flag, value))


def mlt(target, flag):
    """
    Returns a lambda that can be called to get a thread that will load down the CPU
    """
    return lambda: Thread(target=target, args=(flag,))


if __name__ == "__main__":

    f = Value(c_bool, True)  # control flag, will be passed into child thread/process so they can be stopped
    cv = Value(c_longdouble, 0)  # increment value

    child_lists = [mct(increment, f, cv)], [mcp(increment, f, cv)], [mct(increment, f, cv), mlt(load, f)], [mcp(increment, f, cv), mlt(load, f)]

    for delay in range(10):  # maximum run time of 10 seconds

        max_counts = []

        for get_children in child_lists:

            # reset the flag and increment
            f.value = True
            cv.value = 0

            # the child thread/processes will end up in here
            children = []

            for get_child in get_children:
                child = get_child()  # create a new instance of the thread/process to be launched
                child.start()
                children.append(child)

            sleep(delay)

            f.value = False

            for child in children:
                child.join()  # stop the process

            max_counts.append(cv.value)

        s = ""
        for count in max_counts:
            s += str(count) + " "
        print(s)