Writing Tests for github3.py

Unit Tests

In computer programming, unit testing is a method by which individual units of source code, sets of one or more computer program modules together with associated control data, usage procedures, and operating procedures are tested to determine if they are fit for use. Intuitively, one can view a unit as the smallest testable part of an application.

Unit Testing on Wikipedia

In github3.py we use unit tests to make assertions about how the library behaves without making a request to the internet. For example, one assertion we might write would check if custom information is sent along in a request to GitHub.

An existing test like this can be found in tests/unit/test_repos_release.py:

def test_delete(self):
    self.instance.delete()
    self.session.delete.assert_called_once_with(
        self.example_data['url'],
        headers={'Accept': 'application/vnd.github.manifold-preview'}
    )

In this test, we check that the library passes on important headers to the API to ensure the request will work properly. self.instance is created for us and is an instance of the Release class. The test then calls delete to make a request to the API. self.session is a mock object which fakes out a normal session. It does not allow the request through but allows us to verify how github3.py makes a request. We can see that github3.py called delete on the session. We assert that it was only called once and that the only parameters sent were a URL and the custom headers that we are concerned with.

Mocks

Above we talked about mock objects. What are they?

In object-oriented programming, mock objects are simulated objects that mimic the behavior of real objects in controlled ways. A programmer typically creates a mock object to test the behavior of some other object, in much the same way that a car designer uses a crash test dummy to simulate the dynamic behavior of a human in vehicle impacts.

Mock Object on Wikipedia

We use mocks in github3.py to prevent the library from talking directly with GitHub. The mocks we use intercept requests the library makes so we can verify the parameters we use. In the example above, we were able to check that certain parameters were the only ones sent to a session method because we mocked out the session.

You may have noticed in the example above that we did not have to set up the mock object. There is a convenient helper written in tests/unit/helper.py to do this for you.

Example - Testing the Release Object

Here’s a full example of how we test the Release object in tests/unit/test_repos_release.py.

Our first step is to import the UnitHelper class from tests/unit/helper.py and the Release object from github3/repos/release.py.

from .helper import UnitHelper
from github3.repos.release import Release

Then we construct our test class and indicate which class we will be testing (or describing).

class TestRelease(UnitHelper):
    described_class = Release

We can then use the GitHub API documentation about Releases to retrieve example release data. We then can use that as example data for our test like so:

class TestRelease(UnitHelper):
    described_class = Release
    example_data = {
        "url": releases_url("/1"),
        "html_url": "https://github.com/octocat/Hello-World/releases/v1.0.0",
        "assets_url": releases_url("/1/assets"),
        "upload_url": releases_url("/1/assets{?name}"),
        "id": 1,
        "tag_name": "v1.0.0",
        "target_commitish": "master",
        "name": "v1.0.0",
        "body": "Description of the release",
        "draft": False,
        "prerelease": False,
        "created_at": "2013-02-27T19:35:32Z",
        "published_at": "2013-02-27T19:35:32Z"
        }

The above code now will handle making clean and brand new instances of the Release object with the example data and a faked out session. We can now construct our first test.

def test_delete(self):
    self.instance.delete()
    self.session.delete.assert_called_once_with(
        self.example_data['url'],
        headers={'Accept': 'application/vnd.github.manifold-preview'}
    )

Integration Tests

Integration testing is the phase in software testing in which individual software modules are combined and tested as a group.

The purpose of integration testing is to verify functional, performance, and reliability requirements placed on major design items.

Integration tests on Wikipedia

In github3.py we use integration tests to ensure that when we make what should be a valid request to GitHub, it is in fact valid. For example, if we were testing how github3.py requests a user’s information, we would expect a request for a real user’s data to be valid. If the test fails we know either what the library is doing is wrong or the data requested does not exist.

An existing test that demonstrates integration testing can be found in tests/integration/test_repos_release.py:

def test_iter_assets(self):
    """Test the ability to iterate over the assets of a release."""
    cassette_name = self.cassette_name('iter_assets')
    with self.recorder.use_cassette(cassette_name):
        repository = self.gh.repository('sigmavirus24', 'github3.py')
        release = repository.release(76677)
        for asset in release.iter_assets():
            assert isinstance(asset, github3.repos.release.Asset)
        assert asset is not None

In this test we use self.recorder to record our interaction with GitHub. We then proceed to make the request to GitHub that will exercise the code we wish to test. First we request a Repository object from GitHub and then using that we request a Release object. After receiving that release, we exercise the code that lists the assets of a Release. We verify that each asset is an instance of the Asset class and that at the end the asset variable is not None. If asset was None, that would indicate that GitHub did not return any data and it did not exercise the code we are trying to test.

Betamax

Betamax is the library that we use to create the recorder above. It sets up the session object to intercept every request and corresponding response and save them to what it calls cassettes. After you record the interaction it never has to speak to the internet again for that request.

In github3.py there is a helper class (much like UnitHelper) in tests/integration/helper.py which sets everything up for us.

Example - Testing the Release Object

Here’s an example of how we write an integration test for github3.py. The example can be found in tests/integration/test_repos_release.py.

Our first steps are the necessary imports.

import github3

from .helper import IntegrationHelper

Then we start writing our test right away.

class TestRelease(IntegrationHelper):
    def test_delete(self):
        """Test the ability to delete a release."""
        self.token_login()
        cassette_name = self.cassette_name('delete')
        with self.recorder.use_cassette(cassette_name):
            repository = self.gh.repository('github3py', 'github3.py')
            release = repository.create_release(
                '0.8.0.pre', 'develop', '0.8.0 fake release',
                'To be deleted'
                )
            assert release is not None
            assert release.delete() is True

Every test has access to self.gh which is an instance of GitHub. IntegrationHelper provides a lot of methods that allow you to focus on what we are testing instead of setting up for the test. The first of those methods we see in use is self.token_login which handles authenticating with a token. It’s sister method is self.basic_login which handles authentication with basic credentials. Both of these methods will set up the authentication for you on self.gh.

The next convenience method we see is self.cassette_name. It constructs a cassette name for you based on the test class name and the string you provide it.

Every test also has access to self.recorder. This is the Betamax recorder that has been set up for you to record your interactions. The recorder is started when you write

with self.recorder.use_cassette(cassette_name):
    # ...

Everything that talks to GitHub should be written inside of the context created by the context manager there. No requests to GitHub should be made outside of that context.

In that context, we then retrieve a repository and create a release for it. We want to be sure that we will be deleting something that exists so we assert that what we received back from GitHub is not None. Finally we call delete and assert that it returns True.

When you write your new test and record a new cassette, be sure to add the new cassette file to the repository, like so:

git add tests/cassettes/Release_delete.json

Recording Cassettes that Require Authentication/Authorization

If you need to write a test that requires an Authorization (i.e., oAuth token) or Authentication (i.e., username and password), all you need to do is set environment variables when running py.test, e.g.,

GH_AUTH="abc123" py.test
GH_USER="sigmavirus24" GH_PASSWORD="super-secure-password-plz-kthxbai" py.test

If you are concerned that your credentials will be saved, you need not worry. Betamax sanitizes information like that before saving the cassette. It never does hurt to double check though.