Set up CI/CD

!83 (comment 12445)

• .gitlab-ci.yml documentation
• Blog article about new CI/CD features brought into GitLab v11 cycle

Build own Docker containers images with Poetry installed

• #245 (closed)

Pipeline schema

Checks

• Format check: Black black --check

Install + Build

• Install 3.{5,6,7,8}: poetry instal
• Build 3.7: poetry build

Tests

• #241 (closed): test_end_to_end.py: not able to run CLI tests as silkaj is not accessible from the shell in the Docker image. Using Click test utility would be a solution, but it is broken. Use poetry run silkaj sub-cmd for now.

Linux tests for all supported Python versions

• Multiple Python versions: 3.5, 3.6, 3.7, 3.8
  • Handle v3.5 which fails because of Black hack: Use 3.8 image with black installed, uninstall black from Poetry dev env.
  • v3.8: Poetry installation via pip
  • v3.8: libffi-dev and gcc missing for cffi dependency

Coverage

• Generation of the coverage website and badge are done in the main Python version image 3.7 and shared to other jobs through artifact
• Publish code coverage report with GitLab Pages
  • Links does not redirect to working pages: missing index.html
  • Page installation job move at the end of the pipeline to speed-up more import jobs as mandatory
• #146 (closed): Set coverage badge

Publication

• #105 (closed): Pypi distribution poetry publish:
  • Try with new PyPi tokens API
  • Pass secret variable not passed: the variable should match the criteria to be saved
  • Allow to pass the username of the account
  • PyPi Test
    • Create new account with password matching the criteria
    • git Warning: Remove .git, install git?
  • PyPi Stable:
    • Only on tag
    • Make the tag matches the semver regex
  • Check who has access to make actions on this pipeline: only developers + maintainers

Todo/Issues

• poetry shell does not activate the virtualenv: poetry run black
• Use extends instead of YAML anchors
• Docker images: install Poetry via pip. Could remove curl install. Problematic, if Poetry version changes. Install fixed version.
• Use Poetry pre-release: pip install poetry --pre?
• Get rid of doc/install_docker.md which is no longer needed.

Speed-up pipeline

• Run Format, Build, etc into Checks stage. Will it speed up the whole pipeline. Or, will the runners be overloaded? It doesn’t seems so.
  • Linear pipeline takes around 5-6 minutes.
  • Dropped to 4 minutes with PyPi test publication and all checks into Checks stage.
  • 2.5 minutes without PyPi publications
  • With four tests jobs for multiple Python version takes longer. As it seems only three jobs can be triggered all together. Python v3.8 job starts after one of the other are done. This slow down the pipeline to 4 minutes. Looks normal to what we add with previous state: checks and tests stages separated.
• Install manually just needed dependency with add/install: black
• 7 ways to speed up your GitLab CI/CD times
• How To Speed Up Your GitLab CI Pipeline
• Make your CI pipeline fast and awesome with Gitlab Container Registry

Caching

• Caching virtualenv between installation/build and tests images
• Caching of build (wheel, tarball) between build-3.7, and tests-3.7 jobs
• This speed-up the process, not to loose time re-installing, building.
• Cache data among jobs within the pipeline:
  • installed dependencies (including development ones)
  • targets (wheel and archive)
• silkaj accessible: poetry run bin/silkaj

Cache working only within the same Python version

• Install virtualenv into silkaj repository in order to cache it. Caching outside repository seems impossible
• virtualenv installed for a Python version is not compatible with other Python version:
  • Cache could be used for the main Python version images (3.7 today): build, test-3.7, PyPi releases.
  • cached virtualenv only working with the Python version it have been created. Because of the shared objects.
  • v3.7 could be kept for build, test-3.7, and publish jobs.
  • Other test-3.{5,6,8} jobs will have to install dependencies (including dev) for the test jobs.

Parallel jobs have issue sharing the same cache

• CI pipeline with parallel jobs sharing the same cache can result in an incomplete cache
• Cache not always restored on concurrent > 1 and docker executor

Pytest access

• Installed pytest not accessible:
  • cache not shared among jobs
  • Related to two upper issues, should be solved.

Poetry updates

• 1.0.0aX: New maintainer field in pyproject.toml.
• 1.0.0b1:
  • New settings management: local setting in pyproject.toml env variable: POETRY_*
  • settings.virtualenvs.in-project -> prefix commands with POETRY_VIRTUALENVS_IN_ROJECT
• 1.0.0b2:
  • Create configuration folder mkdir -p /root/.config/pypoetry/ and file could be removed

Other

• Try not to create virtualenv for Python v3.{5,6,8} to speed-up
  • Do not create virtualenv: bug, installation with --usercan we cache installed dependencies?

Later

• Checks:
  • #163 (closed): Type check
  • #292 (closed): Pylint
  • #318: Safety: Needs to be fixed

Test:

• Windows, Python v3.7 as suggested version from Python website, which Windows build? 3.7-windowsservercore-1809?
• macOS (not possible)