> https://git.duniter.org/clients/python/silkaj/merge_requests/83#note_12445 - [`.gitlab-ci.yml` documentation](https://docs.gitlab.com/ee/ci/yaml/) - [Blog article about new CI/CD features brought into GitLab v11 cycle](https://about.gitlab.com/2019/06/19/look-back-on-11-11-cicd/) ## Build own Docker containers images with Poetry installed - #245 ## Pipeline schema ### Checks - [x] Format check: Black `black --check` ### Install + Build - [x] Install 3.{5,6,7,8}: `poetry instal` - [x] Build 3.7: `poetry build` #### Tests - [x] #241: `test_end_to_end.py`: not able to run CLI tests as `silkaj` is not accessible from the shell in the Docker image. Using Click test utility would be a solution, but it is broken. Use `poetry run silkaj sub-cmd` for now. ##### Linux tests for all supported Python versions - [x] Multiple Python versions: 3.5, 3.6, 3.7, 3.8 - [x] Handle v3.5 which fails because of Black hack: Use `3.8` image with `black` installed, uninstall `black` from Poetry dev env. - [x] [v3.8: Poetry installation via `pip`](https://github.com/sdispater/poetry/issues/613) - [x] v3.8: `libffi-dev` and `gcc` missing for `cffi` dependency ##### Coverage - [x] Generation of the coverage website and badge are done in the main Python version image 3.7 and shared to other jobs through artifact - [x] [Publish code coverage report with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/) - [x] Links does not redirect to working pages: missing `index.html` - [x] Page installation job move at the end of the pipeline to speed-up more import jobs as mandatory - [x] #146: Set [coverage badge](https://github.com/dbrgn/coverage-badge) ### Publication - [x] #105: Pypi distribution `poetry publish`: - [ ] [Try with new PyPi tokens API](https://github.com/sdispater/poetry/pull/1275) - [x] Pass secret variable not passed: the variable should match the criteria to be saved - [x] Allow to pass the username of the account - [x] PyPi Test - [x] Create new account with password matching the criteria - [x] [`git` Warning](https://github.com/sdispater/poetry/issues/614): Remove `.git`, install `git`? - [x] PyPi Stable: - [x] Only on `tag` - ~~[Make the tag matches the semver regex](https://stackoverflow.com/questions/44834119/gitlab-only-build-for-specific-tag-names)~~ - [x] Check who has access to make actions on this pipeline: only developers + maintainers ## Todo/Issues - [x] [`poetry shell` does not activate the `virtualenv`](https://github.com/sdispater/poetry/issues/571): `poetry run black` - [x] Use [extends](https://docs.gitlab.com/ee/ci/yaml/#extends) instead of YAML anchors - [x] Docker images: install Poetry via `pip`. Could remove `curl` install. Problematic, if Poetry version changes. Install fixed version. - [x] Use Poetry pre-release: `pip install poetry --pre`? - [x] Get rid of `doc/install_docker.md` which is no longer needed. ## Speed-up pipeline - [x] Run `Format`, `Build`, etc into `Checks` stage. Will it speed up the whole pipeline. Or, will the runners be overloaded? It doesn’t seems so. - Linear pipeline takes around 5-6 minutes. - Dropped to 4 minutes with PyPi test publication and all checks into Checks stage. - 2.5 minutes without PyPi publications - With four tests jobs for multiple Python version takes longer. As it seems only three jobs can be triggered all together. Python v3.8 job starts after one of the other are done. This slow down the pipeline to 4 minutes. Looks normal to what we add with previous state: checks and tests stages separated. - [x] Install manually just needed dependency with `add`/`install`: `black` - [7 ways to speed up your GitLab CI/CD times](https://blog.sparksuite.com/7-ways-to-speed-up-gitlab-ci-cd-times-29f60aab69f9?gi=25ff201a0f4a) - [How To Speed Up Your GitLab CI Pipeline](https://insight.full360.com/how-to-speed-up-your-gitlab-ci-pipeline-7af593eb0f4c?gi=9924e0739049) - [Make your CI pipeline fast and awesome with Gitlab Container Registry](https://blog.daftcode.pl/make-your-ci-pipeline-fast-and-awesome-with-gitlab-container-registry-41ef244d1ce6?gi=8e24dae63eff) ### Caching - Caching virtualenv between installation/build and tests images - Caching of build (wheel, tarball) between build-3.7, and tests-3.7 jobs - This speed-up the process, not to loose time re-installing, building. - [Cache](https://docs.gitlab.com/ee/ci/yaml/#cache) data among jobs within the pipeline: - installed dependencies (including development ones) - targets (wheel and archive) - [x] `silkaj` accessible: `poetry run bin/silkaj` #### Cache working only within the same Python version - [x] Install virtualenv into `silkaj` repository in order to cache it. [Caching outside repository seems impossible](https://gitlab.com/gitlab-org/gitlab-ce/issues/4431) - [x] virtualenv installed for a Python version is not compatible with other Python version: - Cache could be used for the main Python version images (3.7 today): build, test-3.7, PyPi releases. - cached virtualenv only working with the Python version it have been created. Because of the [shared objects](https://git.duniter.org/clients/python/silkaj/-/jobs/26994). - v3.7 could be kept for `build`, `test-3.7`, and `publish` jobs. - Other `test-3.{5,6,8}` jobs will have to install dependencies (including dev) for the test jobs. #### Parallel jobs have issue sharing the same cache - [CI pipeline with parallel jobs sharing the same cache can result in an incomplete cache](https://gitlab.com/gitlab-org/gitlab-ce/issues/43980) - [Cache not always restored on concurrent > 1 and docker executor](https://gitlab.com/gitlab-org/gitlab-runner/issues/1151) #### Pytest access - [ ] Installed `pytest` not accessible: - cache not shared among jobs - Related to two upper issues, should be solved. #### Poetry updates - 1.0.0aX: New `maintainer` field in `pyproject.toml`. - 1.0.0b1: - [New settings management](https://github.com/sdispater/poetry/pull/1272): local setting in `pyproject.toml` env variable: `POETRY_*` - `settings.virtualenvs.in-project` -> prefix commands with `POETRY_VIRTUALENVS_IN_ROJECT` - 1.0.0b2: - [Create configuration folder `mkdir -p /root/.config/pypoetry/` and file could be removed](https://github.com/sdispater/poetry/issues/1179) #### Other - Try not to create virtualenv for Python v3.{5,6,8} to speed-up - Do not create virtualenv: [bug](https://github.com/sdispater/poetry/issues/1265), [installation with `--user`](https://github.com/sdispater/poetry/issues/1214)can we cache installed dependencies? ## Later - Checks: - [ ] #163: Type check - [ ] #292: Pylint - [ ] #318: [Safety](https://github.com/pyupio/safety): [Needs to be fixed](https://github.com/pyupio/safety/issues/201) Test: - [ ] Windows, Python v3.7 as suggested version from Python website, which Windows build? `3.7-windowsservercore-1809`? - macOS ([not](https://stackoverflow.com/questions/40824547/docker-image-running-a-mac-os-x-installation) [possible](https://serverfault.com/questions/607443/can-mac-os-x-be-run-inside-docker))