silkaj issueshttps://git.duniter.org/clients/python/silkaj/-/issues2020-02-11T18:31:41+01:00https://git.duniter.org/clients/python/silkaj/-/issues/287Use Black from Poetry2020-02-11T18:31:41+01:00MoulUse Black from PoetryThere is a difference between [Black version 18.9b0](https://packages.debian.org/search?suite=default&section=all&arch=any&searchon=names&keywords=black) emended into the docker container Debian Buster from the Debian package.
Using blac...There is a difference between [Black version 18.9b0](https://packages.debian.org/search?suite=default§ion=all&arch=any&searchon=names&keywords=black) emended into the docker container Debian Buster from the Debian package.
Using black from PyPi within Poetry environment shouldn’t rise such issues. In this case we would get 19.3b0 version.
This shortcut was set-up to speed-up the format job from 1min30 to 45s, to avoid installing runtime and dev dependencies.
- [ ] CI: Install dependencies, use black from Poetry environment
- [ ] Docker:3.8: Remove `black`0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/286Black version issue in Format job2020-02-13T11:10:16+01:00matograineBlack version issue in Format jobTo be confirmed : The version of Black used for pipelines (v18.9b0) seems to be incompatible with the one used in dev (v19.3b0).
If confirmed, the pipeline should use the `dev` version of black, installed with Poetry, not the one availa...To be confirmed : The version of Black used for pipelines (v18.9b0) seems to be incompatible with the one used in dev (v19.3b0).
If confirmed, the pipeline should use the `dev` version of black, installed with Poetry, not the one available in the Debian repos.0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/284Build and tests jobs are not retriggered in case of source code change2020-02-10T20:46:15+01:00MoulBuild and tests jobs are not retriggered in case of source code changeEncountered with !123.
Build and tests jobs are not triggered when a push-force, changing the source code, happen. Only the `format` job is retriggered.
This is due to the fact that the extension (`extends`:) is not inheriting but eras...Encountered with !123.
Build and tests jobs are not triggered when a push-force, changing the source code, happen. Only the `format` job is retriggered.
This is due to the fact that the extension (`extends`:) is not inheriting but erasing the `only:changes` part:
```yml
.code_changes:
only:
changes:
- silkaj/*.py
- tests/*.py
.changes:
extends: .code_changes
only:
changes:
- .gitlab-ci.yml
- pyproject.toml
- poetry.lock
```0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/249Install Poetry stable when v1 is released2020-07-02T17:44:04+02:00MoulInstall Poetry stable when v1 is releasedFollowing #149 and #245, switch back to Poetry stable:
- [x] in [Docker images](https://git.duniter.org/docker/python3/poetry/blob/master/3.5/Dockerfile#L9): https://git.duniter.org/docker/python3/poetry/commit/ac8190c6bd91243fa2c2336cdf...Following #149 and #245, switch back to Poetry stable:
- [x] in [Docker images](https://git.duniter.org/docker/python3/poetry/blob/master/3.5/Dockerfile#L9): https://git.duniter.org/docker/python3/poetry/commit/ac8190c6bd91243fa2c2336cdf28a7d1761c63bb
- [x] in [Poetry installation documentation](https://git.duniter.org/clients/python/silkaj/blob/dev/doc/install_poetry.md#install-poetry)0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/245Build own Docker containers images with Poetry installed2019-12-08T17:45:24+01:00MoulBuild own Docker containers images with Poetry installedIn order to set-up the CI/CD #149, Docker containers with Poetry installed should be created.
Having the maintenance over these images gives the control over futures behaviours.
## ~~Run directly without Poetry~~
Run directly on the D...In order to set-up the CI/CD #149, Docker containers with Poetry installed should be created.
Having the maintenance over these images gives the control over futures behaviours.
## ~~Run directly without Poetry~~
Run directly on the Docker image With Python.
Poetry is somehow using Python installed on the system.
`virtualenvs.create false`
## ~~Pre-built images~~
- https://pythonspeed.com/articles/base-image-python-docker-images/
- https://hub.docker.com/r/nikaro/poetry
- https://hub.docker.com/r/michalmazurek/python-poetry
## Own-built images
- Based on official Python images: [Docker hub](https://hub.docker.com/_/python?tab=description),
[git repository](https://github.com/docker-library/python)
- [x] Install `poetry`, and `libsodium`
- [x] Publish git repository here https://git.duniter.org/docker/python3/poetry
- [x] Python images with Slim Debian is enough
- [x] Buster + [Python version 3.5, 3.6, 3.7, and 3.8](https://hub.docker.com/_/python?tab=description)
- [x] Publish built images on `https://registry.duniter.org/docker/python3/poetry/$PYTHON_VERSION:{tag,latest}`
- https://about.gitlab.com/2016/05/23/gitlab-container-registry/
- https://docs.gitlab.com/ee/user/project/container_registry.html
- [ ] `change: - only $PYTHON_VERSION/Dockerfile`: not working
- [ ] Installing `pip install poetry --pre` fails on v3.8:
- installing `gcc` and `libffi-dev` by-passes the issue
- [ ] Redshift Runner fails building containers due to already existing docker image to build certain containers
- https://stackoverflow.com/questions/44257172/gitlab-ci-cannot-link-to-a-non-running-container
## Later
- [ ] Windows?
- [ ] Could be used by DuniterPy if it uses Poetry0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/149Set up CI/CD2020-06-01T11:56:18+02:00MoulSet up CI/CD> https://git.duniter.org/clients/python/silkaj/merge_requests/83#note_12445
- [`.gitlab-ci.yml` documentation](https://docs.gitlab.com/ee/ci/yaml/)
- [Blog article about new CI/CD features brought into GitLab v11 cycle](https://about.g...> https://git.duniter.org/clients/python/silkaj/merge_requests/83#note_12445
- [`.gitlab-ci.yml` documentation](https://docs.gitlab.com/ee/ci/yaml/)
- [Blog article about new CI/CD features brought into GitLab v11 cycle](https://about.gitlab.com/2019/06/19/look-back-on-11-11-cicd/)
## Build own Docker containers images with Poetry installed
- #245
## Pipeline schema
### Checks
- [x] Format check: Black `black --check`
### Install + Build
- [x] Install 3.{5,6,7,8}: `poetry instal`
- [x] Build 3.7: `poetry build`
#### Tests
- [x] #241: `test_end_to_end.py`: not able to run CLI tests as `silkaj` is not accessible from the shell in the Docker image. Using Click test utility would be a solution, but it is broken. Use `poetry run silkaj sub-cmd` for now.
##### Linux tests for all supported Python versions
- [x] Multiple Python versions: 3.5, 3.6, 3.7, 3.8
- [x] Handle v3.5 which fails because of Black hack: Use `3.8` image with `black` installed, uninstall `black` from Poetry dev env.
- [x] [v3.8: Poetry installation via `pip`](https://github.com/sdispater/poetry/issues/613)
- [x] v3.8: `libffi-dev` and `gcc` missing for `cffi` dependency
##### Coverage
- [x] Generation of the coverage website and badge are done in the main Python version image 3.7 and shared to other jobs through artifact
- [x] [Publish code coverage report with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/)
- [x] Links does not redirect to working pages: missing `index.html`
- [x] Page installation job move at the end of the pipeline to speed-up more import jobs as mandatory
- [x] #146: Set [coverage badge](https://github.com/dbrgn/coverage-badge)
### Publication
- [x] #105: Pypi distribution `poetry publish`:
- [ ] [Try with new PyPi tokens API](https://github.com/sdispater/poetry/pull/1275)
- [x] Pass secret variable not passed: the variable should match the criteria to be saved
- [x] Allow to pass the username of the account
- [x] PyPi Test
- [x] Create new account with password matching the criteria
- [x] [`git` Warning](https://github.com/sdispater/poetry/issues/614): Remove `.git`, install `git`?
- [x] PyPi Stable:
- [x] Only on `tag`
- ~~[Make the tag matches the semver regex](https://stackoverflow.com/questions/44834119/gitlab-only-build-for-specific-tag-names)~~
- [x] Check who has access to make actions on this pipeline: only developers + maintainers
## Todo/Issues
- [x] [`poetry shell` does not activate the `virtualenv`](https://github.com/sdispater/poetry/issues/571): `poetry run black`
- [x] Use [extends](https://docs.gitlab.com/ee/ci/yaml/#extends) instead of YAML anchors
- [x] Docker images: install Poetry via `pip`. Could remove `curl` install. Problematic, if Poetry version changes. Install fixed version.
- [x] Use Poetry pre-release: `pip install poetry --pre`?
- [x] Get rid of `doc/install_docker.md` which is no longer needed.
## Speed-up pipeline
- [x] Run `Format`, `Build`, etc into `Checks` stage. Will it speed up the whole pipeline. Or, will the runners be overloaded? It doesn’t seems so.
- Linear pipeline takes around 5-6 minutes.
- Dropped to 4 minutes with PyPi test publication and all checks into Checks stage.
- 2.5 minutes without PyPi publications
- With four tests jobs for multiple Python version takes longer. As it seems only three jobs can be triggered all together. Python v3.8 job starts after one of the other are done. This slow down the pipeline to 4 minutes. Looks normal to what we add with previous state: checks and tests stages separated.
- [x] Install manually just needed dependency with `add`/`install`: `black`
- [7 ways to speed up your GitLab CI/CD times](https://blog.sparksuite.com/7-ways-to-speed-up-gitlab-ci-cd-times-29f60aab69f9?gi=25ff201a0f4a)
- [How To Speed Up Your GitLab CI Pipeline](https://insight.full360.com/how-to-speed-up-your-gitlab-ci-pipeline-7af593eb0f4c?gi=9924e0739049)
- [Make your CI pipeline fast and awesome with Gitlab Container Registry](https://blog.daftcode.pl/make-your-ci-pipeline-fast-and-awesome-with-gitlab-container-registry-41ef244d1ce6?gi=8e24dae63eff)
### Caching
- Caching virtualenv between installation/build and tests images
- Caching of build (wheel, tarball) between build-3.7, and tests-3.7 jobs
- This speed-up the process, not to loose time re-installing, building.
- [Cache](https://docs.gitlab.com/ee/ci/yaml/#cache) data among jobs within the pipeline:
- installed dependencies (including development ones)
- targets (wheel and archive)
- [x] `silkaj` accessible: `poetry run bin/silkaj`
#### Cache working only within the same Python version
- [x] Install virtualenv into `silkaj` repository in order to cache it. [Caching outside repository seems impossible](https://gitlab.com/gitlab-org/gitlab-ce/issues/4431)
- [x] virtualenv installed for a Python version is not compatible with other Python version:
- Cache could be used for the main Python version images (3.7 today): build, test-3.7, PyPi releases.
- cached virtualenv only working with the Python version it have been created. Because of the [shared objects](https://git.duniter.org/clients/python/silkaj/-/jobs/26994).
- v3.7 could be kept for `build`, `test-3.7`, and `publish` jobs.
- Other `test-3.{5,6,8}` jobs will have to install dependencies (including dev) for the test jobs.
#### Parallel jobs have issue sharing the same cache
- [CI pipeline with parallel jobs sharing the same cache can result in an incomplete cache](https://gitlab.com/gitlab-org/gitlab-ce/issues/43980)
- [Cache not always restored on concurrent > 1 and docker executor](https://gitlab.com/gitlab-org/gitlab-runner/issues/1151)
#### Pytest access
- [ ] Installed `pytest` not accessible:
- cache not shared among jobs
- Related to two upper issues, should be solved.
#### Poetry updates
- 1.0.0aX:
New `maintainer` field in `pyproject.toml`.
- 1.0.0b1:
- [New settings management](https://github.com/sdispater/poetry/pull/1272): local setting in `pyproject.toml` env variable: `POETRY_*`
- `settings.virtualenvs.in-project` -> prefix commands with `POETRY_VIRTUALENVS_IN_ROJECT`
- 1.0.0b2:
- [Create configuration folder `mkdir -p /root/.config/pypoetry/` and file could be removed](https://github.com/sdispater/poetry/issues/1179)
#### Other
- Try not to create virtualenv for Python v3.{5,6,8} to speed-up
- Do not create virtualenv: [bug](https://github.com/sdispater/poetry/issues/1265), [installation with `--user`](https://github.com/sdispater/poetry/issues/1214)can we cache installed dependencies?
## Later
- Checks:
- [ ] #163: Type check
- [ ] #292: Pylint
- [ ] #318: [Safety](https://github.com/pyupio/safety): [Needs to be fixed](https://github.com/pyupio/safety/issues/201)
Test:
- [ ] Windows, Python v3.7 as suggested version from Python website, which Windows build? `3.7-windowsservercore-1809`?
- macOS ([not](https://stackoverflow.com/questions/40824547/docker-image-running-a-mac-os-x-installation) [possible](https://serverfault.com/questions/607443/can-mac-os-x-be-run-inside-docker))0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/146Add a coverage badge2021-02-21T18:11:48+01:00Sébastien DA ROCHAAdd a coverage badgeNow that the test structure is (nearly) in place, we can test the coverage rate (which is quite bad at the moment) and add a badge on the README page.
I see 3 options:
* generate the badge and store it somewhere. I've tried to do this ...Now that the test structure is (nearly) in place, we can test the coverage rate (which is quite bad at the moment) and add a badge on the README page.
I see 3 options:
* generate the badge and store it somewhere. I've tried to do this and store it on the git repo, the code is in coverage_badges branch of my fork. The problem is that I can't load the image trough gitlab (Content-type is always text for security issues), so the image needs to be copied elsewhere (where ?).
* use a service like coveralls/shields.io, but it relies on a external service (possibly closed source)
* use the badge generation from gitlab, but we haven't setup the CI yet :thinking:
I'd rather follow the last option, @Moul what do you think?0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/105Deploy on PyPI from GitLab CD2020-06-24T15:36:19+02:00MoulDeploy on PyPI from GitLab CD> Cf: https://git.duniter.org/clients/python/silkaj/merge_requests/80#note_11085
- [Using Gitlab pipelines to deploy Python packages in production and staging environments](https://medium.com/@pypripackages/using-gitlab-pipelines-to-dep...> Cf: https://git.duniter.org/clients/python/silkaj/merge_requests/80#note_11085
- [Using Gitlab pipelines to deploy Python packages in production and staging environments](https://medium.com/@pypripackages/using-gitlab-pipelines-to-deploy-python-packages-in-production-and-staging-environments-8ab7dc979274)
> Linked to #149.0.8.0MoulMoul