silkaj issueshttps://git.duniter.org/clients/python/silkaj/-/issues2019-08-01T16:29:25+02:00https://git.duniter.org/clients/python/silkaj/-/issues/85Create test struture2019-08-01T16:29:25+02:00MoulCreate test struture- https://docs.python.org/3/library/unittest.html- https://docs.python.org/3/library/unittest.html0.7.0Sébastien DA ROCHASébastien DA ROCHAhttps://git.duniter.org/clients/python/silkaj/-/issues/146Add a coverage badge2021-02-21T18:11:48+01:00Sébastien DA ROCHAAdd a coverage badgeNow that the test structure is (nearly) in place, we can test the coverage rate (which is quite bad at the moment) and add a badge on the README page.
I see 3 options:
* generate the badge and store it somewhere. I've tried to do this ...Now that the test structure is (nearly) in place, we can test the coverage rate (which is quite bad at the moment) and add a badge on the README page.
I see 3 options:
* generate the badge and store it somewhere. I've tried to do this and store it on the git repo, the code is in coverage_badges branch of my fork. The problem is that I can't load the image trough gitlab (Content-type is always text for security issues), so the image needs to be copied elsewhere (where ?).
* use a service like coveralls/shields.io, but it relies on a external service (possibly closed source)
* use the badge generation from gitlab, but we haven't setup the CI yet :thinking:
I'd rather follow the last option, @Moul what do you think?0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/149Set up CI/CD2020-06-01T11:56:18+02:00MoulSet up CI/CD> https://git.duniter.org/clients/python/silkaj/merge_requests/83#note_12445
- [`.gitlab-ci.yml` documentation](https://docs.gitlab.com/ee/ci/yaml/)
- [Blog article about new CI/CD features brought into GitLab v11 cycle](https://about.g...> https://git.duniter.org/clients/python/silkaj/merge_requests/83#note_12445
- [`.gitlab-ci.yml` documentation](https://docs.gitlab.com/ee/ci/yaml/)
- [Blog article about new CI/CD features brought into GitLab v11 cycle](https://about.gitlab.com/2019/06/19/look-back-on-11-11-cicd/)
## Build own Docker containers images with Poetry installed
- #245
## Pipeline schema
### Checks
- [x] Format check: Black `black --check`
### Install + Build
- [x] Install 3.{5,6,7,8}: `poetry instal`
- [x] Build 3.7: `poetry build`
#### Tests
- [x] #241: `test_end_to_end.py`: not able to run CLI tests as `silkaj` is not accessible from the shell in the Docker image. Using Click test utility would be a solution, but it is broken. Use `poetry run silkaj sub-cmd` for now.
##### Linux tests for all supported Python versions
- [x] Multiple Python versions: 3.5, 3.6, 3.7, 3.8
- [x] Handle v3.5 which fails because of Black hack: Use `3.8` image with `black` installed, uninstall `black` from Poetry dev env.
- [x] [v3.8: Poetry installation via `pip`](https://github.com/sdispater/poetry/issues/613)
- [x] v3.8: `libffi-dev` and `gcc` missing for `cffi` dependency
##### Coverage
- [x] Generation of the coverage website and badge are done in the main Python version image 3.7 and shared to other jobs through artifact
- [x] [Publish code coverage report with GitLab Pages](https://about.gitlab.com/2016/11/03/publish-code-coverage-report-with-gitlab-pages/)
- [x] Links does not redirect to working pages: missing `index.html`
- [x] Page installation job move at the end of the pipeline to speed-up more import jobs as mandatory
- [x] #146: Set [coverage badge](https://github.com/dbrgn/coverage-badge)
### Publication
- [x] #105: Pypi distribution `poetry publish`:
- [ ] [Try with new PyPi tokens API](https://github.com/sdispater/poetry/pull/1275)
- [x] Pass secret variable not passed: the variable should match the criteria to be saved
- [x] Allow to pass the username of the account
- [x] PyPi Test
- [x] Create new account with password matching the criteria
- [x] [`git` Warning](https://github.com/sdispater/poetry/issues/614): Remove `.git`, install `git`?
- [x] PyPi Stable:
- [x] Only on `tag`
- ~~[Make the tag matches the semver regex](https://stackoverflow.com/questions/44834119/gitlab-only-build-for-specific-tag-names)~~
- [x] Check who has access to make actions on this pipeline: only developers + maintainers
## Todo/Issues
- [x] [`poetry shell` does not activate the `virtualenv`](https://github.com/sdispater/poetry/issues/571): `poetry run black`
- [x] Use [extends](https://docs.gitlab.com/ee/ci/yaml/#extends) instead of YAML anchors
- [x] Docker images: install Poetry via `pip`. Could remove `curl` install. Problematic, if Poetry version changes. Install fixed version.
- [x] Use Poetry pre-release: `pip install poetry --pre`?
- [x] Get rid of `doc/install_docker.md` which is no longer needed.
## Speed-up pipeline
- [x] Run `Format`, `Build`, etc into `Checks` stage. Will it speed up the whole pipeline. Or, will the runners be overloaded? It doesn’t seems so.
- Linear pipeline takes around 5-6 minutes.
- Dropped to 4 minutes with PyPi test publication and all checks into Checks stage.
- 2.5 minutes without PyPi publications
- With four tests jobs for multiple Python version takes longer. As it seems only three jobs can be triggered all together. Python v3.8 job starts after one of the other are done. This slow down the pipeline to 4 minutes. Looks normal to what we add with previous state: checks and tests stages separated.
- [x] Install manually just needed dependency with `add`/`install`: `black`
- [7 ways to speed up your GitLab CI/CD times](https://blog.sparksuite.com/7-ways-to-speed-up-gitlab-ci-cd-times-29f60aab69f9?gi=25ff201a0f4a)
- [How To Speed Up Your GitLab CI Pipeline](https://insight.full360.com/how-to-speed-up-your-gitlab-ci-pipeline-7af593eb0f4c?gi=9924e0739049)
- [Make your CI pipeline fast and awesome with Gitlab Container Registry](https://blog.daftcode.pl/make-your-ci-pipeline-fast-and-awesome-with-gitlab-container-registry-41ef244d1ce6?gi=8e24dae63eff)
### Caching
- Caching virtualenv between installation/build and tests images
- Caching of build (wheel, tarball) between build-3.7, and tests-3.7 jobs
- This speed-up the process, not to loose time re-installing, building.
- [Cache](https://docs.gitlab.com/ee/ci/yaml/#cache) data among jobs within the pipeline:
- installed dependencies (including development ones)
- targets (wheel and archive)
- [x] `silkaj` accessible: `poetry run bin/silkaj`
#### Cache working only within the same Python version
- [x] Install virtualenv into `silkaj` repository in order to cache it. [Caching outside repository seems impossible](https://gitlab.com/gitlab-org/gitlab-ce/issues/4431)
- [x] virtualenv installed for a Python version is not compatible with other Python version:
- Cache could be used for the main Python version images (3.7 today): build, test-3.7, PyPi releases.
- cached virtualenv only working with the Python version it have been created. Because of the [shared objects](https://git.duniter.org/clients/python/silkaj/-/jobs/26994).
- v3.7 could be kept for `build`, `test-3.7`, and `publish` jobs.
- Other `test-3.{5,6,8}` jobs will have to install dependencies (including dev) for the test jobs.
#### Parallel jobs have issue sharing the same cache
- [CI pipeline with parallel jobs sharing the same cache can result in an incomplete cache](https://gitlab.com/gitlab-org/gitlab-ce/issues/43980)
- [Cache not always restored on concurrent > 1 and docker executor](https://gitlab.com/gitlab-org/gitlab-runner/issues/1151)
#### Pytest access
- [ ] Installed `pytest` not accessible:
- cache not shared among jobs
- Related to two upper issues, should be solved.
#### Poetry updates
- 1.0.0aX:
New `maintainer` field in `pyproject.toml`.
- 1.0.0b1:
- [New settings management](https://github.com/sdispater/poetry/pull/1272): local setting in `pyproject.toml` env variable: `POETRY_*`
- `settings.virtualenvs.in-project` -> prefix commands with `POETRY_VIRTUALENVS_IN_ROJECT`
- 1.0.0b2:
- [Create configuration folder `mkdir -p /root/.config/pypoetry/` and file could be removed](https://github.com/sdispater/poetry/issues/1179)
#### Other
- Try not to create virtualenv for Python v3.{5,6,8} to speed-up
- Do not create virtualenv: [bug](https://github.com/sdispater/poetry/issues/1265), [installation with `--user`](https://github.com/sdispater/poetry/issues/1214)can we cache installed dependencies?
## Later
- Checks:
- [ ] #163: Type check
- [ ] #292: Pylint
- [ ] #318: [Safety](https://github.com/pyupio/safety): [Needs to be fixed](https://github.com/pyupio/safety/issues/201)
Test:
- [ ] Windows, Python v3.7 as suggested version from Python website, which Windows build? `3.7-windowsservercore-1809`?
- macOS ([not](https://stackoverflow.com/questions/40824547/docker-image-running-a-mac-os-x-installation) [possible](https://serverfault.com/questions/607443/can-mac-os-x-be-run-inside-docker))0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/213Write unit tests for the transaction command2021-03-22T17:05:18+01:00MoulWrite unit tests for the transaction commandTransaction is the most critical part of Silkaj. It should be covered with tests.
- [x] simple tx
- [x] intermediaries txs
- [x] multi-output
- [x] #235: one of `amount`, `amountUD`, `allSources` option should be specified
- [ ] many ot...Transaction is the most critical part of Silkaj. It should be covered with tests.
- [x] simple tx
- [x] intermediaries txs
- [x] multi-output
- [x] #235: one of `amount`, `amountUD`, `allSources` option should be specified
- [ ] many other small cases
- [ ] …0.9.0matograinematograinehttps://git.duniter.org/clients/python/silkaj/-/issues/223Make Click context optional to be able to call functions from an external module2021-06-07T21:07:41+02:00MoulMake Click context optional to be able to call functions from an external module## Problem
Since Click migration (v0.7.0), It is not possible to use a function which internally make a request. For external usage and tests without Click is not possible.
`EndPoint` class is coupled with Click context `pass_context` f...## Problem
Since Click migration (v0.7.0), It is not possible to use a function which internally make a request. For external usage and tests without Click is not possible.
`EndPoint` class is coupled with Click context `pass_context` for custom endpoint or GTEST usage for BMA api string generation.
## Would allow
- Write tests which requires requests is not possible (most of the tests)
- @tuxmain to use the transaction functions for [ĞMixer](https://git.duniter.org/tuxmain/gmixer-py) and other projects to take advantages of Silkaj code.
## Solution
In `EndPoint` class:
- Use `RuntimeError` to detect that there is no Click context in case Click is installed and loaded
- Use [`ModuleNotFoundError`](https://docs.python.org/3/library/exceptions.html#ModuleNotFoundError) in case Click is not installed
- Previous point requires Python ≥ 3.6, and I didn’t found a way to import it with `__future__` for Python 3.5 support.
- To keep Python 3.5 support, I can just check for general exception for now. And, when Silkaj will drop Python 3.5, we could add back `ModuleNotFoundError` exception.
## Test
- [x] Test `EndPoint().BMA_ENDPOINT` with click context
- [x] Test `EndPoint().BMA_ENDPOINT` without click context
- [ ] [Figure out how to test a singleton function](https://stackoverflow.com/questions/43619748/destroying-a-singleton-object-in-python/43620075#43620075)
- [x] Test with Python 3.5: Not found how to simulate missing module
## After
- [ ] Write `test_tx_history_generate_table()` test.0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/225Install pytest-asyncio2020-07-02T17:41:44+02:00MoulInstall pytest-asyncioSince Silkaj code is asynchronous, asynchronous tests should be handled.
Install [pytest-asyncio](https://pypi.org/project/pytest-asyncio/)
> #187 depends on this ticket.Since Silkaj code is asynchronous, asynchronous tests should be handled.
Install [pytest-asyncio](https://pypi.org/project/pytest-asyncio/)
> #187 depends on this ticket.0.7.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/235Rework to check that only one option is passed to retrieve the amount of the ...2020-02-24T19:13:20+01:00MoulRework to check that only one option is passed to retrieve the amount of the transaction> Sub-task of #213
### Way to restrict amount, amountUD, allSources
- https://gist.github.com/jacobtolar/fb80d5552a9a9dfc32b12a829fa21c0c
- https://stackoverflow.com/questions/37310718/mutually-exclusive-option-groups-in-python-click
-...> Sub-task of #213
### Way to restrict amount, amountUD, allSources
- https://gist.github.com/jacobtolar/fb80d5552a9a9dfc32b12a829fa21c0c
- https://stackoverflow.com/questions/37310718/mutually-exclusive-option-groups-in-python-click
- https://github.com/pallets/click/issues/257
- https://github.com/pallets/click/issues/1137
### Write tests
- [x] tests0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/241Can not run test with Click utility2019-09-14T12:03:35+02:00MoulCan not run test with Click utilityWe are currently using `subprocess.check_output` for testing the CLI. It does work fine.
To set-up the CI, `subprocess` is calling to `silkaj` from the shell PATH. As it is not defined in the Docker image, it does not work. For, now, I ...We are currently using `subprocess.check_output` for testing the CLI. It does work fine.
To set-up the CI, `subprocess` is calling to `silkaj` from the shell PATH. As it is not defined in the Docker image, it does not work. For, now, I propose to call `poetry run silkaj sub-cmd` to by-pass next issue.
But if we want to use [Click helper to test the CLI](https://click.palletsprojects.com/en/7.x/testing/#basic-testing), for extra features such as tests on the terminal size, it does not work.
It is due to the `aiohttp` session which gets closed by the first test closing the session. Then other tests get this closed session because of the singleton. It looks like Click test utility does not restart from a clean context.0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/257Change txs lost while sending big amount2021-03-08T20:51:55+01:00matograineChange txs lost while sending big amountWhile sending a big amount that need lots of change txs, some change txs are lost and the amount cannot be sent.
When sending many txs, sometimes the connection with the node is lost. We should check that change operations are really se...While sending a big amount that need lots of change txs, some change txs are lost and the amount cannot be sent.
When sending many txs, sometimes the connection with the node is lost. We should check that change operations are really sent before sending the final amount.0.9.0https://git.duniter.org/clients/python/silkaj/-/issues/274Improve PubSec authfile formats handling, test, document2023-06-02T12:07:41+02:00matograineImprove PubSec authfile formats handling, test, documentFollowing #273 and #278:
- [ ] check compatibility with Cesium's and Duniter's PubSec format
- [ ] Add PubSec compatibility to the regex in DuniterPy
- [ ] Write tests for PubSec authfile formats into DuniterPy
- [ ] Import DuniterPy re...Following #273 and #278:
- [ ] check compatibility with Cesium's and Duniter's PubSec format
- [ ] Add PubSec compatibility to the regex in DuniterPy
- [ ] Write tests for PubSec authfile formats into DuniterPy
- [ ] Import DuniterPy regex for filtering the authfiles in Silkaj
---
- [ ] #271: document accepted authfile formatsBackloghttps://git.duniter.org/clients/python/silkaj/-/issues/282Split patched.py into files2021-03-22T17:12:03+01:00matograineSplit patched.py into filesFor lisibility on unit tests, we should split tests/patched.py into a directory :
/tests/patched/money.py
/tests/patched/tui.py
/tests/patched/tx.py
usw...For lisibility on unit tests, we should split tests/patched.py into a directory :
/tests/patched/money.py
/tests/patched/tui.py
/tests/patched/tx.py
usw...0.9.0matograinematograinehttps://git.duniter.org/clients/python/silkaj/-/issues/285Add copyright header in tests source files2020-06-14T21:01:40+02:00MoulAdd copyright header in tests source filesThe tests are starting to become important as the project is starting to implement a quite good amount of tests.
I do not know if this is a good practice to add the copyright header statement in the test source files.
But, I assume thi...The tests are starting to become important as the project is starting to implement a quite good amount of tests.
I do not know if this is a good practice to add the copyright header statement in the test source files.
But, I assume this is still some code which validate, lock the runtime code, which is part of the project with copyright on it.
This is surely a prerequisite for Debian packaging, since it was necessary to add this headers in the runtime code. At that time, there were almost no tests. Today, there is.0.8.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/307Create tests_tui.py file2020-10-23T16:51:58+02:00matograineCreate tests_tui.py fileTests for tui.py are in test_unit_tx.py file and should be moved to test_tui.pyTests for tui.py are in test_unit_tx.py file and should be moved to test_tui.py0.8.0matograinematograinehttps://git.duniter.org/clients/python/silkaj/-/issues/326Integrate coverage in GitLab coverage feature2022-09-25T18:31:04+02:00MoulIntegrate coverage in GitLab coverage feature### Todo
- [x] The [coverage can reports to the terminal and as htmls altogether](https://pytest-cov.readthedocs.io/en/latest/reporting.html):
```
poetry run pytest --cov silkaj --cov-report html:cov_html --cov-report term
```
- Switch ...### Todo
- [x] The [coverage can reports to the terminal and as htmls altogether](https://pytest-cov.readthedocs.io/en/latest/reporting.html):
```
poetry run pytest --cov silkaj --cov-report html:cov_html --cov-report term
```
- Switch from `pytest-cov` to [`coverage`](https://coverage.readthedocs.io/) dependency?:
```bash
poetry run coverage run -m pytest --cov silkaj --cov-report html:cov_html --cov-report term
```
- [x] Then, [specify `coverage` regex](https://forum.gitlab.com/t/python-coverage-always-unknown/47819/3) in [the configuration](https://git.duniter.org/clients/python/silkaj/-/settings/ci_cd#js-general-pipeline-settings) to find it in the logs of the job, to generate the badge and a [chart of the coverage evolution, which is available from GitLab v13.1](https://about.gitlab.com/releases/2020/06/22/gitlab-13-1-released/#graph-code-coverage-changes-over-time-for-a-project).
- [x] Remove now useless `coverage-badge` dependency introduced in #146.
- [x] Update badge in the readme and the repository configuration0.10.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/330Repository restructuration2023-06-29T15:34:15+02:00MoulRepository restructuration- A command per file
- Tools in a file
- `silkaj` and `tests` folders should look similar.
---
- `silkaj`
- [x] `blockchain/`:
- `info` (`argos`)
- `difficulty`
- `blocks`
- `verify`
- `tools`
- [x] `money/`:
...- A command per file
- Tools in a file
- `silkaj` and `tests` folders should look similar.
---
- `silkaj`
- [x] `blockchain/`:
- `info` (`argos`)
- `difficulty`
- `blocks`
- `verify`
- `tools`
- [x] `money/`:
- `transfer`
- `balance`
- `history`
- `tools`
- [x] `wot/`:
- `lookup`
- `wot`
- `certification`
- `membership`
- [x] `revocation`: Failing when `test_transfer::test_generate_and_send_transaction` runs first. `network.send_document()` patching issue?
- Used to work before when they use to run in the other order: `test_revocation/test_tx` -> `wot.test_revocation`/`money.test_transaction`
- Solutions: reverting back the order, or skipping `test_generate_and_send_transaction` which anyway will be removed in %"0.20.0". Fixing the actual issue (doesn’t make sense to fix v1/BMA issue now)
- `tools`
- `ui/`?:
- `tui`
- potential `gui`?
- `auth` -> `authentication`?
- Move `cli_tools` content to `tools`
- [x] `about`
- `checksum`
- [x] `crypto_tools` -> `public_key`, because `pubkey` is commonly used variable
- `constants`
- `g1_monetary_license`
- [x] `network_tools` -> `network`
- `tests`: With the same structure as the sources0.11.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/335Merge the two functions testing transaction_amount()2021-05-13T14:58:08+02:00MoulMerge the two functions testing transaction_amount()Two functions are testing `transaction_amount()` in `test_{,unit_}tx.py`:
- Merge the functions? at least put them together
- Move `test_unit_tx.py` content in `test_tx.py`Two functions are testing `transaction_amount()` in `test_{,unit_}tx.py`:
- Merge the functions? at least put them together
- Move `test_unit_tx.py` content in `test_tx.py`0.9.0matograinematograinehttps://git.duniter.org/clients/python/silkaj/-/issues/337forbid any BMA call during tests2020-09-06T14:23:39+02:00matograineforbid any BMA call during testsIt seems that opening and closing a BMA session during tests (pytest) forbids any further opening of any BMA session.
As a consequence, I think it should be written in CONTRIBUTING.md that any opening/closing of BMA sessions in tests is...It seems that opening and closing a BMA session during tests (pytest) forbids any further opening of any BMA session.
As a consequence, I think it should be written in CONTRIBUTING.md that any opening/closing of BMA sessions in tests is forbidden, and that related functions should be patched.0.9.0https://git.duniter.org/clients/python/silkaj/-/issues/347Write checksum tests on every commands2023-06-02T11:53:42+02:00MoulWrite checksum tests on every commandsThe following discussion from !143 should be addressed:
- [ ] @moul started a [discussion](https://git.duniter.org/clients/python/silkaj/-/merge_requests/143#note_23839):
> Thanks for writing these tests!
>
> The best wou...The following discussion from !143 should be addressed:
- [ ] @moul started a [discussion](https://git.duniter.org/clients/python/silkaj/-/merge_requests/143#note_23839):
> Thanks for writing these tests!
>
> The best would have been to have test written for all command to test that we can pass an uid, a pubkey, pubkey+checksum. That the checksums are validated. I am afraid we don't have time for that. It should be planned for next release.
---
In the case we want to continue using the `<pubkey>:<checksum>` checksum system, and not using #341 `base58.b58(de|en)code_check()`, writing checksum tests on every commands should be done:
- Passing a pubkey
- Passing a pubkey + valid checksum
- Passing a pubkey + invalid checksum
- Passing an `uid` to `wot`, `cert`, …Backloghttps://git.duniter.org/clients/python/silkaj/-/issues/351Delete `blockchain verify` command2023-10-21T09:05:15+02:00MoulDelete `blockchain verify` commandOriginal title: tests are sometimes failing due to network outage
https://git.duniter.org/clients/python/silkaj/-/jobs/42791
```
FAILED tests/test_verify_blocks.py::test_get_chunks[2-1] - TypeError: 'NoneTy...
FAILED tests/test_verify_...Original title: tests are sometimes failing due to network outage
https://git.duniter.org/clients/python/silkaj/-/jobs/42791
```
FAILED tests/test_verify_blocks.py::test_get_chunks[2-1] - TypeError: 'NoneTy...
FAILED tests/test_verify_blocks.py::test_get_chunks[5-10] - TypeError: 'NoneT...
```0.12.0MoulMoulhttps://git.duniter.org/clients/python/silkaj/-/issues/355Set Py3.8 as default image and move coverage to Py3.92021-06-07T18:53:18+02:00MoulSet Py3.8 as default image and move coverage to Py3.9- [x] Set Py 3.8 as default image since tools are installed on
- [x] Move the coverage and the badge generations to v3.9 test job.- [x] Set Py 3.8 as default image since tools are installed on
- [x] Move the coverage and the badge generations to v3.9 test job.0.10.0MoulMoul