Tests are loosely coupled in categories. Both tests and categories
have anchors and can be linked to. The anchors should be stable
enough to be included in commit messages and documentation. Every
test describes the setup, and may introduce terminology used in the
test results. Additional resources (e.g. certificates) required by
the test can be inspected by clicking on the inspect button
(). The results are in tabular form. The
producers are on the left going down, the consumers on the top going
right.
There are two kinds of tests. In producer-consumer tests, the
OpenPGP implementations being tested produce an artifact (e.g. they
encrypt a message), and every implementation is used to consume the
artifact (e.g. to decrypt the encrypted message). In consumer
tests, the artifacts are produced by the test suite, and consumed by
every OpenPGP implementation. In either case, the artifact that is
consumed can be inspected by clicking on the inspect button
() in the second column in every row. If
a producer failed to produce an artifact, or the artifact did not
conform to the expectation, a cross mark (โ) is displayed. Hovering
over it with the mouse pointer reveals the error message in a
tooltip.
Each row now contains the result of consuming the row's artifact
using the different OpenPGP implementations. Here, a check mark
(โ) indicates that the operation was successful. The resulting
output (e.g. the decrypted message) can be found in the tooltip.
Like before, a cross mark (โ) indicates that the operation was
not successful, or the produced artifact did not meet
expectations. Again, details can be found in the tooltip.
If an operation is not supported by an implementation, the result is
marked as failing with a cross mark (โ) with a light-yellow
background and a diagonal line in the bottom-left corner. Details
can be found in the tooltip.
If an operation is not done correctly by an implementation, i.e. the
operation returned success, some data was produced, but the test
suite found an issue with the produced data, or the operation failed
in any other way (say communication with the backend failed), the
result is marked as failing with a cross mark (โ) with a light-red
background and a diagonal line in the bottom-right corner. Details
can be found in the tooltip.
Up to this point, we did not judge whether or not a operation
should be successful or not, we merely recorded the facts. This
answers the question of how implementations react to certain
inputs, and we can quantify that and have an informed
conversation about the consequences. But, we observed that the
bare results were hard to interpret, a problem exacerbated by
the vastness of the results due to combinatorial effects.
To address this, most tests now have an expectation for the outcome,
and an explanation for the expected outcome. (If one of these
expectations disagree with you,
please get
in touch!) If the result of an operation agrees with the
expectation, the result has a green background and has a diagonal
line in the top-left corner. If they disagree, the background is
red and the line is in the top-right corner.
This is an example.
Additional artifacts:
|
Consumer |
FooPGP 1 |
BarPGP 2 |
BazPGP 3 |
PartialPGP 4 |
Expectation |
Comment |
| Producer |
Artifact |
| Base case |
|
โ
|
โ
|
โ
|
โ
|
โ
|
| Well-formed variant |
|
โ
|
โ
|
โ
|
โ
|
โ
|
| Malformed variant |
|
โ
|
โ
|
โ
|
โ
|
โ
|
| Weird variant |
|
โ
|
โ
|
โ
|
โ
|
|
| Producer failure |
โ
|
|
|
|
|
โ
|
An example consumer test result
This is the configuration used to produce this report:
{
"drivers": [
{
"env": {},
"id": null,
"path": "/sops/sqop",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": "{version_root}+pqc",
"path": "/sops/sqop-pqc",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": null,
"path": "/sops/rsop",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": "{version}+psk",
"path": "/sops/rsop-psk",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": null,
"path": "/sops/gosop",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": null,
"path": "/sops/gosop-gopenpgp-v2",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": "{version}+pqc",
"path": "/sops/gosop-pqc",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": null,
"path": "/sops/sop-openpgpjs/sopenpgpjs",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": null,
"path": "/sops/sop-openpgpjs-v5/sop-openpgp.js",
"shutdown": null,
"startup": null
},
{
"env": {
"OPENPGPJS_CUSTOM_PROFILES": "{ \"generate-key\": { \"rfc9980\": { \"description\": \"generate v6 PQC keys using ML-DSA-65+Ed25519/ML-KEM-768+X25519\", \"options\": { \"type\": \"pqc\", \"config\": { \"v6Keys\": true } } }, \"draft-ietf-openpgp-persistent-symmetric-keys-03\": { \"description\": \"generate a persistent symmetric key\", \"options\": { \"type\": \"symmetric\", \"config\": { \"v6Keys\": true } } } } }",
"OPENPGPJS_PATH": "/sops/openpgpjs-psk"
},
"id": "{version}+psk",
"path": "/sops/sop-openpgpjs/sopenpgpjs",
"shutdown": null,
"startup": null
},
{
"env": {
"JAVA_HOME": "/opt/java/openjdk"
},
"id": null,
"path": "/sops/pgpainless-cli-ng",
"shutdown": "/opt/stop-nailgun",
"startup": "/opt/start-nailgun"
},
{
"env": {
"PYTHONPATH": "/opt/PGPy"
},
"id": null,
"path": "/sops/sopgpy",
"shutdown": null,
"startup": null
},
{
"env": {
"LD_LIBRARY_PATH": "/opt/rnp/lib/"
},
"id": null,
"path": "/sops/rnp-sop",
"shutdown": null,
"startup": null
},
{
"env": {
"LD_LIBRARY_PATH": "/opt/rnp-v6/lib/"
},
"id": "{version}+crypto-refresh",
"path": "/sops/rnp-sop",
"shutdown": null,
"startup": null
},
{
"env": {
"LD_LIBRARY_PATH": "/opt/rnp-pqc/lib/"
},
"id": "{version_root}+pqc",
"path": "/sops/rnp-sop",
"shutdown": null,
"startup": null
},
{
"env": {
"GNUPG_BIN": "/opt/chameleon/gpg-sq"
},
"id": "Sq's GnuPG Chameleon 0.13.1",
"path": "/sops/gpgme-sop",
"shutdown": null,
"startup": null
},
{
"env": {
"GNUPG_BIN": "/opt/gnupg-2.4.9/bin/gpg"
},
"id": "{version} (upstream)",
"path": "/sops/gpgme-sop",
"shutdown": null,
"startup": null
},
{
"env": {
"GNUPG_BIN": "/opt/gnupg-2.4.9-freepg-1/bin/gpg"
},
"id": "{version_root} (FreePG)",
"path": "/sops/gpgme-sop",
"shutdown": null,
"startup": null
},
{
"env": {
"GNUPG_BIN": "/usr/bin/gpg"
},
"id": "{version} (Debian)",
"path": "/sops/gpgme-sop",
"shutdown": null,
"startup": null
},
{
"env": {
"GNUPG_BIN": "/usr/bin/gpg1"
},
"id": "{version} (Debian)",
"path": "/sops/gpgme-sop",
"shutdown": null,
"startup": null
},
{
"env": {},
"id": null,
"path": "/sops/msop",
"shutdown": null,
"startup": null
}
],
"rlimits": {
"CPU": 30,
"DATA": 17179869184
}
}