docker: Fix target mount cache paths; relax locks on cargo home.

docker: Persist lychee cache.

Signed-off-by: Jason Volk <jason@zemos.net>
This commit is contained in:
Jason Volk
2025-06-01 11:40:09 +00:00
parent 8d8fcb588b
commit 9507ce9d03
17 changed files with 270 additions and 133 deletions

View File

@@ -55,7 +55,7 @@ so please be patient.
2. Some unsavory options are required for some targets. It might be possible to omit these if
you're not building the full tree. Otherwise I've included them in the create command below.
- To run the complement compliance suite we need the `--allow-insecure-entitlement netwok.host`.
- To run the complement compliance suite we need the `--allow-insecure-entitlement network.host`.
This requirement is probably a defect in Complement.
Finally create
@@ -84,3 +84,48 @@ cues from the primary user of this system, the [GitHub CI](https://github.com/ma
building the targets for `complement-tester` and `complement-testee` using `bake.sh` and then
invoking `complement.sh`. You can take cues again from another user of this in the
[GitHub CI](https://github.com/matrix-construct/tuwunel/blob/main/.github/workflows/test.yml#L79).
## Notes
- For CI our builders are more persistent rather than being created and destroyed for
each invocation. The builder is meant to be safely reused across operations, but
for concurrent operations this is tricky, see the next point.
- For CI our builders are isolated only by actor/user. This will probably change to
`actor/repo/branch` as it's easy to cause issues with concurrent builds right now.
However we can't accept destroying the builder after each use, so we'll likely
choose `actor/repo/branch` with the expectation of one build at a time under those
constraints. Some external caching might need to be contrived between builders for
deduplication but with care such that malicious actors cannot poison data used
by other actors, otherwise it defeats the purpose of builder isolation.
#### On Target Caches
The challenge here is to get all aspects of the target directory perfectly optimal
within the many constraints including cargo issues and our goals. This is highly complex
because we have to provide each image being built with a safe environment yet share as
much as possible between builds. This includes maximum reuse of prior builds but without
unnecessary dirtying or more serious unexpected conflicts.
We first create a hausdorff space based on builds which could never benefit from sharing
and would always be unsound (even silently) if they came into contact; so all cache id's
are prefixed by the matrix components:
`${sys_name}/${sys_version}/${rust_target}/${rust_toolchain}/${cargo_profile}`. This is
important because we can simplify the mount path inside the image which is important for
the absolute paths generated by fingerprints and dependency files. We need to keep those
the same if any builds expect to share them.
The top-level of target directory is immediately partitioned by cargo into different
profiles. Note that we already imposed separation based on profile but we still have to
deal with that subdirectory in the mount, which has special-cases for the dev, test, and
bench profiles. These directories at their top-level are the final artifact area which is
not concurrency safe and there are open issues in cargo for explicit artifact directories.
Within these unsafe directories are sub-directories which contain hash-sharded components
making them safe again for shared caching, so long as the path to them remains consistent
for all images mounting.
As you can see this is already getting very complicated. If this is done wrong lots of
different bad things can happen such as broken builds from bad conflicts, constant
rebuilds from modest conflicts, or over-use of resources from too much separation.
TODO