Refactor to use a consistent code path for determining the build
args for a service image regardless of whether BuildKit or the
classic builder is being used.
After recent changes, these code paths had diverged, so the classic
builder was missing the proxy variables from the Docker client
config.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
This was left over from debugging, but we should not block.
OTel will handle the connection in the background.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Swap the default implementation now that batching is merged.
Keeping the `docker cp` based implementation around for the
moment, but it needs to be _explicitly_ disabled now by setting
`COMPOSE_EXPERIMENTAL_WATCH_TAR=0`.
After the next release, we should remove the `docker cp`
implementation entirely.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
If an optional dependency exits successfully (exit code of 0),
with a service condition of `service_completed_successfully`,
don't log a warning.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
* When waiting for dependencies, `select` on the context as well
as the ticker
* Write multiple progress events "transactionally" (i.e. hold the
lock for the duration to avoid other events being interleaved)
* Do not change "finished" steps back to "in progress" to prevent
flickering
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Move builder and nodes initialization code up, avoiding to recreate/load them for every service build.
Signed-off-by: Silvin Lubecki <silvin.lubecki@docker.com>
Adjust the debouncing logic so that it applies to all inbound file
events, regardless of whether they match a sync or rebuild rule.
When the batch is flushed out, if any event for the service is a
rebuild event, then the service is rebuilt and all sync events for
the batch are ignored. If _all_ events in the batch are sync events,
then a sync is triggered, passing the entire batch at once. This
provides a substantial performance win for the new `tar`-based
implementation, as it can efficiently transfer the changes in bulk.
Additionally, this helps with jitter, e.g. it's not uncommon for
there to be double-writes in quick succession to a file, so even if
there's not many files being modified at once, it can still prevent
some unnecessary transfers.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Support services with scale > 1 for the tar watch sync.
Add a "lossy" multi-writer specific to pipes that writes the
tar data to each `io.PipeWriter`, which is connected to `stdin`
for the `tar` process being exec'd in the container.
The data is written serially to each writer. This could be
adjusted to do concurrent writes but that will rapidly increase
the I/O load, so is not done here - in general, 99% of the
time you'll be developing (and thus using watch/sync) with a
single replica of a service.
If a write fails, the corresponding `io.PipeWriter` is removed
from the active set and closed with an error.
This means that a single container copy failing won't stop
writes to the others that are succeeding. Of course, they will
be in an inconsistent state afterwards still, but that's a
different problem.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
We cannot guarantee the exact value of `CapEff` across
environments, and this test has started failing some places,
e.g. Docker Desktop, and now GitHub Actions (likely due to
a kernel upgrade on the runners or similar).
By setting `privileged: true` on the build, we're asking for
the `security.insecure` entitlement on the build. A safe
assumption is that will include `CAP_SYS_ADMIN`, which won't
be present otherwise, so mask the `CapEff` value and check
for that.
It's worth noting that realistically, the build won't even
be able to complete without the correct entitlement, since the
`Dockerfile` uses `RUN --security=insecure`, so this is really
an additional sanity check.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Includes a fix for CVE-2023-29409
go1.20.7 (released 2023-08-01) includes a security fix to the crypto/tls
package, as well as bug fixes to the assembler and the compiler. See the
Go 1.20.7 milestone on our issue tracker for details:
- https://github.com/golang/go/issues?q=milestone%3AGo1.20.7+label%3ACherryPickApproved
- full diff: https://github.com/golang/go/compare/go1.20.6...go1.20.7
From the mailing list announcement:
[security] Go 1.20.7 and Go 1.19.12 are released
Hello gophers,
We have just released Go versions 1.20.7 and 1.19.12, minor point releases.
These minor releases include 1 security fixes following the security policy:
- crypto/tls: restrict RSA keys in certificates to <= 8192 bits
Extremely large RSA keys in certificate chains can cause a client/server
to expend significant CPU time verifying signatures. Limit this by
restricting the size of RSA keys transmitted during handshakes to <=
8192 bits.
Based on a survey of publicly trusted RSA keys, there are currently only
three certificates in circulation with keys larger than this, and all
three appear to be test certificates that are not actively deployed. It
is possible there are larger keys in use in private PKIs, but we target
the web PKI, so causing breakage here in the interests of increasing the
default safety of users of crypto/tls seems reasonable.
Thanks to Mateusz Poliwczak for reporting this issue.
View the release notes for more information:
https://go.dev/doc/devel/release#go1.20.7
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Just moving some code around in preparation for an alternative
sync implementation that can do bulk transfers by using `tar`.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
It's no longer used in docker/cli, and doesn't do anything other than
creating an empty struct, so replacing it (as we're planning to
deprecate that function)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
When building, if images are being pushed, ensure that only
named images (i.e. services with a populated `image` field)
are attempted to be pushed.
Services without `image` get an auto-generated name, which
will be a "Docker library" reference since they're in the
format `$project-$service`, which is implicitly the same as
`docker.io/library/$project-$service`. A push for that is
never desirable / will always fail.
The key here is that we cannot overwrite the `<svc>.image`
field when doing builds, as we need to be able to check for
its presence to determine whether a push makes sense.
Fixes#10813.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Fix forthcoming via https://github.com/compose-spec/compose-go/pull/436
which addresses some symlink limitations. These can
actually effect other platforms but are most common
on macOS because the test creates temporary directories,
which are symlinked on macOS.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Lots of our phony Compose files launch pointless long-lived processes
so we can assert on state. However, this means they often don't respond
well to signals on their own, requiring Compose to timeout and kill
them when doing a `down`.
Add in lots of `init: true` where appropriate so that we don't block
for no reason while running E2E tests all over the place.
Additionally, a couple tests have gotten a cleanup so they don't leave
behind containers. I still want to build this into the framework in
the future, but this is easier for the moment and won't cause any
trouble in the future.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
go1.20.6 (released 2023-07-11) includes a security fix to the net/http package,
as well as bug fixes to the compiler, cgo, the cover tool, the go command,
the runtime, and the crypto/ecdsa, go/build, go/printer, net/mail, and text/template
packages. See the Go 1.20.6 milestone on our issue tracker for details.
https://github.com/golang/go/issues?q=milestone%3AGo1.20.6+label%3ACherryPickApproved
Full diff: https://github.com/golang/go/compare/go1.20.5...go1.20.6
These minor releases include 1 security fixes following the security policy:
net/http: insufficient sanitization of Host header
The HTTP/1 client did not fully validate the contents of the Host header.
A maliciously crafted Host header could inject additional headers or entire
requests. The HTTP/1 client now refuses to send requests containing an
invalid Request.Host or Request.URL.Host value.
Thanks to Bartek Nowotarski for reporting this issue.
Includes security fixes for [CVE-2023-29406 ][1] and Go issue https://go.dev/issue/60374
[1]: https://github.com/advisories/GHSA-f8f7-69v5-w4vx
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Add an end-to-end test that covers the core watch functionality,
i.e. CRUD on files & directories.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
The local Go package module path was missing from a couple of jobs,
which made them slower than needed since they were re-downloading
a bunch of dependencies.
In particular, this makes `make lint` waaaay faster!
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
* Run `down` before and after test to not leave around containers
* Kill the `wait` process that's waiting on `infinity`
* NOTE: If the test is actually working, this should exit once
the `down` happens, but this ensures that we kill everything
we start
I'd like to generalize more of this into the framework, but this
is a quick fix to prevent filling up CI machines with tons of
processes over time.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
closes#10783
Compose Spec mentions that default values for secrets is `0444` aka. world-readable permissions. However, the value was previously set to `0400`.
Signed-off-by: Shan Desai <shantanoo.desai@gmail.com>
As part of the fix for #10668, the logic was adjusted so that the
default (highest-priority) network is used in the `ContainerCreate`,
and then the remaining networks are connected via calls to
`NetworkConnect` before starting the container.
Unfortunately, `ServiceConfig::NetworksByPriority` is neither
deterministic nor stable when networks have the same priority.
It's non-deterministic because the order of networks from parsing
YAML is random, since they are loaded into a Go map (which have
random iteration order). Additionally, it's not using a `SortStable`
in `compose-go`, so even if the load order was predictable, it
still might produce different results.
While I look at improving `compose-go` here to prevent this from
tripping us up in the future, this fix looks at _all_ networks for
a service and ignores the "default" one now. Before, it would
always skip the first one in the slice since that _should_ have
been the "default".
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Engine API only allows at most one network to be connected as
part of the ContainerCreate API request. Compose will pick the
highest priority network.
Afterwards, the remaining networks (if any) are connected before
the container is actually started.
The big change here is that, previously, the highest-priority
network was connected in the create, and then disconnected and
immediately reconnected along with all the others. This was
racy because evidently connecting the container to the network
as part of the create isn't synchronous, so sometimes when Compose
tried to disconnect it, the API would return an error like:
```
container <id> is not connected to the network <network>
```
To avoid needing to disconnect and immediately reconnect, the
network config logic has been refactored to ensure that it sets
up the network config correctly the first time.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
* Move all the initialization code out of `main.go`
* Ensure spans are reported when there's an error with the
command
* Attach the Compose version & active Docker context to the
resource instead of the span
* Name the root CLI span `cli/<cmd>` for clarity and grab
the full subcommand path (e.g. `alpha-viz` instead of just
`viz`)
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
* If there's no `otel` key (or the value is `null`) in the config,
don't return an error
* Propagate error from the exporter instead of panicking
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Do not set a hardcoded default timeout of 10 seconds when restarting / stopping but use the container `StopTimeout` (defaults to 10 seconds).
Also fixed custom timeout not used when invoking `up`.
Signed-off-by: Robbert Segeren <robbert.segeren@easyflex.nl>
This is a bunch of OTEL initialization code. It's all in
`internal/` because there are re-usable parts here, but Compose
isn't the right spot. Once we've stabilized the interfaces a bit
and the need arises, we can move it to a separate module.
Currently, a single span is produced to wrap the root Compose
command.
Compose will respect the standard OTEL environment variables
as well as OTEL metadata from the Docker context. Both can be
used simultaneously. The latter is intended for local system
integration and is restricted to Unix sockets / named pipes.
None of this is enabled by default. It requires setting the
`COMPOSE_EXPERIMENTAL_OTEL=1` environment variable to
gate it during development.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Attempting to fix the state of codecov action checks right now,
which are behaving very erratically.
Using the new functionality in Go 1.20 to merge multiple reports,
so now the unit & E2E coverage data reports are stored as artifacts
and then downloaded, merged, and finally uploaded to codecov as a
new job.
Additionally, add a `codecov.yml` config and try to turn down the
aggressiveness of it for CI checks.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Previously, this was telling us "but was not created for project
[project-it-was-created-for]", which is wrong. I opted to make the
message super explicit and print both the actual project and the
expected project.
Signed-off-by: Jeremy Fleischman <jeremyfleischman@gmail.com>
Write the warning using `logrus.Warn`. The function being used was
coming from `cfssl`'s log package, which was presumably the result
of auto-import being _slightly_ too aggressive.
(Note: `cfssl` is still an indirect dependency after this.)
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
docker/docker already depends on oauth2 v0.1.0, which differs only in
dependencies;
https://github.com/golang/oauth2/compare/6fdb5e3db783...v0.1.0
The replace rule is still needed to help with some dependency hell when
resolving the version to use;
go mod tidy
go: downloading cloud.google.com/go v0.75.0
github.com/docker/compose/v2/pkg/compose imports
github.com/moby/buildkit/client imports
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc tested by
go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc.test imports
google.golang.org/grpc/interop imports
golang.org/x/oauth2/google imports
cloud.google.com/go/compute/metadata: ambiguous import: found package cloud.google.com/go/compute/metadata in multiple modules:
cloud.google.com/go v0.75.0 (/Users/thajeztah/go/pkg/mod/cloud.google.com/go@v0.75.0/compute/metadata)
cloud.google.com/go/compute/metadata v0.2.1 (/Users/thajeztah/go/pkg/mod/cloud.google.com/go/compute/metadata@v0.2.1)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
go1.20.4 (released 2023-05-02) includes three security fixes to the html/template
package, as well as bug fixes to the compiler, the runtime, and the crypto/subtle,
crypto/tls, net/http, and syscall packages. See the Go 1.20.4 milestone on our
issue tracker for details:
https://github.com/golang/go/issues?q=milestone%3AGo1.20.4+label%3ACherryPickApproved
release notes: https://go.dev/doc/devel/release#go1.20.4
full diff: https://github.com/golang/go/compare/go1.20.3...go1.20.4
from the announcement:
> These minor releases include 3 security fixes following the security policy:
>
> - html/template: improper sanitization of CSS values
>
> Angle brackets (`<>`) were not considered dangerous characters when inserted
> into CSS contexts. Templates containing multiple actions separated by a '/'
> character could result in unexpectedly closing the CSS context and allowing
> for injection of unexpected HMTL, if executed with untrusted input.
>
> Thanks to Juho Nurminen of Mattermost for reporting this issue.
>
> This is CVE-2023-24539 and Go issue https://go.dev/issue/59720.
>
> - html/template: improper handling of JavaScript whitespace
>
> Not all valid JavaScript whitespace characters were considered to be
> whitespace. Templates containing whitespace characters outside of the character
> set "\t\n\f\r\u0020\u2028\u2029" in JavaScript contexts that also contain
> actions may not be properly sanitized during execution.
>
> Thanks to Juho Nurminen of Mattermost for reporting this issue.
>
> This is CVE-2023-24540 and Go issue https://go.dev/issue/59721.
>
> - html/template: improper handling of empty HTML attributes
>
> Templates containing actions in unquoted HTML attributes (e.g. "attr={{.}}")
> executed with empty input could result in output that would have unexpected
> results when parsed due to HTML normalization rules. This may allow injection
> of arbitrary attributes into tags.
>
> Thanks to Juho Nurminen of Mattermost for reporting this issue.
>
> This is CVE-2023-29400 and Go issue https://go.dev/issue/59722.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Some error messages have been tweaked slightly, this adapts the
assertions to work on both Engine v20.10.x and v23.x.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
If we go to inspect a container that we got an event for and it
no longer exists on the server, handle clean up without erroring
out.
Fixes#10373.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
The IndexServerAddress field was as part of the initial Windows implementation
of the engine. For legal reasons, Microsoft Windows (and thus Docker images
based on Windows) were not allowed to be distributed through non-Microsoft
infrastructure. As a temporary solution, a dedicated "registry-win-tp3.docker.io"
registry was created to serve Windows images.
Using separate registries was not an ideal solution, and a more permanent
solution was created by introducing "foreign image layers" in the distribution
spec, after which the "registry-win-tp3.docker.io" ceased to exist, and
removed from the engine.
This replaces the code that calls out to the "/info" endpoint to use the
GetAuthConfigKey() function instead.
Related PR in docker/cli:
b4ca1c7368
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
go mod tidy sometimes messes up grouping of imports, and may
need a bit of hand-holding in those cases :)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Use latest Go minor release. Note: this release included fixes for
several CVEs, but they do not impact Compose.
Small errors have been fixed to keep the linter happy.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Instead of trying to make this work nicely cross-platform,
just push the Coverage logic into the GitHub Actions job,
as that's really where we care about it.
(It's surprisingly difficult to make this nicely portable;
to make PowerShell not error out if the path does not exist
you have to ignore ALL errors and the way that env vars are
passed to processes is not the same.)
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
This was running two tests in parallel that would build/delete the
same images. Run in serial instead since that's not safe.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Was getting segfaults with multiple services using
`x-develop` and `watch` at the same time. Turns out
the Moby path matcher lazily initializes the regex
pattern internally the first time it's used, so it's
not goroutine-safe.
Change here is to not use a global instance for the
ephemeral path matcher, but a per-watcher instance.
Additionally, the data race detector caught a couple
other issues that were easy enough to fix:
* Use the lock that's used elsewhere for convergence
before manipulating
* Eliminate concurrent map access when triggering
rebuilds
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
This approach mimics Tilt's behavior[^1]:
1. At sync time, `stat` the path on host
2. If the path does not exist -> `rm` from container
3. If the path exists -> sync to container
By handling things this way, we're always syncing based on the true
state, regardless of what's happened in the interim. For example, a
common pattern in POSIX tools is to create a file and then rename it
over an existing file. Based on timing, this could be a sync, delete,
sync (every file gets seen & processed) OR a delete, sync (by the
the time we process the event, the "temp" file is already gone, so
we just delete it from the container, where it never existed, but
that's fine since we deletes are idempotent thanks to the `-f` flag
on `rm`).
Additionally, when syncing, if the `stat` call shows it's for a
directory, we ignore it. Otherwise, duplicate, nested copies of the
entire path could get synced in. (On some OSes, an event for the
directory gets dispatched when a file inside of it is modified. In
practice, I think we might want this pushed further down in the
watching code, but since we're already `stat`ing the paths here now,
it's a good place to handle it.)
Lastly, there's some very light changes to the text when it does a
full rebuild that will list out the (merged) set of paths that
triggered it. We can continue to improve the output, but this is
really helpful for understanding why it's rebuilding.
[^1]: db7f887b06/internal/controllers/core/liveupdate/reconciler.go (L911)
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
While this stripping does decrease the binary size by some amount, it also removes the ability for `govulncheck` (https://go.dev/blog/vuln) to scan the binary for actual uses of vulnerable functions, requiring the user to clone the code locally and hope they're testing against the same version of the stdlib, etc that the binary was built with. If we stop passing `-s`, then we can then run `govulncheck` on the binary directly (making it easier to flag both false positives in CVE scans _and_ actual issues worth looking into).
Here's an example of the output on a freshly built binary with this change:
```console
$ govulncheck ./bin/build/docker-compose
govulncheck is an experimental tool. Share feedback at https://go.dev/s/govulncheck-feedback.
Using govulncheck@v0.0.0 with
vulnerability data from https://vuln.go.dev (last modified 27 Feb 23 16:29 UTC).
Scanning your binary for known vulnerabilities...
No vulnerabilities found.
```
Compared to the 1.16.0 release binary:
```console
$ govulncheck ./docker-compose
go: downloading golang.org/x/vuln v0.0.0-20230224180816-edec1fb0a9c7
govulncheck is an experimental tool. Share feedback at https://go.dev/s/govulncheck-feedback.
Using govulncheck@v0.0.0 with
vulnerability data from https://vuln.go.dev (last modified 27 Feb 23 16:29 UTC).
Scanning your binary for known vulnerabilities...
govulncheck: vulncheck.Binary: reading go:func.*: no symbol "go:func.*"
```
It's not 100% apples-to-apples, but the size difference between these binaries is ~46MiB for the 1.16.0 release and ~52MiB for the binary I built from this commit.
Signed-off-by: Tianon Gravi <admwiggin@gmail.com>
This test keeps failing with a timeout in Windows. I don't actually
think it should take that long to bring up an nginx container, so
I'm guessing that there's something else going on that's causing
trouble.
Increase the verbosity when running Compose commands: I think this
will generally make E2E test failures easier to diagnose by always
logging the full command that's going to be run and also capturing
stdout.
Add a health check and use `--wait` when launching the fixture for
the pause test. Combined with the verbosity increase, this should
make it easier to understand what's going on here.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Big change here is to import the ephemeral ignore set from Tilt.
The `.git` directory is also ignored for now: this restriction
should probably be lifted and made configurable in the future,
but it's not generally important to watch and triggers a LOT of
events (e.g. Git creates `index.lock` files that will appear and
disappear rapidly as terminals/IDEs/etc interact with Git, even
for read-only operations).
The Tilt-provided ephemeral file set has been slowly devised over
time based on temporary files that can cause trouble. We can also
look at a more robust/configurable solution here in the future,
but thse provide a reasonable out-of-the-box configuration for
the moment.
There's also some small tweaks to the output to add missing
newlines in a few edge cases and such.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
I misunderstood the cause of the symptom in #10261 - thought that
we'd explicitly turned off CGO for Windows with some of the build
changes recently, but we don't even have `gcc` on the CI node, so
it's actually just `-race` entirely that's the trouble.
For right now, going the easy route and disabling it. We can look
at getting a C toolchain on the Windows machine later.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
```
go: -race requires cgo; enable cgo by setting CGO_ENABLED=1
```
We're explicitly using CGO on macOS now for FSEvents support and
purposefully NOT using CGO on other platforms since we don't need
it.
The race detector (`-race`) requires it, however, so for the e2e
make task, it should alway be on.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
`exportloopref` - detects captures of loop variable without
re-assignment
NOTE: There can be false negatives with this linter to avoid being
overly strict and annoying!
Also upgraded `golangci-lint` to latest (v1.43.0 published 2021-11-03).
I've been running the test suite (`./internal/engine` in particular)
with `-count X` a lot recently to catch timing-related test failures.
After running enough times, the tests start failing due to too many
open files, so I did an audit and found a few places where files or
readers weren't always being closed.
* lint: add make lintfix and run it
Fixes all errors like:
```
File is not `goimports`-ed with -local github.com/tilt-dev/tilt (goimports)
```
* git: change to use TrimSuffix
* build: remove unnecessary cast
`WalkDir` is new in Go 1.16 and avoids calling `os.Lstat` on
every visited file and directory. In most cases, we don't need
that info, so this will help reduce I/O when listing files,
which can be helpful for particularly big monorepos.
Unused code linter isn't particularly smart about platform build
tags, so since this func is only used by the "naive" (non-macOS)
file watcher, it needs to live with that or it gets flagged as
dead code when linting on macOS.
In most places in Tilt, we try to use absolute paths everywhere.
So this makes things more consistent with the rest of Tilt, and lets us be
a bit more flexible in how we handle subdirs and parent dirs in ignores.
Fixes https://github.com/tilt-dev/tilt/issues/3740
Here's our new watch strategy on Darwin in a nutshell:
1. Create an fsevents stream for events "since" the last event ID that
we saw globally.
2. Add a path that we want to watch
3. Add that path to a map of paths that we're watching _directly_.
4. Restart the event stream to pick up the new path.
5. Ignore all events for all watches until we've seen a `HistoryDone`
event.
6. Ignore the first `ItemCreated` event for paths we're watching that
are also directories
7. Otherwise, forward along all events.
Without this I get an exception when building multiple images in a
compose run.
```
fatal error: concurrent map writes
goroutine 16 [running]:
github.com/docker/compose/v2/pkg/compose.(*composeService).build.func1({0x2cba4e0, 0xc00019b2c0}, {0xc000233150?, 0xd?})
github.com/docker/compose/v2/pkg/compose/build.go:95 +0x652
github.com/docker/compose/v2/pkg/compose.(*graphTraversal).run.func1()
github.com/docker/compose/v2/pkg/compose/dependencies.go:127 +0x63
golang.org/x/sync/errgroup.(*Group).Go.func1()
golang.org/x/sync@v0.1.0/errgroup/errgroup.go:75 +0x64
created by golang.org/x/sync/errgroup.(*Group).Go
golang.org/x/sync@v0.1.0/errgroup/errgroup.go:72 +0xa5
...
```
I'm not sure why the map is currently shared, but with this patch
applied my docker-compose build run works.
Signed-off-by: Edward Muller <emuller@fastly.com>
"docker compose config --output out.yml" resulted an empty file
when the generated output was smaller than 4097 bytes.
bufio.Writer doesn't seem necessary since only one write operation will happen.
This way there is no need for a new bufio.Writer that could be flushed.
Thanks for @thaJeztah for the idea of using os.WriteFile
Issue https://github.com/docker/compose/issues/10121
Signed-off-by: Ákos Takács <takacs.akos@it-sziget.hu>
currently the version displayed is the one installed and not the one use for the tests
Signed-off-by: Guillaume Lours <705411+glours@users.noreply.github.com>
Do not use the older `Volumes` field in the API; instead rely on
the more robust `Mounts`. For Linux containers, it seems that it's
fine to set both of these. For Windows containers (WCOW), however,
there appears to be a Moby bug that causes it to normalize the
anonymous (`Volumes`) variant to lowercase, which can result in
duplicative volume definitions and an error when trying to start
the container.
Fixes#9577.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
This package is a leftover from when the "docker" cli and the "dockerd"
cli both lived in the same repository. The package in docker/docker will
be (re)moved soon, so replace it with the implementation in docker/cli,
which is the right one :)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Includes security fixes for net/http (CVE-2022-41717, CVE-2022-41720),
and os (CVE-2022-41720).
These minor releases include 2 security fixes following the security policy:
- os, net/http: avoid escapes from os.DirFS and http.Dir on Windows
The os.DirFS function and http.Dir type provide access to a tree of files
rooted at a given directory. These functions permitted access to Windows
device files under that root. For example, os.DirFS("C:/tmp").Open("COM1")
would open the COM1 device.
Both os.DirFS and http.Dir only provide read-only filesystem access.
In addition, on Windows, an os.DirFS for the directory \(the root of the
current drive) can permit a maliciously crafted path to escape from the
drive and access any path on the system.
The behavior of os.DirFS("") has changed. Previously, an empty root was
treated equivalently to "/", so os.DirFS("").Open("tmp") would open the
path "/tmp". This now returns an error.
This is CVE-2022-41720 and Go issue https://go.dev/issue/56694.
- net/http: limit canonical header cache by bytes, not entries
An attacker can cause excessive memory growth in a Go server accepting
HTTP/2 requests.
HTTP/2 server connections contain a cache of HTTP header keys sent by
the client. While the total number of entries in this cache is capped,
an attacker sending very large keys can cause the server to allocate
approximately 64 MiB per open connection.
This issue is also fixed in golang.org/x/net/http2 vX.Y.Z, for users
manually configuring HTTP/2.
Thanks to Josselin Costanzi for reporting this issue.
This is CVE-2022-41717 and Go issue https://go.dev/issue/56350.
View the release notes for more information:
https://go.dev/doc/devel/release#go1.19.4
And the milestone on the issue tracker:
https://github.com/golang/go/issues?q=milestone%3AGo1.19.4+label%3ACherryPickApproved
Full diff: https://github.com/golang/go/compare/go1.19.3...go1.19.4
The golang.org/x/net fix is in 1e63c2f08a
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The scan tip has been shown for two years, and most users will know
about it by now. Presenting the message also involved checking if the
plugin was installed, and wether or not the message was shown before,
which also caused some overhead, so cleaning up the output a bit.
The corresponding DOCKER_SCAN_SUGGEST environment-variable is also
removed with this.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
This method looked slightly incomplete. If the port wasn't found,
it'd return `err`, but that was always `nil`, so we'd print out
`:0`.
Now, we construct a nice error message with the targeted port and
the ones we found.
The `--protocol` flag is also now case-insensitive to prevent any
weirdness/confusion there.
Co-authored-by: Nick Sieger <nicksieger@gmail.com>
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
See compose-spec/compose-go#325 for the acutal spec change. This
propagates it to the Engine API object and adds an E2E test via
Cucumber 🥒Fixes#9873.
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
Evidently `ping` doesn't respond to `SIGTERM`, so use `init` to
get Tini supervising it. This changes the exit code to 143 since
it's not hitting the 10s timeout and getting a `SIGKILL` (137).
Signed-off-by: Milas Bowman <milas.bowman@docker.com>
This restores compatibility with go1.18, which was broken since commit;
c062238ea4
cmd.Environ() is new in go1.19, and not needed for this specific case.
Without this, trying to use this package in code that uses go1.18 will fail;
builder/remotecontext/git/gitutils.go:216:23: cmd.Environ undefined (type *exec.Cmd has no field or method Environ)
Changing to use `os.Environ()` instead restores compatibility with go1.18
Full diff: f9cb47a052...5aac513617
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
to match the version used by the cli (the cli doesn't use go.mod, so go modules
doesn't automatically pick that up);
1d6c6e2367/vendor.mod (L14)
Used code doesn't change, but we want to keep the older github.com/danieljoos/wincred v1.1.0
out of the dependency tree :)
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
BuildKit and Buildx no longer require this replace rule (it probably only was
needed in buildkit, which used this version to compile).
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
The file had multiple "requires" blocks, which made it harder to find which
dependencies were used. Some direct modules also were in the "indirect" block.
While updating, also updated some comments.
Signed-off-by: Sebastiaan van Stijn <github@gone.nl>
Currently the emacs ignore patterns include `**/.#*` (lock files), but
doesn't include `**/#*#` (autosave files;
https://www.emacswiki.org/emacs/AutoSave, not to be confused with
`**/*~` backup files which are ignored.)
Add autosave files.
When creating files in Go, the stdlib will create (and then rapidly
delete) files ending with `-go-tmp-umask` to determine the umask
to use for permission purposes.
This can cause trouble with Live Update because the files tend to
vanish underneath it, for example.
Fixes#5117.
e2e-compose:## Run end to end local tests in plugin mode. Set E2E_TEST=TestName to run a single test
docker compose version
go test$(TEST_FLAGS)$(TEST_COVERAGE_FLAGS) -count=1 ./pkg/e2e
go test$(TEST_FLAGS) -count=1 ./pkg/e2e
.PHONY:e2e-compose-standalone
e2e-compose-standalone:## Run End to end local tests in standalone mode. Set E2E_TEST=TestName to run a single test
docker-compose version
go test$(TEST_FLAGS) -v -count=1 -parallel=1 --tags=standalone ./pkg/e2e
.PHONY:test-cucumber
test-cucumber:
go test$(TEST_FLAGS) -v -count=1 -parallel=1 ./e2e
.PHONY:build-and-e2e-compose
build-and-e2e-compose:builde2e-compose## Compile the compose cli-plugin and run end to end local tests in plugin mode. Set E2E_TEST=TestName to run a single test
@@ -77,6 +89,7 @@ build-and-e2e-compose-standalone: build e2e-compose-standalone ## Compile the co
.PHONY:mocks
mocks:
mockgen --version >/dev/null 2>&1|| go install github.com/golang/mock/mockgen@v1.6.0
Docker Compose is a tool for running multi-container applications on Docker
defined using the [Compose file format](https://compose-spec.io).
A Compose file is used to define how the one or more containers that make up
A Compose file is used to define how one or more containers that make up
your application are configured.
Once you have a Compose file, you can create and start your application with a
single command: `docker compose up`.
# About update and backward compatibility
Docker Compose V2 is a major version bump release of Docker Compose. It has been completely rewritten from scratch in Golang (V1 was in Python). The installation instructions for Compose V2 differ from V1. V2 is not a standalone binary anymore, and installation scripts will have to be adjusted. Some commands are different.
For a smooth transition from legacy docker-compose 1.xx, please consider installing [compose-switch](https://github.com/docker/compose-switch) to translate `docker-compose ...` commands into Compose V2's `docker compose .... `. Also check V2's `--compatibility` flag.
# Where to get Docker Compose
### Windows and macOS
@@ -34,7 +37,7 @@ for Windows and macOS.
You can download Docker Compose binaries from the
[release page](https://github.com/docker/compose/releases) on this repository.
Rename the relevant binary for your OS to `docker-compose` and copy it to `$HOME/.docker/cli-plugins`
Rename the relevant binary for your OS to `docker-compose` and copy it to `$HOME/.docker/cli-plugins`
Or copy it into one of these folders to install it system-wide:
@@ -47,7 +50,7 @@ Or copy it into one of these folders to install it system-wide:
Quick Start
-----------
Using Docker Compose is basically a three-step process:
Using Docker Compose is a three-step process:
1. Define your app's environment with a `Dockerfile` so it can be
reproduced anywhere.
2. Define the services that make up your app in `docker-compose.yml` so
@@ -77,3 +80,8 @@ Want to help develop Docker Compose? Check out our
fmt.Fprint(os.Stderr,"--progress is a global compose flag, better use `docker compose --progress xx build ...")
}
returnrunBuild(ctx,backend,opts,args)
}),
ValidArgsFunction:completeServiceNames(p),
}
cmd.Flags().BoolVar(&opts.push,"push",false,"Push service images.")
cmd.Flags().BoolVarP(&opts.quiet,"quiet","q",false,"Don't print anything to STDOUT")
cmd.Flags().BoolVar(&opts.pull,"pull",false,"Always attempt to pull a newer version of the image.")
cmd.Flags().StringVar(&opts.progress,"progress",buildx.PrinterModeAuto,fmt.Sprintf(`Set type of progress output (%s)`,strings.Join(printerModes,", ")))
cmd.Flags().StringArrayVar(&opts.args,"build-arg",[]string{},"Set build-time variables for services.")
cmd.Flags().StringVar(&opts.ssh,"ssh","","Set SSH authentications used when building service images. (use 'default' for using your default SSH Agent)")
cmd.Flags().StringVar(&opts.builder,"builder","","Set builder to use.")
cmd.Flags().Bool("parallel",true,"Build images in parallel. DEPRECATED")
f.StringVar(&o.EnvFile,"env-file","","Specify an alternate environment file.")
f.StringArrayVar(&o.EnvFiles,"env-file",nil,"Specify an alternate environment file.")
f.StringVar(&o.ProjectDir,"project-directory","","Specify an alternate working directory\n(default: the path of the, first specified, Compose file)")
f.StringVar(&o.WorkDir,"workdir","","DEPRECATED! USE --project-directory INSTEAD.\nSpecify an alternate working directory\n(default: the path of the, first specified, Compose file)")
f.BoolVar(&o.Compatibility,"compatibility",false,"Run compose in backward compatibility mode")
flags.BoolVar(&opts.removeOrphans,"remove-orphans",removeOrphans,"Remove containers for services not defined in the Compose file.")
flags.IntVarP(&opts.timeout,"timeout","t",10,"Specify a shutdown timeout in seconds")
flags.BoolVarP(&opts.volumes,"volumes","v",false,"Remove named volumes declared in the `volumes` section of the Compose file and anonymous volumes attached to containers.")
flags.IntVarP(&opts.timeout,"timeout","t",0,"Specify a shutdown timeout in seconds")
flags.BoolVarP(&opts.volumes,"volumes","v",false,`Remove named volumes declared in the "volumes" section of the Compose file and anonymous volumes attached to containers.`)
flags.StringVar(&opts.images,"rmi","",`Remove images used by services. "local" remove only images that don't have a custom tag ("local"|"all")`)
flags.BoolVar(&up.noStart,"no-start",false,"Don't start the services after creating them.")
flags.BoolVar(&up.cascadeStop,"abort-on-container-exit",false,"Stops all containers if any container was stopped. Incompatible with -d")
flags.StringVar(&up.exitCodeFrom,"exit-code-from","","Return the exit code of the selected service container. Implies --abort-on-container-exit")
flags.IntVarP(&create.timeout,"timeout","t",10,"Use this timeout in seconds for container shutdown when attached or when containers are already running.")
flags.IntVarP(&create.timeout,"timeout","t",0,"Use this timeout in seconds for container shutdown when attached or when containers are already running.")
✔ DRY-RUN MODE - Container nginx-golang-mysql-backend-1 Started 0.0s
✔ DRY-RUN MODE - Container nginx-golang-mysql-proxy-1 Started Started
```
From the example above, you can see that the first step is to pull the image defined by `db` service, then build the `backend` service.
Next, the containers are created. The `db` service is started, and the `backend` and `proxy` wait until the `db` service is healthy before starting.
Dry Run mode works with almost all commands. You cannot use Dry Run mode with a command that doesn't change the state of a Compose stack such as `ps`, `ls`, `logs` for example.
✔ DRY-RUN MODE - Container nginx-golang-mysql-backend-1 Started 0.0s
✔ DRY-RUN MODE - Container nginx-golang-mysql-proxy-1 Started Started
```
From the example above, you can see that the first step is to pull the image defined by `db` service, then build the `backend` service.
Next, the containers are created. The `db` service is started, and the `backend` and `proxy` wait until the `db` service is healthy before starting.
Dry Run mode works with almost all commands. You cannot use Dry Run mode with a command that doesn't change the state of a Compose stack such as `ps`, `ls`, `logs` for example.
usage:docker compose
pname:docker
plink:docker.yaml
cname:
- docker compose build
- docker compose convert
- docker compose config
- docker compose cp
- docker compose create
- docker compose down
@@ -130,9 +171,10 @@ cname:
- docker compose unpause
- docker compose up
- docker compose version
- docker compose wait
clink:
- docker_compose_build.yaml
- docker_compose_convert.yaml
- docker_compose_config.yaml
- docker_compose_cp.yaml
- docker_compose_create.yaml
- docker_compose_down.yaml
@@ -156,6 +198,7 @@ clink:
- docker_compose_unpause.yaml
- docker_compose_up.yaml
- docker_compose_version.yaml
- docker_compose_wait.yaml
options:
- option:ansi
value_type:string
@@ -178,8 +221,19 @@ options:
experimentalcli:false
kubernetes:false
swarm:false
- option:dry-run
value_type:bool
default_value:"false"
description:Execute command in dry run mode
deprecated:false
hidden:false
experimental:false
experimentalcli:false
kubernetes:false
swarm:false
- option:env-file
value_type:string
value_type:stringArray
default_value:'[]'
description:Specify an alternate environment file.
deprecated:false
hidden:false
@@ -208,6 +262,16 @@ options:
experimentalcli:false
kubernetes:false
swarm:false
- option:parallel
value_type:int
default_value:"-1"
description:Control max parallelism, -1 for unlimited
deprecated:false
hidden:false
experimental:false
experimentalcli:false
kubernetes:false
swarm:false
- option:profile
value_type:stringArray
default_value:'[]'
@@ -218,6 +282,16 @@ options:
experimentalcli:false
kubernetes:false
swarm:false
- option:progress
value_type:string
default_value:auto
description:Set type of progress output (auto, tty, plain, quiet)
EXPERIMENTAL - Watch build context for service and rebuild/refresh containers when files are updated
long:|
EXPERIMENTAL - Watch build context for service and rebuild/refresh containers when files are updated
usage:docker compose alpha watch [SERVICE...]
pname:docker compose alpha
plink:docker_compose_alpha.yaml
options:
- option:quiet
value_type:bool
default_value:"false"
description:hide build output
deprecated:false
hidden:false
experimental:false
experimentalcli:false
kubernetes:false
swarm:false
inherited_options:
- option:dry-run
value_type:bool
default_value:"false"
description:Execute command in dry run mode
deprecated:false
hidden:false
experimental:false
experimentalcli:false
kubernetes:false
swarm:false
deprecated:false
hidden:false
experimental:false
experimentalcli:true
kubernetes:false
swarm:false
Some files were not shown because too many files have changed in this diff
Show More
Reference in New Issue
Block a user
Blocking a user prevents them from interacting with repositories, such as opening or commenting on pull requests or issues. Learn more about blocking a user.
