Backport latest changes from main to release/0.7 (#1498)

* Fix problems related to latest postgres docker image (#1433) Decrease the length of generated database name. * Bump boto3 from 1.34.144 to 1.35.32 (#1434) Bumps [boto3](https://github.com/boto/boto3) from 1.34.144 to 1.35.32. - [Release notes](https://github.com/boto/boto3/releases) - [Commits](boto/boto3@1.34.144...1.35.32) --- updated-dependencies: - dependency-name: boto3 dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump the github-com-aws-aws-sdk-go-v2 group across 1 directory with 3 updates (#1435) Bumps the github-com-aws-aws-sdk-go-v2 group with 3 updates in the / directory: [github.com/aws/aws-sdk-go-v2](https://github.com/aws/aws-sdk-go-v2), [github.com/aws/aws-sdk-go-v2/config](https://github.com/aws/aws-sdk-go-v2) and [github.com/aws/aws-sdk-go-v2/service/s3](https://github.com/aws/aws-sdk-go-v2). Updates `github.com/aws/aws-sdk-go-v2` from 1.30.3 to 1.31.0 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@v1.30.3...v1.31.0) Updates `github.com/aws/aws-sdk-go-v2/config` from 1.27.27 to 1.27.40 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@config/v1.27.27...config/v1.27.40) Updates `github.com/aws/aws-sdk-go-v2/service/s3` from 1.58.2 to 1.64.1 - [Release notes](https://github.com/aws/aws-sdk-go-v2/releases) - [Commits](aws/aws-sdk-go-v2@service/s3/v1.58.2...service/s3/v1.64.1) --- updated-dependencies: - dependency-name: github.com/aws/aws-sdk-go-v2 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-com-aws-aws-sdk-go-v2 - dependency-name: github.com/aws/aws-sdk-go-v2/config dependency-type: direct:production update-type: version-update:semver-patch dependency-group: github-com-aws-aws-sdk-go-v2 - dependency-name: github.com/aws/aws-sdk-go-v2/service/s3 dependency-type: direct:production update-type: version-update:semver-minor dependency-group: github-com-aws-aws-sdk-go-v2 ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump google.golang.org/api from 0.188.0 to 0.199.0 (#1428) Bumps [google.golang.org/api](https://github.com/googleapis/google-api-go-client) from 0.188.0 to 0.199.0. - [Release notes](https://github.com/googleapis/google-api-go-client/releases) - [Changelog](https://github.com/googleapis/google-api-go-client/blob/main/CHANGES.md) - [Commits](googleapis/google-api-go-client@v0.188.0...v0.199.0) --- updated-dependencies: - dependency-name: google.golang.org/api dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump mlflow from 2.14.3 to 2.16.2 (#1424) Bumps [mlflow](https://github.com/mlflow/mlflow) from 2.14.3 to 2.16.2. - [Release notes](https://github.com/mlflow/mlflow/releases) - [Changelog](https://github.com/mlflow/mlflow/blob/master/CHANGELOG.md) - [Commits](mlflow/mlflow@v2.14.3...v2.16.2) --- updated-dependencies: - dependency-name: mlflow dependency-type: direct:development update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump black from 24.4.2 to 24.8.0 (#1377) Bumps [black](https://github.com/psf/black) from 24.4.2 to 24.8.0. - [Release notes](https://github.com/psf/black/releases) - [Changelog](https://github.com/psf/black/blob/main/CHANGES.md) - [Commits](psf/black@24.4.2...24.8.0) --- updated-dependencies: - dependency-name: black dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump cloud.google.com/go/storage from 1.43.0 to 1.44.0 (#1438) Bumps [cloud.google.com/go/storage](https://github.com/googleapis/google-cloud-go) from 1.43.0 to 1.44.0. - [Release notes](https://github.com/googleapis/google-cloud-go/releases) - [Changelog](https://github.com/googleapis/google-cloud-go/blob/main/CHANGES.md) - [Commits](googleapis/google-cloud-go@pubsub/v1.43.0...spanner/v1.44.0) --- updated-dependencies: - dependency-name: cloud.google.com/go/storage dependency-type: direct:production update-type: version-update:semver-minor ... Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Add aws default region (#1472) * update to latest ui ref (#1181) * Add a aws_default_region env var for dev/test * Fix python tests (#1475) * update to latest ui ref (#1181) * Fix python integration tests * Add Attestations (#1490) * Test attestations * Run release in fork * Add attestations: write * Move attestions to job level * Move to job level? * Permissions twice? * Attest docker images * Revert ci.yml change * Move attestation to separate job * Use correct output name * Add dependency to docker-release * Use different echo * Code review jojo * Fix errors * Remove if * Fix attestation generation (#1491) * Add release environment to attest-docker-images (#1492) * Login to Docker Hub before publishing container image attestations (#1493) * Mention mlflow-go (#1496) * Attempt to add multiarch digest (#1495) --------- Signed-off-by: dependabot[bot] <[email protected]> Co-authored-by: Software Developer <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Geoffrey Wilson <[email protected]> Co-authored-by: Florian Verdonck <[email protected]>
G-Research · Jan 31, 2025 · 1ae2afd · 1ae2afd
1 parent 9cdfa3d
commit 1ae2afd
Show file tree

Hide file tree

Showing 12 changed files with 1,031 additions and 804 deletions.
diff --git a/.devcontainer/.env b/.devcontainer/.env
@@ -12,5 +12,6 @@ POSTGRES_HOSTNAME=localhost
 # s3 compatible storage settings.
 AWS_ACCESS_KEY_ID=user
 AWS_SECRET_ACCESS_KEY=password
+AWS_DEFAULT_REGION=us-east-1
 FML_S3_ENDPOINT_URI=http://localhost:9000
 FML_GS_ENDPOINT_URI=http://localhost:4443/storage/v1/
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
@@ -431,5 +431,6 @@ jobs:
       contents: write
       pages: write
       id-token: write
+      attestations: write
     secrets: inherit
     uses: ./.github/workflows/release.yml
diff --git a/.github/workflows/release.yml b/.github/workflows/release.yml
@@ -14,6 +14,7 @@ jobs:
     environment: release
     permissions:
       id-token: write
+      attestations: write
     steps:
       - name: Download artifact
         uses: actions/download-artifact@v4
@@ -22,6 +23,11 @@ jobs:
           merge-multiple: true
           path: wheelhouse
 
+      - name: Add attestations
+        uses: actions/attest-build-provenance@v1
+        with:
+          subject-path: wheelhouse/*.whl
+
       - name: Publish package distributions to PyPI
         uses: pypa/gh-action-pypi-publish@release/v1
         with:
@@ -103,6 +109,7 @@ jobs:
           password: ${{ secrets.DOCKER_PASSWORD }}
 
       - name: Push to Docker Hub
+        id: publish-docker-hub
         run: |
           tags=(${{ steps.tags.outputs.tags }})
           for image in fasttrackml-oci-*.tar
@@ -117,8 +124,50 @@ jobs:
           docker buildx imagetools create \
             $(printf -- "--tag ${{ vars.DOCKER_REPO }}:%s " ${tags[@]}) \
             $(printf "${{ vars.DOCKER_REPO }}@%s " ${digests[@]})
+
+          # Generate the multiarch manifest digest using dry-run and skopeo
+          multiarch_digest=$(docker buildx imagetools create \
+            $(printf -- "--tag ${{ vars.DOCKER_REPO }}:%s " ${tags[@]}) \
+            $(printf "${{ vars.DOCKER_REPO }}@%s " ${digests[@]}) \
+            --dry-run | perl -pe 'chomp if eof' | skopeo manifest-digest /dev/stdin)
+
+          # Add the multiarch digest to the digests array
+          digests+=($multiarch_digest)
           echo "::endgroup::"
 
+          # Log the multiarch digest for reference
+          echo "Multiarch digest added to digests array: $multiarch_digest"
+
+          # Output digests as a JSON array for use in a matrix
+          echo "digests=$(printf '%s\n' "${digests[@]}" | jq -R . | jq -cs .)" >> $GITHUB_OUTPUT
+    outputs:
+      digests: ${{ steps.publish-docker-hub.outputs.digests }}
+
+  attest-docker-images:
+    name: Publish Docker images attestations
+    runs-on: ubuntu-latest
+    environment: release
+    needs: docker-release
+    permissions:
+      id-token: write
+      attestations: write
+    strategy:
+      matrix:
+        digest: ${{ fromJson(needs.docker-release.outputs.digests) }}
+    steps:
+      - name: Login to Docker Hub
+        uses: docker/login-action@v3
+        with:
+          username: ${{ secrets.DOCKER_USERNAME }}
+          password: ${{ secrets.DOCKER_PASSWORD }}
+
+      - name: Attest Each Docker Image
+        uses: actions/attest-build-provenance@v2
+        with:
+          subject-name: index.docker.io/${{ vars.DOCKER_REPO }}
+          subject-digest: ${{ matrix.digest }}
+          push-to-registry: true
+
   chart-release:
     name: Publish Helm chart
     if: startsWith(github.ref, 'refs/tags/v') 

diff --git a/Pipfile b/Pipfile
@@ -8,7 +8,7 @@ black = "*"
 isort = "*"
 
 [dev-packages]
-mlflow = "==2.14.3"
+mlflow = "==2.16.2"
 boto3 = "*"
 psycopg2-binary = "*"
 google-cloud-storage = "*"

diff --git a/Pipfile.lock b/Pipfile.lock
diff --git a/README.md b/README.md
@@ -10,6 +10,11 @@ ships with the visualization UI of both MLflow and Aim.
 As the name implies, the emphasis is on speed -- fast logging, fast
 retrieval.
 
+## mlflow-go
+
+> [!TIP]
+> We are excited to be developing the next evolution of performant experiment tracking at [mlflow/mlflow-go-backend](https://github.com/mlflow/mlflow-go-backend)!
+
 ### Quickstart
 
 #### Run the tracking server

diff --git a/go.mod b/go.mod
@@ -8,17 +8,17 @@ replace (
 )
 
 require (
-	cloud.google.com/go/storage v1.43.0
+	cloud.google.com/go/storage v1.44.0
 	dagger.io/dagger v0.11.6
 	dario.cat/mergo v1.0.0
 	github.com/DATA-DOG/go-sqlmock v1.5.2
 	github.com/G-Research/fasttrackml-ui-aim v0.31705.78
 	github.com/G-Research/fasttrackml-ui-mlflow v0.20902.9
 	github.com/PuerkitoBio/goquery v1.9.2
 	github.com/apache/arrow/go/v14 v14.0.2
-	github.com/aws/aws-sdk-go-v2 v1.30.3
-	github.com/aws/aws-sdk-go-v2/config v1.27.27
-	github.com/aws/aws-sdk-go-v2/service/s3 v1.58.2
+	github.com/aws/aws-sdk-go-v2 v1.31.0
+	github.com/aws/aws-sdk-go-v2/config v1.27.40
+	github.com/aws/aws-sdk-go-v2/service/s3 v1.64.1
 	github.com/coreos/go-oidc/v3 v3.11.0
 	github.com/go-python/gpython v0.2.0
 	github.com/gofiber/fiber/v2 v2.52.5
@@ -36,7 +36,7 @@ require (
 	github.com/spf13/viper v1.19.0
 	github.com/stretchr/testify v1.9.0
 	github.com/zeebo/assert v1.3.0
-	google.golang.org/api v0.188.0
+	google.golang.org/api v0.199.0
 	gorm.io/driver/postgres v1.5.9
 	gorm.io/driver/sqlite v1.4.3
 	gorm.io/gorm v1.25.11
@@ -46,53 +46,67 @@ require (
 require github.com/go-jose/go-jose/v4 v4.0.2 // indirect
 
 require (
-	cloud.google.com/go/auth v0.7.0 // indirect
-	cloud.google.com/go/auth/oauth2adapt v0.2.2 // indirect
+	cel.dev/expr v0.16.1 // indirect
+	cloud.google.com/go/auth v0.9.5 // indirect
+	cloud.google.com/go/auth/oauth2adapt v0.2.4 // indirect
+	cloud.google.com/go/monitoring v1.21.0 // indirect
 	filippo.io/edwards25519 v1.1.0 // indirect
+	github.com/GoogleCloudPlatform/opentelemetry-operations-go/detectors/gcp v1.24.1 // indirect
+	github.com/GoogleCloudPlatform/opentelemetry-operations-go/exporter/metric v0.48.1 // indirect
+	github.com/GoogleCloudPlatform/opentelemetry-operations-go/internal/resourcemapping v0.48.1 // indirect
+	github.com/census-instrumentation/opencensus-proto v0.4.1 // indirect
+	github.com/cespare/xxhash/v2 v2.3.0 // indirect
+	github.com/cncf/xds/go v0.0.0-20240905190251-b4127c9b8d78 // indirect
+	github.com/envoyproxy/go-control-plane v0.13.0 // indirect
+	github.com/envoyproxy/protoc-gen-validate v1.1.0 // indirect
 	github.com/go-jose/go-jose/v3 v3.0.1 // indirect
 	github.com/golang-jwt/jwt/v5 v5.2.0 // indirect
+	github.com/planetscale/vtprotobuf v0.6.1-0.20240319094008-0393e58bdf10 // indirect
 	github.com/sosodev/duration v1.2.0 // indirect
+	go.opentelemetry.io/contrib/detectors/gcp v1.29.0 // indirect
+	go.opentelemetry.io/otel/sdk v1.29.0 // indirect
+	go.opentelemetry.io/otel/sdk/metric v1.29.0 // indirect
+	google.golang.org/grpc/stats/opentelemetry v0.0.0-20240907200651-3ffb98b2c93a // indirect
 )
 
 require (
-	cloud.google.com/go v0.115.0 // indirect
-	cloud.google.com/go/compute/metadata v0.4.0 // indirect
-	cloud.google.com/go/iam v1.1.10 // indirect
+	cloud.google.com/go v0.115.1 // indirect
+	cloud.google.com/go/compute/metadata v0.5.2 // indirect
+	cloud.google.com/go/iam v1.2.1 // indirect
 	github.com/99designs/gqlgen v0.17.44 // indirect
 	github.com/Khan/genqlient v0.7.0 // indirect
 	github.com/adrg/xdg v0.4.0 // indirect
 	github.com/andybalholm/brotli v1.0.5 // indirect
 	github.com/andybalholm/cascadia v1.3.2 // indirect
-	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.3 // indirect
-	github.com/aws/aws-sdk-go-v2/credentials v1.17.27 // indirect
-	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.11 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.15 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.15 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.0 // indirect
-	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.15 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.3 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.17 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.17 // indirect
-	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.15 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sso v1.22.4 // indirect
-	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.26.4 // indirect
-	github.com/aws/aws-sdk-go-v2/service/sts v1.30.3 // indirect
-	github.com/aws/smithy-go v1.20.3 // indirect
+	github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.5 // indirect
+	github.com/aws/aws-sdk-go-v2/credentials v1.17.38 // indirect
+	github.com/aws/aws-sdk-go-v2/feature/ec2/imds v1.16.14 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/configsources v1.3.18 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/endpoints/v2 v2.6.18 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/ini v1.8.1 // indirect
+	github.com/aws/aws-sdk-go-v2/internal/v4a v1.3.18 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/accept-encoding v1.11.5 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/checksum v1.3.20 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/presigned-url v1.11.20 // indirect
+	github.com/aws/aws-sdk-go-v2/service/internal/s3shared v1.17.18 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sso v1.23.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/ssooidc v1.27.4 // indirect
+	github.com/aws/aws-sdk-go-v2/service/sts v1.31.4 // indirect
+	github.com/aws/smithy-go v1.21.0 // indirect
 	github.com/davecgh/go-spew v1.1.2-0.20180830191138-d8f796af33cc // indirect
 	github.com/felixge/httpsnoop v1.0.4 // indirect
 	github.com/fsnotify/fsnotify v1.7.0 // indirect
-	github.com/go-logr/logr v1.4.1 // indirect
+	github.com/go-logr/logr v1.4.2 // indirect
 	github.com/go-logr/stdr v1.2.2 // indirect
 	github.com/go-sql-driver/mysql v1.8.1 // indirect
 	github.com/goccy/go-json v0.10.2 // indirect
 	github.com/gofiber/template v1.8.3 // indirect
 	github.com/gofiber/utils v1.1.0 // indirect
 	github.com/golang/groupcache v0.0.0-20210331224755-41bb18bfe9da // indirect
-	github.com/golang/protobuf v1.5.4 // indirect
 	github.com/google/flatbuffers v23.5.26+incompatible // indirect
-	github.com/google/s2a-go v0.1.7 // indirect
-	github.com/googleapis/enterprise-certificate-proxy v0.3.2 // indirect
-	github.com/googleapis/gax-go/v2 v2.12.5 // indirect
+	github.com/google/s2a-go v0.1.8 // indirect
+	github.com/googleapis/enterprise-certificate-proxy v0.3.4 // indirect
+	github.com/googleapis/gax-go/v2 v2.13.0 // indirect
 	github.com/hashicorp/hcl v1.0.0 // indirect
 	github.com/inconshreveable/mousetrap v1.1.0 // indirect
 	github.com/jackc/pgpassfile v1.0.0 // indirect
@@ -126,28 +140,28 @@ require (
 	github.com/vektah/gqlparser/v2 v2.5.11 // indirect
 	github.com/zeebo/xxh3 v1.0.2 // indirect
 	go.opencensus.io v0.24.0 // indirect
-	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.49.0 // indirect
-	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.49.0 // indirect
-	go.opentelemetry.io/otel v1.26.0 // indirect
-	go.opentelemetry.io/otel/metric v1.26.0 // indirect
-	go.opentelemetry.io/otel/trace v1.26.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc v0.54.0 // indirect
+	go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.54.0 // indirect
+	go.opentelemetry.io/otel v1.29.0 // indirect
+	go.opentelemetry.io/otel/metric v1.29.0 // indirect
+	go.opentelemetry.io/otel/trace v1.29.0 // indirect
 	go.uber.org/atomic v1.9.0 // indirect
 	go.uber.org/multierr v1.9.0 // indirect
-	golang.org/x/crypto v0.25.0 // indirect
+	golang.org/x/crypto v0.27.0 // indirect
 	golang.org/x/exp v0.0.0-20231110203233-9a3e6036ecaa
 	golang.org/x/mod v0.17.0 // indirect
-	golang.org/x/net v0.27.0 // indirect
-	golang.org/x/oauth2 v0.21.0
-	golang.org/x/sync v0.7.0 // indirect
-	golang.org/x/sys v0.22.0 // indirect
-	golang.org/x/text v0.16.0 // indirect
-	golang.org/x/time v0.5.0 // indirect
+	golang.org/x/net v0.29.0 // indirect
+	golang.org/x/oauth2 v0.23.0
+	golang.org/x/sync v0.8.0 // indirect
+	golang.org/x/sys v0.25.0 // indirect
+	golang.org/x/text v0.18.0 // indirect
+	golang.org/x/time v0.6.0 // indirect
 	golang.org/x/tools v0.21.1-0.20240508182429-e35e4ccd0d2d // indirect
 	golang.org/x/xerrors v0.0.0-20231012003039-104605ab7028 // indirect
-	google.golang.org/genproto v0.0.0-20240708141625-4ad9e859172b // indirect
-	google.golang.org/genproto/googleapis/api v0.0.0-20240701130421-f6361c86f094 // indirect
-	google.golang.org/genproto/googleapis/rpc v0.0.0-20240708141625-4ad9e859172b // indirect
-	google.golang.org/grpc v1.64.1 // indirect
+	google.golang.org/genproto v0.0.0-20240903143218-8af14fe29dc1 // indirect
+	google.golang.org/genproto/googleapis/api v0.0.0-20240903143218-8af14fe29dc1 // indirect
+	google.golang.org/genproto/googleapis/rpc v0.0.0-20240903143218-8af14fe29dc1 // indirect
+	google.golang.org/grpc v1.67.0 // indirect
 	google.golang.org/protobuf v1.34.2 // indirect
 	gopkg.in/ini.v1 v1.67.0 // indirect
 	gopkg.in/yaml.v3 v3.0.1