Skip to content

Commit

Permalink
Merge pull request #349 from pivotal-cf/feat/world-class-dev-experience
Browse files Browse the repository at this point in the history 
Feat/world class dev experience
  • Loading branch information
@wayneadams
wayneadams authored Mar 15, 2023
2 parents 238b901 + 80dc459 commit 7d150f5
Show file tree
Hide file tree
Showing 2,738 changed files with 1,356,904 additions and 519 deletions.
5 changes: 5 additions & 0 deletions .github/workflows/test.yml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -33,6 +33,11 @@ jobs:
- name: Build
run: go build -v ./...

- name: Setup SSH
uses: webfactory/[email protected]
with:
ssh-private-key: ${{ secrets.RELENG_CI_BOT_KEY }}

- name: Unit Tests
run: |
go test ./...
2 changes: 2 additions & 0 deletions .gitignore
View file
Original file line number Diff line number Diff line change
Expand Up @@ -13,3 +13,5 @@ env.yml
*.key
*.pem
.vscode

vendor
3 changes: 3 additions & 0 deletions .gitmodules
View file
Original file line number Diff line number Diff line change
@@ -1,3 +1,6 @@
[submodule "internal/acceptance/workflows/hello-tile"]
path = internal/acceptance/workflows/hello-tile
url = [email protected]:crhntr/hello-tile
[submodule "internal/commands/testdata/test_tile/hello-tile"]
path = internal/commands/testdata/test_tile/hello-tile
url = [email protected]:crhntr/hello-tile.git
242 changes: 133 additions & 109 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -81,126 +81,23 @@ Commands:
version prints the kiln release version
```
### `fetch`
The `fetch` command downloads bosh release tarballs from an AWS S3 bucket to a
a local directory specified by the `--releases-directory` flag. It discovers
releases based on information from both the Kilnfile and an Kilnfile.lock file.
The Kilnfile.lock file name is expected to be a file in the same directory as the
specified Kilnfile with `lock` as as the filename extension.
The S3 object name is determined based on using regular expression capture
groups.
Kiln verifies that the checksum (SHA1) of the downloaded release matches
checksum specified for the release in the Kilnfile.lock file. If the checksums do
not match, then the releases that don't match will be deleted from disk. *Since
BOSH releases from different directors with the same packages result in complied
releases with different hashes this may result in some problems where if you
download a release that was compiled with a different director those releases
will be deleted.*
Kiln will not download releases if an existing release exists with the correct
release version and checksum.
#### Kilnfile
The Kilnfile must also have information about how to access the S3 Bucket.
Two types of release sources are allowed in the list under the `release_sources`
key:
1. `type: bosh.io`. For this type, no other keys are required/allowed.
2. `type: s3`. The following other keys **required** in this case.
- `publishable` (boolean): true if this bucket contains releases that are suitable to ship to customers
- `bucket`: must be the name of the s3 bucket
- `region`: must be the region of the bucket
- `access_key_id`: must be an IAM access key id that has read permission for the
specified bucket
- `secret_access_key`: must be the secret for the specified `access_key_id`
- `release_path:`: a (text/template package) template expression used to build the
full-path to a release in the S3 bucket. The template should evaluate to the exact
path within the s3 bucket for a given release name+version+stemcell combination.
The template has access to the following fields:
- release name (e.g. `{{.Name}}`)
- release version (e.g. `{{.Version}}`)
- stemcell OS (e.g. `{{.StemcellOS}}`)
- stemcell version (e.g. `{{.StemcellVersion}}`)
- There's also access to a `trimSuffix` helper (e.g. `{{trimSuffix .Name "-release"}}`)
### Kilnfile.lock
This file contains the full list of specific versions of all releases that will
go into the tile AND the target stemcell.
Currently the releases for the Kilnfile.lock file can not be generated by kiln.
The update command is in development and only (loosely) supports updating the
stemcell based on stemcells on https://network.pivotal.io. On PAS Release
Engineering we use a consourse task in our CI to generate the Kilnfile.lock
file.
The file has two top level members `releases` and `stemcell_criteria`.
The `releases` member is an array of members with each element having the following members.
- `name`: bosh release name
- `sha1`: checksum of the tarball
- `version`: semantic version of the release
The `stemcell_criteria ` member is an array of members with each element having the following members.
- `name`: bosh release name
- `sha1`: checksum of the tarball
- `version`: semantic version of the release
### Example with Variable Interpolation
```
$ cat Kilnfile
release_sources:
- type: s3
compiled: true
bucket: compiled-releases
region: us-west-1
access_key_id: $(variable "aws_access_key_id")
secret_access_key: $(variable "aws_secret_access_key")
path_template: 2.6/{{trimSuffix .Name "-release"}}/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz
```
*Credentials like S3 keys are not stored in git repos. To support separating
that information from non-sensitive configuration, you can reference variables
like you do in tile config.*
```
$ lpass show --notes 'pas-releng-fetch-releases'
---
aws_access_key_id: SOME_REALLY_SECRET_ID
aws_secret_access_key: SOME_REALLY_SECRET_KEY
```
Interpolating this file in kiln would look something like this.
```
kiln fetch --kilnfile random-Kilnfile --variables-file <(lpass show --notes 'pas-releng-fetch-releases')
```
### `bake`
It takes release and stemcell tarballs, metadata YAML, and JavaScript migrations
as inputs and produces an OpsMan-compatible tile as its output.
Here is an example command line:
The produce a tile, you simply need to be within a tile directory and execute the following command:
```
$ kiln bake \
--version 2.0.0 \
--metadata /path/to/metadata.yml \
--releases-directory /path/to/releases \
--stemcells-directory /path/to/stemcells/first \
--stemcells-directory /path/to/stemcells/second \
--migrations-directory /path/to/migrations \
--output-file /path/to/cf-2.0.0-build.4.pivotal
$ kiln bake
```
This will ensure that you have the necessary releases by first calling `kiln fetch`.
Refer to the [example-tile](example-tile) for a complete example showing the
different features kiln supports.
#### Options
<details>
<summary>Additional bake options</summary>
##### `--bosh-variables-directory`
Expand Down Expand Up @@ -484,6 +381,133 @@ provides_product_versions:
version: $( version )
```
</details>
### `test`
The `test` command exercises to ginkgo tests under the `/<tile>/test/manifest` path of the `pivotal/tas` repos (where `<tile>` is tas, ist, or tasw).
Running these tests require a docker daemon to be running.
Here is an example command line:
```
$ kiln test \
--ginkgo-manifest-flags "-randomizeAllSpecs -randomizeAllSpecs" \
--tile-path ist
```
<details>
<summary>Additional test options</summary>
##### `--ginkgo-manifest-flags`
The `--ginkgo-manifest-flags` flag can be used to pass through Ginkgo test flags. The defaults being passed through are `-r -slowSpecThreshold 15`. Pass `help` as a flag to retrieve the available options for the embeded version of ginkgo.
##### `--tile-path`
The `--tile-path` flag can be set the path the the directory you wish to test. It defaults to the current working directory.
</details>
### `fetch`
The `fetch` command downloads bosh release tarballs from an AWS S3 bucket to a
a local directory specified by the `--releases-directory` flag. It discovers
releases based on information from both the Kilnfile and an Kilnfile.lock file.
The Kilnfile.lock file name is expected to be a file in the same directory as the
specified Kilnfile with `lock` as as the filename extension.
The S3 object name is determined based on using regular expression capture
groups.
Kiln verifies that the checksum (SHA1) of the downloaded release matches
checksum specified for the release in the Kilnfile.lock file. If the checksums do
not match, then the releases that don't match will be deleted from disk. *Since
BOSH releases from different directors with the same packages result in complied
releases with different hashes this may result in some problems where if you
download a release that was compiled with a different director those releases
will be deleted.*
Kiln will not download releases if an existing release exists with the correct
release version and checksum.
#### Kilnfile
The Kilnfile must also have information about how to access the S3 Bucket.
Two types of release sources are allowed in the list under the `release_sources`
key:
1. `type: bosh.io`. For this type, no other keys are required/allowed.
2. `type: s3`. The following other keys **required** in this case.
- `publishable` (boolean): true if this bucket contains releases that are suitable to ship to customers
- `bucket`: must be the name of the s3 bucket
- `region`: must be the region of the bucket
- `access_key_id`: must be an IAM access key id that has read permission for the
specified bucket
- `secret_access_key`: must be the secret for the specified `access_key_id`
- `release_path:`: a (text/template package) template expression used to build the
full-path to a release in the S3 bucket. The template should evaluate to the exact
path within the s3 bucket for a given release name+version+stemcell combination.
The template has access to the following fields:
- release name (e.g. `{{.Name}}`)
- release version (e.g. `{{.Version}}`)
- stemcell OS (e.g. `{{.StemcellOS}}`)
- stemcell version (e.g. `{{.StemcellVersion}}`)
- There's also access to a `trimSuffix` helper (e.g. `{{trimSuffix .Name "-release"}}`)
### Kilnfile.lock
This file contains the full list of specific versions of all releases that will
go into the tile AND the target stemcell.
Currently the releases for the Kilnfile.lock file can not be generated by kiln.
The update command is in development and only (loosely) supports updating the
stemcell based on stemcells on https://network.pivotal.io. On PAS Release
Engineering we use a consourse task in our CI to generate the Kilnfile.lock
file.
The file has two top level members `releases` and `stemcell_criteria`.
The `releases` member is an array of members with each element having the following members.
- `name`: bosh release name
- `sha1`: checksum of the tarball
- `version`: semantic version of the release
The `stemcell_criteria ` member is an array of members with each element having the following members.
- `name`: bosh release name
- `sha1`: checksum of the tarball
- `version`: semantic version of the release
### Example with Variable Interpolation
```
$ cat Kilnfile
release_sources:
- type: s3
compiled: true
bucket: compiled-releases
region: us-west-1
access_key_id: $(variable "aws_access_key_id")
secret_access_key: $(variable "aws_secret_access_key")
path_template: 2.6/{{trimSuffix .Name "-release"}}/{{.Name}}-{{.Version}}-{{.StemcellOS}}-{{.StemcellVersion}}.tgz
```
*Credentials like S3 keys are not stored in git repos. To support separating
that information from non-sensitive configuration, you can reference variables
like you do in tile config.*
```
$ lpass show --notes 'pas-releng-fetch-releases'
---
aws_access_key_id: SOME_REALLY_SECRET_ID
aws_secret_access_key: SOME_REALLY_SECRET_KEY
```
Interpolating this file in kiln would look something like this.
```
kiln fetch --kilnfile random-Kilnfile --variables-file <(lpass show --notes 'pas-releng-fetch-releases')
```
### Template functions
#### `select`
Expand Down
Loading

0 comments on commit 7d150f5

Please sign in to comment.