Skip to content

Commit

Permalink
Merge pull request #52 from AcalephStorage/release/0.1.3
Browse files Browse the repository at this point in the history 
Release/0.1.3
  • Loading branch information
@madziefe
madziefe committed May 23, 2016
2 parents eae0280 + a3e944d commit 19c5223
Show file tree
Hide file tree
Showing 25 changed files with 1,949 additions and 349 deletions.
7 changes: 3 additions & 4 deletions Dockerfile
View file
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM golang:1.6-alpine
FROM alpine:edge

ENV GOPATH /go
ENV SWAGGER_UI /swagger/dist
Expand All @@ -10,12 +10,11 @@ RUN mkdir /swagger && tar xvzf third_party/swagger.tar.gz -C /swagger

# create and remove downloaded libraries
RUN apk update && \
apk add make git && \
apk add make git go ca-certificates && \
make && \
mv build/bin/kontinuous /bin && \
mv build/bin/kontinuous-cli /bin && \
rm -rf /go && \
apk del --purge make git && \
apk del --purge git && \
rm -rf /var/cache/apk/*

EXPOSE 3005
Expand Down
237 changes: 32 additions & 205 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -26,88 +26,47 @@ We've got lots more planned, see the [Roadmap](#roadmap) or Github issues to get

### Getting Started

The script `scripts/kontinuous-deploy` is a quick way of running `kontinuous` in a K8s cluster. The general syntax is:
Before running Kontinuous, it needs to be added as a github OAuth Application [here](https://github.com/settings/applications/new). The `Client ID` and `Client Secret` will be used in running Kontinuous.

The `kubernetes-cli` can bootstrap a kontinuous setup on a running Kubernetes cluster. This requires `kubectl` to be in the `PATH` and configured to access the cluster.

```
$ kontinuous-deploy --namespace {k8s-namespace} --auth-secret {base64url encoded secret} --s3-access-key {s3 access key} --s3-secret-key {s3 secret key}
$ kontinuous-cli --namespace {namespace} \
--auth-secret {base64 encoded secret} \
--github-client-id {github client id} \
--github-client-secret {github client secret}
```

This will launch `kontinuous` via the locally configured `kubectl` in the given namespace together with `etcd`, `minio`, and a docker `registry`. This expects that the kubernetes cluster supports the LoadBalancer service.

Alternatively, for more customization, a sample yaml file for running kontinuous and its dependencies in Kubernetes can be found [here](./k8s-spec.yml.example). See [below](#running-in-kubernetes) for how to configure secrets.

Once running, add a [.pipeline.yml](#pipeline-spec) to the root of your Github repo and configure the webhooks.

Example pipelines can be found in [/examples](./examples)

The [CLI client](#clients) or [API](#api) can be used to view build status or logs.

### Dependencies

Running kontinuous requires the following to be setup:

- **etcd**

`etcd` is used as a backend for storing pipeline and build details. This is a dedicated instance to avoid poluting with the Kubernetes etcd cluster.

- **minio**
Parameters:

`minio` is used to store the logs and artifacts. S3 could also be used as it is compatible with `minio`, although this has not been tested yet.

- **docker registry**
| parameter | description |
|------------------------|------------------------------------------------------------------------------------------------------------------------------|
| --namespace | The namespace to deploy Kontinuous to. This defaults to `kontinuous` |
| --auth-secret | A base64 encoded secret. This is used by kontinuous to provide JWT for authentication. This can be any base64 encoded string |
| --github-client-id | The Github client ID provided when registering kontinuous as a Github OAuth application |
| --github-client-secret | The Github client secret provided when registering kontinuous as a Github OAuth application |

`registry` is used to store internal docker images.

- **kubernetes**

Kontinuous uses Kubernetes Jobs heavily so will require at least version 1.1 with Jobs enabled


### Running in Kubernetes

Kontinuous is meant to run inside a kubernetes cluster, preferrably by a Deployment or Replication Controller.

The docker image can be found here: [quay.io/acaleph/kontinuous](https://quay.io/acaleph/kontinuous)

The following environment variables needs to be defined:

| Environment Variable | Description | Example |
|----------------------|-----------------------------------------|------------------------|
| KV_ADDRESS | The etcd address | etcd:2379 |
| S3_URL | The minio address | http://minio:9000 |
| KONTINUOUS_URL | The address where kontinuous is running | http://kontinuous:3005 |
| INTERNAL_REGISTRY | The internal registry address | internal-registry:5000 |

A Kubernetes Secret also needs to be defined and mounted to the Pod. The secret should have a key named `kontinuous-secrets` and should contain the following data (must be base64 encoded):

```json
{
"AuthSecret": "base64 encoded auth secret",
"S3SecretKey": "s3 secret key",
"S3AccessKey": "s3 access key"
}
```

`AuthSecret` is the secret for authenticating requests. This is needed by the clients to communicate with kontinuous through JWT.
This will launch `kontinuous` via the locally configured `kubectl` in the given namespace together with `etcd`, `minio`, a docker `registry`, and `kontinuous-ui`. This expects that the kubernetes cluster supports the LoadBalancer service.

`S3SecretKey` and `S3AccessKey` are the keys needed to access minio (or S3).
Once a public IP for `kontinuous-ui` is available, the Github OAuth Application settings needs to be modified to reflect the actual IP address of `kontinuous-ui` for the Homepage and Callback URL.

The secret needs to be mounted to the Pod to the path `/.secret`.
Alternatively, for more customization, a sample yaml file for running kontinuous and its dependencies in Kubernetes can be found [here](./k8s-spec.yml.example). More details can be found [here](docs/setup.md).

## Using Kontinuous
Once running, add a [.pipeline.yml](#pipeline-spec) to the root of your Github repo and configure the webhooks.

### Preparing the repository
Example pipelines can be found in [/examples](./examples)

#### Pipeline Spec
The [CLI client](#clients) or [API](#api) can be used to view build status or logs.

The repository needs to define a build pipeline in the repository root called `.pipeline.yml`
## Pipeline Specification

Here's a sample `.pipeline.yml`:
Pipeline specification should be at the root directory of the repository. This defines the stages of the builds. More details about pipeline spec creation can be found [here](docs/pipeline.md).

```yaml
---
apiVersion: v1alpha1
kind: Pipeline
apiVersion: v1alpha1
metadata:
name: kontinuous
namespace: acaleph
Expand All @@ -124,165 +83,33 @@ spec:
type: ci-cd
notif:
- type: slack
metadata:
url: slackurl #taken from secret
username: slackuser #taken from secret
channel: slackchannel #taken from secret
secrets:
- notifcreds
- docker-credentials
stages:
- name: Build Docker Image
type: docker_build
- name: Unit Test
type: command
params:
command:
- make
- test
- name: Publish to Quay
type: docker_publish
params:
external_registry: quay.io
external_image_name: acaleph/kontinuous
require_credentials: "TRUE"
username: user # taken from secret
password: password # taken from secret
email: email # taken from secret
```
The format is something similar to K8s Specs. Here are more details on some of the fields:
- `namespace` - the namespace to run the build
- `matchLabels`/`labels` - the labels that are used for building the job
- `stages` - defines the stages of the build pipeline

The general definition of a stage is:

```yaml
name: Friendly name
type: {docker_build,command,docker_publish}
params:
key: value
```
- `type` can be: `docker_build`, `docker_publish`, `command`, or `deploy`.
- `params` is a map of parameters to be loaded as environment variables.

#### Notification

- `type` can be: `slack`.
- `metadata` is a map of values needed for a certain notification type. The metadata value should be a **key** from one of the secrets file defined

`metadata` of notification `type=slack` has the following keys:

- `url` is a slack incoming messages webhook url.
- `channel` is optional. If set, it will override default channel
- `username` is optional. If set, it will override default username
In the future releases, kontinuous notification will support other notification services. e.g. email, hipchat, etc.

### Secrets

- `secrets` is a list of secrets that will be used as values for stages and notification.

#### Stages

`docker_build` builds a Docker Image and pushes the images to a internal registry. It can work without additional params. By default, it uses the `Dockerfile` inside the repository root. Optional params are:

- `dockerfile_path` - the path where the Dockerfile is located
- `dockerfile_name` - the file name of the Dockerfile

After a build, the image is stored inside the internal docker registry.
This example only has one stage, build a docker image.
`docker_publish` pushes the previously build Docker image to an external registry. It requires the following params:

- `external_registry` - the external registry name (eg. quay.io)
- `external_image_name` - the name of the image (eg. acaleph/kontinuous)

Optional params:

- `require_crendentials` - defaults to `false`. Set to `true` if registry requires authentication

Required secrets:

- `dockeruser`
- `dockerpass`
- `dockeremail`

These secrets needs to be defined in at least one of the secrets provided.

The image that will be pushed is the image that was previously built. This does not work for now if no image was created.

`command` runs a command on the newly create docker image or on the image specified. Required param is `command` which is a list of string defining the command to execute.

Optional params are:

- `args` - a list of string to serve as the arguments for the command
- `image` - the image to run the commands in. If not specified, the previous built image will be used.
- `dependencies` - a list of Kubernetes spec files to run as dependencies for running the command. Useful when running integration tests.

`deploy` deploys a Kubernetes Spec file (yaml) to kubernetes.


### Authentication

There are currently two ways of authenticating with kontinuous. One is by using Github's OAuth Web Application Flow and the other one is with JSON Web Tokens.

#### Github OAuth

This is the authorization process used by `kontinuous-ui`. This is a 3 step process detailed [here](https://developer.github.com/v3/oauth/#web-application-flow) with a slightly different variation:

1. Kontinuous needs to be registered as a Github OAuth Application [here](https://github.com/settings/applications/new).

2. Redirect users to request Github Access (step 1 in web application flow):

```
GET https://github.com/login/oauth/authorize
```

3. Send authorization code to Kontinuous:

```
POST {kontinuous-url}/api/v1/login/github?code={auth_code}&state={state_from_step_1}
```

This should return a JSON Web Token in the body that can be used to authenticate further requests.


#### JSON Web Token

Currently, only Github Repositories are supported. A github token needs to be generated in order to access the repositories.
## Clients
To generate a github token, follow this [link](https://github.com/settings/tokens/new).
There are two clients currently available:
Make sure to enable access to the following:
### Kontinuous CLI
- repo
- admin:repo_hook
- user
The CLI tool is the one that is used in the gettings started section. It can bootstrap Kontinuous to a running Kubernetes Cluster and can access details on Kontinuous pipelines and builds.
The script `scripts/jwt-gen` can generate a JSON Web Token to be used for authentication with Kontinuous.
More info about the CLI can be found [here](https://github.com/AcalephStorage/kontinuous/tree/develop/cli) and the binary can be downloaded [here](https://github.com/AcalephStorage/kontinuous/releases).
```console
$ scripts/jwt-gen --secret {base64url encoded secret} --github-token {github-token}
```
### Kontinuous UI
This generates a JSON Web Token and can be added to the request header as `Authorization: Bearer {token}` to authenticate requests.

The generated token's validity can be verified at [jwt.io](https://jwt.io).
Kontinuous UI is a web based client for Kontinuous. Bootstrapping Kontinuous using the CLI will install the UI on the Kubernetes Cluster too. More info about the UI can be found [here](https://github.com/AcalephStorage/kontinuous-ui).
## API
kontinuous is accessible from it's API and docs can be viewed via Swagger.

The API doc can be accessed via `{kontinuous-address}/apidocs`

## Clients

At the moment, there is a basic cli client binary [here](https://github.com/AcalephStorage/kontinuous/releases) and code available [here](https://github.com/AcalephStorage/kontinuous/tree/develop/cli).

A Web based Dashboard is under development.
Kontinuous is accessible from it's API and docs can be viewed via Swagger. More details about using the API and Authentication can be found [here](docs/api.md).
## Development
Expand Down
4 changes: 2 additions & 2 deletions api/auth.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -115,8 +115,8 @@ func (a *AuthResource) githubLogin(req *restful.Request, res *restful.Response)
Path: "login/oauth/access_token",
}
q := reqUrl.Query()
q.Set("client_id", os.Getenv("GH_CLIENT_ID"))
q.Set("client_secret", os.Getenv("GH_CLIENT_SECRET"))
q.Set("client_id", os.Getenv("GITHUB_CLIENT_ID"))
q.Set("client_secret", os.Getenv("GITHUB_CLIENT_SECRET"))
q.Set("code", authCode)
q.Set("state", state)
reqUrl.RawQuery = q.Encode()
Expand Down
31 changes: 31 additions & 0 deletions api/build.go
View file
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,16 @@ func (b *BuildResource) extend(ws *restful.WebService) {
Param(ws.PathParameter("buildNumber", "build number").DataType("int")).
Writes(ps.Build{}).
Filter(requireAccessToken))

ws.Route(ws.DELETE("/{owner}/{repo}/builds/{buildNumber}").To(b.delete).
Doc("Remove build details").
Operation("delete").
Param(ws.PathParameter("owner", "repository owner name").DataType("string")).
Param(ws.PathParameter("repo", "repository name").DataType("string")).
Param(ws.PathParameter("buildNumber", "build number").DataType("int")).
Writes(ps.Build{}).
Filter(requireAccessToken))

}

func (b *BuildResource) create(req *restful.Request, res *restful.Response) {
Expand Down Expand Up @@ -177,6 +187,27 @@ func (b *BuildResource) create(req *restful.Request, res *restful.Response) {
res.WriteEntity(build)
}

func (b *BuildResource) delete(req *restful.Request, res *restful.Response) {

owner := req.PathParameter("owner")
repo := req.PathParameter("repo")
buildNumber := req.PathParameter("buildNumber")
pipeline, err := findPipeline(owner, repo, b.KVClient)

if err != nil {
jsonError(res, http.StatusNotFound, err, fmt.Sprintf("Unable to find pipeline %s/%s", owner, repo))
return
}

build, err := findBuild(buildNumber, pipeline, b.KVClient)
if err != nil {
jsonError(res, http.StatusNotFound, err, fmt.Sprintf("Unable to find build %s for %s/%s", buildNumber, owner, repo))
return
}

build.Delete(pipeline.ID, b.KVClient, b.MinioClient)
}

func (b *BuildResource) list(req *restful.Request, res *restful.Response) {
owner := req.PathParameter("owner")
repo := req.PathParameter("repo")
Expand Down
Loading

0 comments on commit 19c5223

Please sign in to comment.