INSTALL_UBUNTU16.md

devstats installation on Ubuntu

Prerequisites:

• Ubuntu 16.04 LTS (quite old, but longest support).
• Some of the operations below can be CPU/RAM intensive. It is recommended to use a minimum of 8 cores and 30GB RAM or higher.
• Make sure you have enough disk space for postgres database. This tutorial used 50GB.
• golang, this tutorial uses Go 1.9 - link
  • sudo apt-get update
  • sudo apt-get install golang-1.9-go git psmisc jsonlint yamllint gcc
  • sudo ln -s /usr/lib/go-1.9 /usr/lib/go
  • mkdir $HOME/data; mkdir $HOME/data/dev
• Update git to version 2.11.0 or above :
  • sudo add-apt-repository ppa:git-core/ppa -y
  • sudo apt-get update
  • sudo apt-get install git -y
  • git --version

1. Configure Go:

   • For example add to ~/.bash_profile and/or ~/.profile:

   GOROOT=/usr/lib/go; export GOROOT
   GOPATH=$HOME/data/dev; export GOPATH
   PATH=$PATH:$GOROOT/bin:$GOPATH/bin; export PATH
   

   • Logout and login again.
   • golint: go get -u golang.org/x/lint/golint
   • goimports: go get golang.org/x/tools/cmd/goimports
   • goconst: go get github.com/jgautheron/goconst/cmd/goconst
   • usedexports: go get github.com/jgautheron/usedexports
   • errcheck: go get github.com/kisielk/errcheck
   • cors: go get -u github.com/rs/cors
   • jsoniter: go get -u github.com/json-iterator/go
   • Go Postgres client: install with: go get github.com/lib/pq
   • Go unicode text transform tools: install with: go get golang.org/x/text/transform and go get golang.org/x/text/unicode/norm
   • Go YAML parser library: install with: go get gopkg.in/yaml.v2
   • Go GitHub API client: go get github.com/google/go-github/github
   • Go OAuth2 client: go get golang.org/x/oauth2
   • Go SQLite3 client: go get github.com/mattn/go-sqlite3

2. Go to $GOPATH/src/github.com/cncf and clone devstats and devstatscode there:

   • git clone https://github.com/cncf/devstats.git.
   • git clone https://github.com/cncf/devstatscode.git.
   • cd devstats.
   • Set reuse TCP connections: ./cron/sysctl_config.sh

3. If you want to make changes and PRs, please clone devstats and devstatscode from GitHub UI, and clone your forked version instead, like this:

   • git clone https://github.com/your_github_username/devstats.git
   • git clone https://github.com/your_github_username/devstatscode.git

4. Go to devstatscode directory, so you are in ~/dev/go/src/github.com/cncf/devstatscode directory and compile binaries:

   • make

5. If compiled sucessfully then execute test coverage that doesn't need databases:

   • make test
   • Tests should pass.

6. Install binaries:

   • sudo mkdir /etc/gha2db
   • sudo chmod 777 /etc/gha2db
   • make install
   • sudo make install

7. Go to devstats directory, so you are in ~/dev/go/src/github.com/cncf/devstats and install scripts, metrics and other config/data files:

   • make install

8. Install Postgres database (link):

   • Make sure you database uses en_US.UTF-8 locale and collation.
   • apt-get install postgresql
   • sudo -i -u postgres
   • psql
   • Postgres only allows local connections by default so it is secure, we don't need to disable external connections:
   • Instructions to enable external connections (not recommended): http://www.thegeekstuff.com/2014/02/enable-remote-postgresql-connection/?utm_source=tuicool
   • Set bigger maximum number of connections, at least 200 or more: /etc/postgresql/X.Y/main/postgresql.conf. Default is 100. max_connections = 300.
   • You can also set shared_buffers = ... to something like 25% of your RAM. This is optional.
   • ./devel/set_psql_password.sh to set postgres user password.
   • chown -R postgres /var/log/postgresql.

9. Inside psql client shell:

   • create database gha;
   • create database devstats;
   • create user gha_admin with password 'your_password_here';
   • grant all privileges on database "gha" to gha_admin;
   • grant all privileges on database "devstats" to gha_admin;
   • alter user gha_admin createdb;
   • create extension if not exists pgcrypto;
   • Leave the shell and create logs table for devstats: sudo -u postgres psql devstats < util_sql/devstats_log_table.sql.
   • PG_PASS=... ONLY="devstats gha" ./devel/create_ro_user.sh.
   • PG_PASS=... ONLY="devstats gha" ./devel/create_psql_user.sh devstats_team.
   • In case of problems both scripts (create_ro_user.sh and create_psql_user.sh) support DROP=1, NOCREATE=1 env variables.

10. Leave psql shell, and get newest Kubernetes database dump:

    • wget https://devstats.cncf.io/gha.dump.
    • mv gha.dump /tmp.
    • sudo -u postgres pg_restore -d gha /tmp/gha.dump (restore DB dump)
    • rm /tmp/gha.dump

11. Databases installed, you need to test if all works fine, use database test coverage (on cncf/devstats repo):

    • PG_PASS=your_postgres_pwd make
    • Tests should pass.

12. We have both databases running and Go tools installed, let's try to sync database dump from k8s.devstats.cncf.io manually:

    • Set reuse TCP connections: sudo ./cron/sysctl_config.sh
    • On some VMs tcp_tw_recycle will be unavailable, ignore the warning.
    • We need to prefix call with GHA2DB_LOCAL=1 to enable using tools from "./" directory
    • You need to have GitHub OAuth token, either put this token in /etc/github/oauth file or specify token value via GHA2DB_GITHUB_OAUTH=deadbeef654...10a0 (here you token value)
    • If you want to use multiple tokens, create /etc/github/oauths file that contain list of comma separated OAuth keys or specify token values via GHA2DB_GITHUB_OAUTH=key1,key2,...,keyN
    • If you really don't want to use GitHub OAuth2 token, specify GHA2DB_GITHUB_OAUTH=- - this will force tokenless operation (via public API), it is a lot more rate limited than OAuth2 which gives 5000 API points/h
    • To import time series data for the first time (Postgres database is at the state when Kubernetes SQL dump was made on k8s.devstats.cncf.io):
    • PG_PASS=pwd ONLY=kubernetes ./devel/reinit.sh
    • This can take a while (depending how old is psql dump gha.sql.xz on k8s.devstats.cncf.io. It is generated daily at 3:00 AM UTC.
    • Command should be successfull.

13. We need to setup cron job that will call sync every hour (10 minutes after 1:00, 2:00, ...)

    • You need to open crontab.entry file, it looks like this for single project setup (this is obsolete, please use devstats mode instead):

    8 * * * * PATH=$PATH:/path/to/your/GOPATH/bin GHA2DB_CMDDEBUG=1 GHA2DB_PROJECT=kubernetes PG_PASS='...' gha2db_sync 2>> /tmp/gha2db_sync.err 1>> /tmp/gha2db_sync.log
    20 3 * * * PATH=$PATH:/path/to/your/GOPATH/bin cron_db_backup.sh gha 2>> /tmp/gha2db_backup.err 1>> /tmp/gha2db_backup.log
    */5 * * * * PATH=$PATH:/path/to/your/GOPATH/bin GOPATH=/your/gopath GHA2DB_CMDDEBUG=1 GHA2DB_PROJECT_ROOT=/path/to/repo PG_PASS="..." GHA2DB_SKIP_FULL_DEPLOY=1 webhook 2>> /tmp/gha2db_webhook.err 1>> /tmp/gha2db_webhook.log
    

    • For multiple projects you can use devstats instead of gha2db_sync and cron/cron_db_backup_all.sh instead of cron/cron_db_backup.sh.

    7 * * * * PATH=$PATH:/path/to/GOPATH/bin PG_PASS="..." devstats 2>> /tmp/gha2db_sync.err 1>> /tmp/gha2db_sync.log
    30 3 * * * PATH=$PATH:/path/to/GOPATH/bin cron_db_backup_all.sh 2>> /tmp/gha2db_backup.err 1>> /tmp/gha2db_backup.log
    */5 * * * * PATH=$PATH:/path/to/GOPATH/bin GOPATH=/go/path GHA2DB_CMDDEBUG=1 GHA2DB_PROJECT_ROOT=/path/to/repo GHA2DB_DEPLOY_BRANCHES="production,master" PG_PASS=... GHA2DB_SKIP_FULL_DEPLOY=1 webhook 2>> /tmp/gha2db_webhook.err 1>> /tmp/gha2db_webhook.log
    

    • First crontab entry is for automatic GHA sync.
    • Second crontab entry is for automatic daily backup of GHA database.
    • Third crontab entry is for Continuous Deployment - this a Travis Web Hook listener server, it deploys project when specific conditions are met, details here.
    • You need to change "..." PG_PASS to the real postgres password value and copy this line.
    • You need to change "/path/to/your/GOPATH/bin" to the value of "$GOPATH/bin", you cannot use $GOPATH in crontab directly.
    • Run crontab -e and put this line at the end of file and save.
    • Cron job will update Postgres database at 0:10, 1:10, ... 23:10 every day.
    • It outputs logs to /tmp/gha2db_sync.out and /tmp/gha2db_sync.err and also to gha Postgres database: into table gha_logs.
    • Check database values and logs about 15 minutes after full hours, like 14:15:
    • Check max event created date: select max(created_at) from gha_events and logs select * from gha_logs order by dt desc limit 20.

14. Install Grafana, see MULTIPROJECT.md.

    • Follow: http://docs.grafana.org/installation/debian/
    • wget https://s3-us-west-2.amazonaws.com/grafana-releases/release/grafana_5.x.x_amd64.deb
    • sudo apt-get install -y adduser libfontconfig
    • sudo dpkg -i grafana_5.x.x_amd64.deb
    • sudo service grafana-server start
    • Configure Grafana, as described here.
    • service grafana-server restart
    • Go to Grafana UI (localhost:3000), choose sign out, and then access localhost:3000 again. You should be able to view dashboards as a guest. To login again use http://localhost:3000/login.
    • Install Apache as described here.
    • You can also enable SSL, to do so you need to follow SSL instruction in SSL (that requires domain name).

15. To change all Grafana page titles (starting with "Grafana - ") and icons use this script:

    • GRAFANA_DATA=/usr/share/grafana/ ./grafana/{{project}}/change_title_and_icons.sh.
    • GRAFANA_DATA can also be /usr/share/grafana.prometheus/ for example, see MULTIPROJECT.md.
    • Replace GRAFANA_DATA with your Grafana data directory.
    • service grafana-server restart
    • In some cases browser and/or Grafana cache old settings in this case temporarily move Grafana's settings.js file:
    • mv /usr/share/grafana/public/app/core/settings.js /usr/share/grafana/public/app/core/settings.js.old, restart grafana server and restore file.

16. To enable Continuous deployment using Travis, please follow instructions here.

17. You can create new metrics (as SQL files and YAML definitions) and dashboards in Grafana (export as JSON).

18. PRs and suggestions are welcome, please create PRs and Issues on the GitHub.

More details

• Local Development.
• README
• USAGE