gitlab-ci.yml file can be split to increase readability

[aboe76]

Is your feature request related to a problem?

gitlab-ci.yml is now a big long file with different tasks, which is hard to read and understand.

Describe the solution you'd like

gitlab-ci.yml file has the option to include external yml files:
see: https://docs.gitlab.com/ee/ci/yaml/#include

Especially the local and remote includes are interesting.

Describe alternatives you've considered

not needed

Additional context

improve readability and interoperability between the formulas is also possible.
No need to duplicate code (dry)

[myii]

Thanks for raising this issue, @aboe76. Just some initial responses for now.

• There's no doubt that this initial .gitlab-ci.yml could be improved, since we were in desperate need for a Travis CI replacement, there are bound to be many issues which have crept in.
• Using the include feature is definitely one way this could be tackled. Some thoughts, though:
  • Splitting this into numerous YAML files makes things confusing in other ways, such as linking all the bits together again to see the whole.
  • Where will all of these files be kept? We have already spoken about the file clutter in each of our repos.
  • Making things more DRY is great for the organisation but how about people using this template? The various YAML files may be in different repos across our org.
  • The current .gitlab-ci.yml is a single template managed from the ssf-formula, like .travis.yml and the other files -- splitting these will bring in a lot more complexity.
  • The YAML node anchors cannot be used across the included files, so this will in turn, make things less DRY and scatter the specific configurations throughout the file(s) rather than at the top in one place.
  • Speaking of which, perhaps the YAML node anchors make the file look longer than it really is; comparing with the .travis.yml that is still available, there's not that much difference overall (233 vs. 204 lines) -- perhaps it just looks longer because it's not as familiar as the Travis CI structure we're used to?
• Not sure if this will make much difference but there is also a matrix feature in GitLab CI that should be investigated.

[aboe76]

@myii, if I'm not correct most of the gitlab-ci.yml that doesn't change to often could stay in the ssf-formula and only be included in the main gitlab-ci.yml file directly without the need to copy/merge those files.

this is a more comprehensive example location:
https://docs.gitlab.com/ee/ci/yaml/includes.html

[aboe76]

I'm just learning more from gitlab, because the company I work for has been accepted into the partners.gitlab.com program, so I got a lot of useful information that I still need to figure out, but this was one of the first things that came to mind.

[baby-gnu]

Hello.

In my work, we setup quite a few gitlab CI pipelines and I'm now more confident on how it's working.

I have a personal repository where I'm pushing some common jobs templates.

• Using the include feature is definitely one way this could be tackled. Some thoughts, though:
  • Splitting this into numerous YAML files makes things confusing in other ways, such as linking all the bits together again to see the whole.

I agree, that the reason I only define hidden template jobs to be extended by user in their .gitlab-ci.yaml.

• Where will all of these files be kept? We have already spoken about the file clutter in each of our repos.

I propose to put the ci-tools project under gitlab saltstack-formulas/infrastructure group

• Making things more DRY is great for the organisation but how about people using this template? The various YAML files may be in different repos across our org.

The usage of include is quite simple by using either a reference on the same gitlab instance or remote URL for external self hosted instance

For example:

include:
  # We include the definitions before using them
  - project: baby-gnu/ci-tools
    ref: main
    file: /templates/Rules.yaml
  - project: baby-gnu/ci-tools
    ref: main
    file: /templates/Git.yaml

stages:
  - lint
  
commitlint: {extends: '.git:commitlint'}

• The current .gitlab-ci.yml is a single template managed from the ssf-formula, like .travis.yml and the other files -- splitting these will bring in a lot more complexity.

I think this is not an issue because:

1. the common job templates are define only once
2. the per project .gitlab-ci.yaml are more manageable by avoiding any updates for job templates

• The YAML node anchors cannot be used across the included files, so this will in turn, make things less DRY and scatter the specific configurations throughout the file(s) rather than at the top in one place.

The extends keyword is a good replacement of YAML anchors.

• Not sure if this will make much difference but there is also a matrix feature in GitLab CI that should be investigated.

This could be investigated in a second (or third) time, but I'm not sure it will be an improvement over defining several jobs which are run in parallel.