ENT-11526: Changed mission-portal apache restart to graceful to minimize service interruptions

[craigcomstock]

In our CI system deployment tests we go through test steps quickly.
Fairly often the apache config is updated by policy and a restart is initiated.
By changing to apachectl graceful we should avoid some of the server availability during tests problems we have been seeing.

Ticket: ENT-11526
Changelog: title

with cfengine/core#5480

[craigcomstock]

@cf-bottom jenkins please, thanks.

[cf-bottom]

Sure, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/10580/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-10580/

[craigcomstock]

bootstrap-tests is a known issue fixed for ENT-8264 in https://github.com/cfengine/mission-portal/pull/2321

98-delete-project failure

6	:	1712282346.958	     0.460	0	0	22	0	curl -k -i --show-error --fail -u admin:admin -H "content-type: application/json" -X DELETE https://54.216.44.96/api/build/projects/1 && echo

is a bit like what I was trying to fix but in this case it is a 500 error which would not be solved by a graceful restart of apache.

54.216.44.96	  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
54.216.44.96	                                 Dload  Upload   Total   Spent    Left  Speed
54.216.44.96	
54.216.44.96	  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
54.216.44.96	  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
54.216.44.96	curl: (22) The requested URL returned error: 500 Internal Server Error

Will check if there is another ticket for 500 errors during this test.

[craigcomstock]

have organized tickets and made one common ticket for 500 errors related to failed to open stream, permission denied, dispatch.php, vendor/autoload.php etc... ENT-11182

[craigcomstock]

@cf-bottom jenkins please, now with a core change to add ExecReload using apachectl graceful to core systemd units.

[cf-bottom]

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/10602/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-10602/

[nickanderson]

I can see how this might make sense. I would be concerned that graceful is aysnc and then after the promise runs we wouldn't be in the expected state until some undetermined period of time later.

[vpodzime]

Makes sense. 👍

[craigcomstock]

Looks like many platforms didn't have httpd running during bootstrap-tests restart test and sequential-tests during upgrades showed an error that may indicate why:

3.249.208.255	Failed to start cf-apache.service: Unit cf-apache.service is not loaded properly: Exec format error.

Will check it out.

[craigcomstock]

typo in core, fixed, @cf-bottom jenkins retry please.

[cf-bottom]

Alright, I triggered a build:

Jenkins: https://ci.cfengine.com/job/pr-pipeline/10603/

Packages: http://buildcache.cfengine.com/packages/testing-pr/jenkins-pr-pipeline-10603/

[craigcomstock]

rhel-9 hub in 99-cfbs-init test said

54.74.76.234	curl: (22) The requested URL returned error: 500 Internal Server Error

Looking in support tarball and var/cfengine/httpd/logs/error_log I think this is a known issue

[Thu Apr 11 23:41:58.598136 2024] [php:warn] [pid 58068] [client 91.232.32.37:46788] PHP Warning:  Unknown: Failed to open stream: Permission denied in Unknown on line 0
[Thu Apr 11 23:41:58.598753 2024] [php:error] [pid 58068] [client 91.232.32.37:46788] PHP Fatal error:  Failed opening required '/var/cfengine/httpd/htdocs/api/dispatch.php' (include_path='.:') in Unknown on line 0

Known issue: https://northerntech.atlassian.net/browse/ENT-11182

---

Selenium failure: InventoryTest::testExcludeIncludeFilter, looks like a timing fluke in selenium test to me. Logged a ticket: https://northerntech.atlassian.net/browse/ENT-11546

[craigcomstock]

Will wait to cherry pick for a week or so to see how/if this improves our CI. Will set a reminder for myself for 2024-04-19.