Sympl issueshttps://gitlab.com/sympl.io/sympl/-/issues2019-04-07T21:09:37Zhttps://gitlab.com/sympl.io/sympl/-/issues/18Installation stalls at `Setting up symbiosis-mysql`2019-04-07T21:09:37ZPaul CammishInstallation stalls at `Setting up symbiosis-mysql`The install stage of the gitlab-ci process stalls and apparently site there indefinitely at the `Setting up symbiosis-mysql (2018:0719) ...` stage, likely requesting input.
Will probably need to be run separately to confirm and come up...The install stage of the gitlab-ci process stalls and apparently site there indefinitely at the `Setting up symbiosis-mysql (2018:0719) ...` stage, likely requesting input.
Will probably need to be run separately to confirm and come up with a fix.Packages Build and InstallPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/11symbiosis-xmpp has unmet dependencies2019-04-01T23:53:37ZPaul Cammishsymbiosis-xmpp has unmet dependencies```
The following packages have unmet dependencies:
symbiosis-xmpp : Depends: prosody-mod-auth-dovecot but it is not installable
``````
The following packages have unmet dependencies:
symbiosis-xmpp : Depends: prosody-mod-auth-dovecot but it is not installable
```Packages Build and InstallPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/10symbiosis-monit has unmet dependencies2019-04-03T21:46:12ZPaul Cammishsymbiosis-monit has unmet dependencies```
The following packages have unmet dependencies:
symbiosis-monit : Depends: ruby-dbus-systemd but it is not installable
``````
The following packages have unmet dependencies:
symbiosis-monit : Depends: ruby-dbus-systemd but it is not installable
```Packages Build and Installhttps://gitlab.com/sympl.io/sympl/-/issues/9symbiosis-key has unmet dependencies2019-04-01T23:53:41ZPaul Cammishsymbiosis-key has unmet dependencies```
The following packages have unmet dependencies:
symbiosis-key : Depends: bytemark-keys but it is not installable
``````
The following packages have unmet dependencies:
symbiosis-key : Depends: bytemark-keys but it is not installable
```Packages Build and InstallPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/8symbiosis-email-activesync has unmet dependecies2019-04-01T23:53:45ZPaul Cammishsymbiosis-email-activesync has unmet dependecies```
The following packages have unmet dependencies:
symbiosis-email-activesync : Depends: d-push but it is not installable
``````
The following packages have unmet dependencies:
symbiosis-email-activesync : Depends: d-push but it is not installable
```Packages Build and InstallPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/7symbiosis-email has unmet dependencies2019-04-01T23:53:49ZPaul Cammishsymbiosis-email has unmet dependencies```
The following packages have unmet dependencies:
symbiosis-email : Depends: ruby-cracklib but it is not installable
``````
The following packages have unmet dependencies:
symbiosis-email : Depends: ruby-cracklib but it is not installable
```Packages Build and InstallPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/6symbiosis-common has unmet dependencies2019-04-03T23:10:01ZPaul Cammishsymbiosis-common has unmet dependencies```
The following packages have unmet dependencies:
symbiosis-common : Depends: ruby-acme-client (>= 0.3.5) but it is not installable
Depends: ruby-linux-netlink but it is not installable
Depends:...```
The following packages have unmet dependencies:
symbiosis-common : Depends: ruby-acme-client (>= 0.3.5) but it is not installable
Depends: ruby-linux-netlink but it is not installable
Depends: ruby-cracklib but it is not installable
```Packages Build and InstallPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/3Install steps fail in gitlab-ci2019-04-01T23:53:54ZPaul CammishInstall steps fail in gitlab-ciShould be able to test install the packages, and successfully run the relevant tests, which will likely need to be re-written.Should be able to test install the packages, and successfully run the relevant tests, which will likely need to be re-written.Packages Build and InstallPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/1Job Failed #185536396 - Missing curl & gnupg packages2019-04-01T23:53:58ZPaul CammishJob Failed #185536396 - Missing curl & gnupg packagesJob [#185536396](https://gitlab.com/sympl.host/sympl/-/jobs/185536396) failed for 67b9e5f9b28a4d80ac2a8e212613fe2fd46ee5df:
Some runners seem to be missing these packages, so they should be added to the list to be installed manually, ju...Job [#185536396](https://gitlab.com/sympl.host/sympl/-/jobs/185536396) failed for 67b9e5f9b28a4d80ac2a8e212613fe2fd46ee5df:
Some runners seem to be missing these packages, so they should be added to the list to be installed manually, just in case.Packages Build and InstallPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/206symbiosis-test skips phpmyadmin tests2019-05-28T11:58:10ZPaul Cammishsymbiosis-test skips phpmyadmin testsIt looks like due to the changes to MariaDB, the tests which expect to log in to phpmyadmin as root/debian-sys-maint are failing.
```
Skipping phpmyadmin debian-sys-maint auth test - password not found.
Skipping phpmyadmin root auth tes...It looks like due to the changes to MariaDB, the tests which expect to log in to phpmyadmin as root/debian-sys-maint are failing.
```
Skipping phpmyadmin debian-sys-maint auth test - password not found.
Skipping phpmyadmin root auth test - password not found.
```
This should be fairly simple to fix to use the generated 'admin' username/password, and ensure the passwordless logins fail.Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/205"Quota exceeded (mailbox for user is full)"2019-05-28T11:58:11ZPaul Cammish"Quota exceeded (mailbox for user is full)"Symbiosis-test outputs `Quota exceeded (mailbox for user is full)` twice while running. This may be a bug, or it may be operating normally. Either way it should be fixed or supressed.Symbiosis-test outputs `Quota exceeded (mailbox for user is full)` twice while running. This may be a bug, or it may be operating normally. Either way it should be fixed or supressed.Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/204"Not running MySQL backup tests, since not all the requirements are in place."2019-05-28T11:58:08ZPaul Cammish"Not running MySQL backup tests, since not all the requirements are in place."It looks like the relevant ruby libraries are missing for symbiosis-test from the repo/install (and would have been on the build box), but an attempt to track the relevant version down didn't come up with a perfect match.
This can proba...It looks like the relevant ruby libraries are missing for symbiosis-test from the repo/install (and would have been on the build box), but an attempt to track the relevant version down didn't come up with a perfect match.
This can probably just be rewritten in bash, as it's some simple SQL queries.Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/186Symbiosis: Testing in gitlab2019-04-22T17:57:10ZPaul CammishSymbiosis: Testing in gitlabImported from https://www.github.com/BytemarkHosting/symbiosis/issues/53
At the moment the install / dist-upgrade / upgrade tests get weirdly-far in gitlab-ci then fails. Here's a quick summary of how the tests used to work on maker2 (a...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/53
At the moment the install / dist-upgrade / upgrade tests get weirdly-far in gitlab-ci then fails. Here's a quick summary of how the tests used to work on maker2 (as I understand it), and then later I'll go into detail on why the tests fail in gitlab-ci
The Current Situation
================
1. autotest sets up a VM using schroot magic i don't fully understand
2. the VM boots up with systemd and all that jazz, uses DHCP & SLAAC to configure its networking, and automatically runs all the scripts in the autotest folder, keeping all the output as a log file. Once done, it shuts down
3. autotest cracks open the VM's filesystem and reads the logfile. Somehow it detects failures and fails if there was a failure then it exits with a nonzero error code so that maker2 knows
Some Feelings About The Current Situation
===============================
Patrick said something about autotest using the console to talk to the tests, and something else much scarier about the VM sshing into the host to run something.
This doesn't work on gitlab-ci, and is also kinda hacky, for a few reasons.
* the scripts in the autotest folder aren't particularly focussed. In addition to actually running tests, they do these and probably others:
* add an admin user
* install all the packages needed by symbiosis from a big list of packages
* install symbiosis
* opening up the filesystem of the VM so you can prod it is pretty gross
On the plus side it works, and it would only take a bit of effort to port the whole schroot setup over to gitlab-ci (but would have to run using a shell runner)
Why the tests fail in gitlab-ci
=====================
When gitlab-ci runs a container it starts bash in the context of the container. Effectively, bash is PID 1 for the container. There's no init-system to talk to to get stuff going. I believe the apt-get install step for some packages starts them using /etc/init.d (probably something about the package detecting a lack of systemd and putting a proper sysvinit script in) which would explain why a lot of the tests actually succeed. BUT SOME OF THEM FAIL, and we should really be doing a much more realistic test than running our symbiosis full-system tests in a docker container that isn't a full symbiosis system.
With that in mind:
A More Realistic Test Proposal
======================
We're still going to want to run symbiosis in a VM, I think. To do a realistic full-install / dist-upgrade test we need to have a realistic system, which the docker container environment isn't. We need a systemd to talk to so we can schedule restarts, that sort of thing.
We will need some test-specific configurations (particularly repo URLs) too. And we'll need to be able to orchestrate the testing and fail the build when the tests fail.
We could create an image prior to the testing which would have a user account with passwordless sudo and a .ssh/authorized_keys . The private key would be kept in the [secret variables](https://gitlab.bytemark.co.uk/open-source/symbiosis/settings/ci_cd) section of the project on gitlab, and so would be presented to the gitlab-ci script as an env var.
In the gitlab-ci script we'd start the VM with qemu, as we do for bytemark/bytemark-packer-templates, then use ansible to copy over the tests, install the symbiosis packages, and run the tests. We could write our ansible playbook so that it captures the logs and copies them back to the runner and have the gitlab-ci script spit the logs out, then exit with ansible's exit code.
This would make our test output more readable and shorter, not be quite as weird the current autotest setup on maker2, probably not require also running a DHCP server.
The work we'd need to do:
* add an ansible layer to docker-images/layers
* rewrite the `autotest/` scripts as ansible playbooks
* make a base VM image with the necessary networking & ssh setup
Thoughts @pcherry , @jcarter ?Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/19New runner configuration is needed2019-04-11T21:59:48ZPaul CammishNew runner configuration is neededAt present it's all running inside docker - this is okay for simple jobs but not for testing, as its not the same as a real dedi or VM.
gitlab-runner supports VirtualBox VMs used for this purpose, so I will need to set this up.At present it's all running inside docker - this is okay for simple jobs but not for testing, as its not the same as a real dedi or VM.
gitlab-runner supports VirtualBox VMs used for this purpose, so I will need to set this up.Testing SuitePaul CammishPaul Cammish2019-04-12https://gitlab.com/sympl.io/sympl/-/issues/5Internal test suite is broken2019-04-22T17:58:31ZPaul CammishInternal test suite is brokenWill likely need redoing from scratch for modern installs, maybe merged with #4 for a single test suite.Will likely need redoing from scratch for modern installs, maybe merged with #4 for a single test suite.Testing Suitehttps://gitlab.com/sympl.io/sympl/-/issues/4Full test suite is needed2019-05-28T11:58:07ZPaul CammishFull test suite is neededThe basics should cover (off the top of my head)...
* backup
* [x] Backups should write sizable data into /var/backup
* [x] MySQL backups should happen okay
* common
* [x] Let's Encrypt certs should be able to be issued okay
* [x] Exi...The basics should cover (off the top of my head)...
* backup
* [x] Backups should write sizable data into /var/backup
* [x] MySQL backups should happen okay
* common
* [x] Let's Encrypt certs should be able to be issued okay
* [x] Existing legacy certs (config/ssl.key etc) should work and override
* cron
* [x] Individual cron jobs should run on each site.
* [x] Normal functional crons should happen as expected.
* dns
* [x] DNS Should be generated as per template, based on changes
* [x] Modified files should not be changed
* doc
* [x] Man pages should load okay for each command
* email
* [x] Email should be accepted and delivered for a specific mailbox
* [x] Email should be forwarded as expected
* [x] Default forwards should work as expected
* [x] Sieve filters should run normally.
* firewall
* [x] Changes to the files in /etc/symbiosis/firewall should be echoed in iptables
* ftpd
* [x] FTP connections should be accepted and end up in the relevant directory
* httpd
* [x] Sites should serve content
* [x] Certs added should enable HTTPS
* [x] Logs should be written to public/logs
* monit
* [x] Stopped services should restart when needed
* [x] Unneeded services should stop when not needed
* mysql
* [x] MySQL connections should operate normally and allow DB creation/changes/deletion.
* phpmyadmin
* [x] Users should be able to log in via basic auth over HTTPS only
* webmail
* [x] Existing users should be able to log in via webmail
* xmpp
* [x] Existing users should be able to log in via XMPPTesting SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/222A new 'theres no site here yet' page is needed2019-06-07T10:37:50ZPaul CammishA new 'theres no site here yet' page is neededThe old one had 2012-eta Bytemark branding, but I should be able to do something better - just need a proper logo for Sympl created.The old one had 2012-eta Bytemark branding, but I should be able to do something better - just need a proper logo for Sympl created.Rebranding Symbiosis to SymplPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/216The auth/helper processes don't seem to be running after a reboot2019-06-02T21:18:53ZPaul CammishThe auth/helper processes don't seem to be running after a rebootspecifically:
```
/usr/sbin/pure-authd --run /usr/sbin/symbiosis-ftpd-check-password --socket /var/run/pure-ftpd/pure-authd.sock
/usr/bin/ruby /usr/sbin/symbiosis-email-poppassd
/usr/bin/ruby /usr/sbin/symbiosis-email-dict-proxy
```specifically:
```
/usr/sbin/pure-authd --run /usr/sbin/symbiosis-ftpd-check-password --socket /var/run/pure-ftpd/pure-authd.sock
/usr/bin/ruby /usr/sbin/symbiosis-email-poppassd
/usr/bin/ruby /usr/sbin/symbiosis-email-dict-proxy
```Rebranding Symbiosis to SymplPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/215Command line scripts need to be renamed, references to symbiosis in filesyste...2019-06-06T11:01:27ZPaul CammishCommand line scripts need to be renamed, references to symbiosis in filesystem removedWIP in https://gitlab.mythic-beasts.com/sympl/sympl_stretch/merge_requests/19
Any existing command line scripts should have symlinks/helpers created to them from the old names for compatibility.
Ruby Libraries can stay where they are f...WIP in https://gitlab.mythic-beasts.com/sympl/sympl_stretch/merge_requests/19
Any existing command line scripts should have symlinks/helpers created to them from the old names for compatibility.
Ruby Libraries can stay where they are for now (this can be tackled later), but things like /etc/symbiosis and dpkg copies need to be moved/renamed (and symlinks created).
* [x] package:core
* [x] package:backup
* [x] package:common
* [x] package:cron
* [x] package:dns
* [x] package:email
* [x] package:firewall
* [x] package:ftpd
* [x] package:httpd
* [x] package:monit
* [x] package:mysql
* [x] package:phpmyadmin
* [x] package:updater
* [x] package:webmail
----
* [x] update all version numbers
* [x] double-check all copyright filesRebranding Symbiosis to SymplPaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/214Packages needed to be renamed2019-05-31T17:05:57ZPaul CammishPackages needed to be renamedbytemark-symbiosis, symbiosis-* packages needed to be renamed to match the new sympl naming.
bytemark-symbiosis -> sympl-core
symbiosis-* -> sympl-*bytemark-symbiosis, symbiosis-* packages needed to be renamed to match the new sympl naming.
bytemark-symbiosis -> sympl-core
symbiosis-* -> sympl-*Rebranding Symbiosis to SymplPaul CammishPaul Cammish