Sympl issueshttps://gitlab.com/sympl.io/sympl/-/issues2019-06-09T23:33:49Zhttps://gitlab.com/sympl.io/sympl/-/issues/170Symbiosis: Request: Write mysql root credentials to /root/.my.cnf when imaging.2019-06-09T23:33:49ZPaul CammishSymbiosis: Request: Write mysql root credentials to /root/.my.cnf when imaging.Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/56
(Note: This may be something for imager, or Stretch, but applied to Symbiosis *images* only)
It's never clear that the `root`, `admin` and mysql `root@localhost` ...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/56
(Note: This may be something for imager, or Stretch, but applied to Symbiosis *images* only)
It's never clear that the `root`, `admin` and mysql `root@localhost` users all have the same password in a newly imaged machine, which leads to users likely changing the root/admin passwords like they should, and not making note of the `root@localhost` password we set for mysql.
Simply writing the below to `/root/.my.cnf` (with relevant permissions) would make password recovery simpler, and allow the user to log in directly.
```config
[client]
user=root
password="<example>"
```
There's a small outside risk to this, by keeping it in `/root` would negate most of this, and make things simpler for users.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/219Administrative user should be named something unique / permissions tidyup2019-06-09T23:34:46ZPaul CammishAdministrative user should be named something unique / permissions tidyupAs is, Symbiosis has an 'admin' user which is used for most functions, and (in theory) some level of security, although this is weakly enforced, and there is a possibility of a name collision.
A better alternative would probably be to h...As is, Symbiosis has an 'admin' user which is used for most functions, and (in theory) some level of security, although this is weakly enforced, and there is a possibility of a name collision.
A better alternative would probably be to have a user called 'sympl' or similar which would own the configs, files and so on, and then it would be safe to chown/chmod items in the config directory to prevent access to things which should be secure.
This looks to be a fairly simple change, and would go with giving the user a proper home directory (/home/sympl) rather than forcing them to use /srv, which becomes untidy quickly, and gives us the opportunity to pre-populate some basic settings (prompt, etc) for ease of use.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/17symbiosis-password-test doesn't do anything serious2019-06-10T15:01:48ZPaul Cammishsymbiosis-password-test doesn't do anything seriousThis will also need the old ruby-cracklib code swapping to use ruby-password.
As is, it won't check for weak passwords, which is it's core function.This will also need the old ruby-cracklib code swapping to use ruby-password.
As is, it won't check for weak passwords, which is it's core function.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/16symbiosis-encrypt-password doesn't check for weak passwords2019-06-10T15:01:46ZPaul Cammishsymbiosis-encrypt-password doesn't check for weak passwordsNeeds to be updated to use ruby-password rather than cracklibNeeds to be updated to use ruby-password rather than cracklibSympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/13poppass_handler.rb no longer checks passwords for complexity2019-06-10T15:01:30ZPaul Cammishpoppass_handler.rb no longer checks passwords for complexity`email/lib/symbiosis/email/poppass_handler.rb` has been switched from ruby-cracklib to plain ruby-password.
As part of the change (quick fix), it no longer enforces password complexity, allowing weak and possibly compromisable passwords.`email/lib/symbiosis/email/poppass_handler.rb` has been switched from ruby-cracklib to plain ruby-password.
As part of the change (quick fix), it no longer enforces password complexity, allowing weak and possibly compromisable passwords.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/209Build tests should check upgrading from a current install2019-06-11T12:02:28ZPaul CammishBuild tests should check upgrading from a current installThis should prevent an upgrade breaking all the versions.This should prevent an upgrade breaking all the versions.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/225sympl: sympl-common and sympl-core are seperate packages but have the same pu...2019-06-11T16:40:56ZPaul Cammishsympl: sympl-common and sympl-core are seperate packages but have the same purposeI can't see a reason for this any more, so it may make sense to just merge them into one package - there's no need to have them separate as sympl-core was 'bytemark-symbiosis' and just a metapackage with the recommended packages to insta...I can't see a reason for this any more, so it may make sense to just merge them into one package - there's no need to have them separate as sympl-core was 'bytemark-symbiosis' and just a metapackage with the recommended packages to install and a MOTD.
To cut down on extraneous info, they can probably be merged, retaining the sympl-core name.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/223Ruby scripts have output noise when run in verbose.2019-06-07T14:21:08ZPaul CammishRuby scripts have output noise when run in verbose.The --verbose fag sets the ruby $VERBOSE variable, with is outputting various warnings.
Changing the name of this variable should avoid the collision.
symbiosis-dns-generate --verbose
```
Falling back to gcc to determine sizeof size_t....The --verbose fag sets the ruby $VERBOSE variable, with is outputting various warnings.
Changing the name of this variable should avoid the collision.
symbiosis-dns-generate --verbose
```
Falling back to gcc to determine sizeof size_t.
/usr/lib/ruby/vendor_ruby/diffy/diff.rb:43: warning: method redefined; discarding old diff
/usr/lib/ruby/vendor_ruby/erubis/enhancer.rb:517: warning: instance variable @prefixrexp not initialized
```
symbiosis-firewall --verbose
```
Falling back to gcc to determine sizeof size_t.
readnews defined twice. Ignoring definition for port 532
dicom defined twice. Ignoring definition for port 11112
```
symbiosis-firewall-blacklist --verbose
```
Falling back to gcc to determine sizeof size_t.
```
symbiosis-firewall-whitelist --verbose
```
Falling back to gcc to determine sizeof size_t.
```
symbiosis-httpd-generate-stats --verbose
```
Falling back to gcc to determine sizeof size_t.
/usr/lib/ruby/vendor_ruby/diffy/diff.rb:43: warning: method redefined; discarding old diff
```
symbiosis-httpd-rotate-logs --verbose
```
Falling back to gcc to determine sizeof size_t.
```
symbiosis-ssl
```
net/http: warning: Content-Type did not set; using application/x-www-form-urlencoded
net/http: warning: Content-Type did not set; using application/x-www-form-urlencoded
```Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/152Symbiosis: Method redefined' and 'variable not initialized' warnings returned...2019-06-07T14:39:39ZPaul CammishSymbiosis: Method redefined' and 'variable not initialized' warnings returned from symbiosis-httpd-configure when '--verbose' flag usedImported from https://www.github.com/BytemarkHosting/symbiosis/issues/122
Running `symbiosis-httpd-configure` with the `--verbose` flag appended, e.g `symbiosis-httpd-configure -vdf`, returns the following:
<pre>
root@symbiosis2:/etc/e...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/122
Running `symbiosis-httpd-configure` with the `--verbose` flag appended, e.g `symbiosis-httpd-configure -vdf`, returns the following:
<pre>
root@symbiosis2:/etc/exim4# symbiosis-httpd-configure -vdf
/usr/lib/ruby/vendor_ruby/diffy/diff.rb:43: warning: method redefined; discarding old diff
Domain: symbiosis2.default.aladlow.uk0.bigv.io
Current SSL set 6: signed by /C=US/O=Let's Encrypt/CN=Let's Encrypt Authority X3, expires 2018-09-07 22:00:16 UTC
This site has SSL enabled, and is using the host's primary IPs -- continuing with SNI.
SSL is enabled -- using SSL template
Adding to configurations
Configuration: example.site.net.conf
Forcing re-creation of configuration due to --force.
/usr/lib/ruby/vendor_ruby/diffy/diff.rb:70: warning: instance variable @tempfiles not initialized
Syntax OK
</pre>
Notably:
`/usr/lib/ruby/vendor_ruby/diffy/diff.rb:43: warning: method redefined; discarding old diff`
`/usr/lib/ruby/vendor_ruby/diffy/diff.rb:70: warning: instance variable @tempfiles not initialized`
These probably shouldn't be displayed as standard.
Sympl v9.0 (for Debian Stretch)https://gitlab.com/sympl.io/sympl/-/issues/210Packages should be published in a repo2019-06-08T22:06:25ZPaul CammishPackages should be published in a repoThis will include properly signed packages, via the Mythic Beasts repo.This will include properly signed packages, via the Mythic Beasts repo.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/197Publish packages properly ;)2019-06-07T10:49:35ZPaul CammishPublish packages properly ;)The installation instructions smell a little -- getting a proper repo might be a nice touch.
You might find [Bintray](https://bintray.com/signup/oss) one way of doing it. I came across it for TV headend.The installation instructions smell a little -- getting a proper repo might be a nice touch.
You might find [Bintray](https://bintray.com/signup/oss) one way of doing it. I came across it for TV headend.Sympl v9.0 (for Debian Stretch)https://gitlab.com/sympl.io/sympl/-/issues/200There should be a simpler way of selecting the default site2019-04-22T17:53:40ZPaul CammishThere should be a simpler way of selecting the default siteThis would replace the zz-mass-hosting configuration (which can cause odd issues with subdomains as it matches more than it needs) and replace the need for `/srv/$(hostname)` and simply set the SSL certs to the default site for things li...This would replace the zz-mass-hosting configuration (which can cause odd issues with subdomains as it matches more than it needs) and replace the need for `/srv/$(hostname)` and simply set the SSL certs to the default site for things like FTP, and use a simple Apache redirect to handle it properly.
Something like a symlink of `/srv/default_site` -> `/srv/example.com` would do the job, although this change would need to be detected and the relevant services reloaded/restarted as needed.Future Planshttps://gitlab.com/sympl.io/sympl/-/issues/199symbiosis-ssl reports `Failed: signature type 'none' in JWS header is not sup...2019-06-07T10:51:28ZPaul Cammishsymbiosis-ssl reports `Failed: signature type 'none' in JWS header is not supported` when trying to get cert.As mentioned in #198
```
* Examining certificates for example.domain
No valid certificate sets found.
Fetching a new certificate from LetsEncrypt.
!! Failed: signature type 'none' in JWS header is not supported,...As mentioned in #198
```
* Examining certificates for example.domain
No valid certificate sets found.
Fetching a new certificate from LetsEncrypt.
!! Failed: signature type 'none' in JWS header is not supported, expected one of RS256, ES256, ES384 or ES512
* Examining certificates for localhost.localdomain
Current SSL set 0: self-signed for /CN=localhost.localdomain, expires 2020-04-15 16:32:06 UTC
```
Possible dependency or other issueSympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/198Warning during installation about gcc not being found2019-06-07T10:51:32ZPaul CammishWarning during installation about gcc not being foundDuring install, I see the following:
```
Setting up symbiosis-common (2018:0616) ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(passwd) reque...During install, I see the following:
```
Setting up symbiosis-common (2018:0616) ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(group) request, exiting
Shadow passwords are now on.
Adding 'admin' account
Adding user `admin' ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Adding new group `admin' (1001) ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Adding new user `admin' (1001) with group `admin' ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Not creating home directory `/srv'.
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Adding user `admin' to group `adm' ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Adding user admin to group adm
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Done.
Adding user `admin' to group `www-data' ...
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Adding user admin to group www-data
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
sent invalidate(group) request, exiting
sent invalidate(passwd) request, exiting
sent invalidate(group) request, exiting
Done.
sh: 1: gcc: not found
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- faraday (LoadError)
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/vendor_ruby/acme-client.rb:3:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/vendor_ruby/symbiosis/ssl/letsencrypt.rb:6:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/bin/symbiosis-ssl:161:in `<main>'
W: SSL certificate generation failed. Retrying with a self-signed certificate...
sh: 1: gcc: not found
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- faraday (LoadError)
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/vendor_ruby/acme-client.rb:3:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/vendor_ruby/symbiosis/ssl/letsencrypt.rb:6:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/bin/symbiosis-ssl:161:in `<main>'
sh: 1: gcc: not found
/usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require': cannot load such file -- faraday (LoadError)
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/vendor_ruby/acme-client.rb:3:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/vendor_ruby/symbiosis/ssl/letsencrypt.rb:6:in `<top (required)>'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/lib/ruby/2.3.0/rubygems/core_ext/kernel_require.rb:55:in `require'
from /usr/bin/symbiosis-ssl:161:in `<main>'
Created symlink /etc/systemd/system/multi-user.target.wants/symbiosis-skel.path → /lib/systemd/system/symbiosis-skel.path.
symbiosis-skel.service is a disabled or a static unit, not starting it.
```
Installation seems to continue and succeed.Sympl v9.0 (for Debian Stretch)Paul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/196Symbiosis: Using `/srv/<domains/mailboxes/<user>/forward` with an external de...2019-06-20T15:02:39ZPaul CammishSymbiosis: Using `/srv/<domains/mailboxes/<user>/forward` with an external destinations fails SPF and DKIM checksImported from https://www.github.com/BytemarkHosting/symbiosis/issues/54
As `/srv/<domains/mailboxes/<user>/forward` simply relays the mail, it often causes SPF and DKIM signed mail to fail. This is particularly bad as the more importan...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/54
As `/srv/<domains/mailboxes/<user>/forward` simply relays the mail, it often causes SPF and DKIM signed mail to fail. This is particularly bad as the more important the mail (banks etc) the more tight their SPK/DKIM will be, and the destination mail server may simply drop invalid mail, or in many cases (see [SNDS](https://wiki2.bytemark.co.uk/Microsoft_Smart_Network_Data_Service)) simply flag all mail coming in that route as spam.
This can be worked around by using a basic Sieve filter (created in a mail client or Roundcube, etc) instead of a basic forward, which generates a header in the final mailbox like this:
```
Delivered-To: under100words@example.net
Received: by 10.157.19.41 with SMTP id f38csp1124363ote;
Fri, 24 Mar 2017 02:25:49 -0700 (PDT)
X-Received: by 10.28.157.150 with SMTP id g144mr1955711wme.89.1490347549347;
Fri, 24 Mar 2017 02:25:49 -0700 (PDT)
Return-Path: <admin@carth.ebonhawk.example.uk0.bigv.io>
Received: from carth.ebonhawk.example.uk0.bigv.io (under100words.com. [2001:41c8:51:7a3::163])
by mx.google.com with ESMTPS id x4si1960991wmx.113.2017.03.24.02.25.48
for <under100words@example.net>
(version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128);
Fri, 24 Mar 2017 02:25:49 -0700 (PDT)
Received-SPF: neutral (google.com: 2001:41c8:51:7a3::163 is neither permitted nor denied by best guess record for domain of admin@carth.ebonhawk.example.uk0.bigv.io) client-ip=2001:41c8:51:7a3::163;
Authentication-Results: mx.google.com;
dkim=pass header.i=@bytemark.co.uk;
spf=neutral (google.com: 2001:41c8:51:7a3::163 is neither permitted nor denied by best guess record for domain of admin@carth.ebonhawk.example.uk0.bigv.io) smtp.mailfrom=admin@carth.ebonhawk.example.uk0.bigv.io;
dmarc=pass (p=NONE sp=NONE dis=NONE) header.from=bytemark.co.uk
Received: from admin by carth.ebonhawk.example.uk0.bigv.io with local (Exim 4.84_2) (envelope-from <admin@carth.ebonhawk.example.uk0.bigv.io>) id 1crLTg-0006yS-L8 for under100words@example.net; Fri, 24 Mar 2017 09:25:48 +0000
X-Sieve: Pigeonhole Sieve 0.4.2
X-Sieve-Redirected-From: nobody@under100words.com
Received: from yrk.mx.bytemark.co.uk ([2001:41c9:0:1019::81:84]) by carth.ebonhawk.example.uk0.bigv.io with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <paul.cammish@bytemark.co.uk>) id 1crLTf-0006yJ-0J for nobody@under100words.com; Fri, 24 Mar 2017 09:25:48 +0000
DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=bytemark.co.uk; s=20131115; h=Content-Type:MIME-Version:Date:Message-ID:Subject:From:To; bh=0UTMmBvQCD1aDtGQ0LM/wKwl+36AyMeb9pEvAf/Lve8=; b=jCmTawHtj3wZvPA+DnmtoyYpJYmznfAbAR2ZZkF0WmkbB9MD0BdivSofH1ww4DoYM3tH3BtjFt02Bn0NpKpNeNX74c8l5lVQ6QB2uVc2dDBVhma/MLbceDvrjAsOXGnomgWIFPZNLzOAcXF/mS+5ctw7nRGg44nxxlXMKAymbzTvzOGiiyA0rX2DgKhrn3SDhBNNC2dW8AsiZ6lqs9jZz99bqJlMwzhGz709MfPfzdvnIKsnt9sBc/hL+KAu0lUH1l0ySSNeLqyU6CTU91B3ULnVkQoX9EMh6Rim3430thOQ0VAfon9kp3l7lVbaXfDOkcK2PLxRBYSqnmD5uVR3dQ==;
Received: from ratatoskr.bytemark.co.uk ([2001:41c9:0:1017::48] helo=ratatoskr.dh.bytemark.co.uk) by yrk.mx.bytemark.co.uk with esmtps (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <paul.cammish@bytemark.co.uk>) id 1crLTe-0006jU-Kd for nobody@under100words.com; Fri, 24 Mar 2017 09:25:46 +0000
Received: from [2001:41c8:3:1::186] by ratatoskr.dh.bytemark.co.uk with esmtpsa (TLS1.2:ECDHE_RSA_AES_128_GCM_SHA256:128) (Exim 4.84_2) (envelope-from <paul.cammish@bytemark.co.uk>) id 1crLTe-0005hF-I3 for nobody@under100words.com; Fri, 24 Mar 2017 09:25:46 +0000
To: nobody@under100words.com
From: Paul <paul@example.co.uk>
Subject: Test forward
Organization: Bytemark Hosting
Message-ID: <bdf891dd-fb0c-f7ca-3f89-7d7a00ff49ee@bytemark.co.uk>
Date: Fri, 24 Mar 2017 09:25:44 +0000
User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:45.0) Gecko/20100101 Thunderbird/45.7.0
MIME-Version: 1.0
Content-Type: multipart/signed; micalg=pgp-sha256; protocol="application/pgp-signature"; boundary="1kPfKMM3kgqtkMjBuqBIR3uSp4MpjjNV9"
Sender: Symbiosis Administrator <admin@carth.ebonhawk.example.uk0.bigv.io>
```
This works with a simple sieve rule like:
```
# rule:[Redirect all mail.]
if true
{
redirect "under100words@example.net";
stop;
}
```
It would be great if the forward file generated the relevant sieve rule, and massively improve deliverability.Future Planshttps://gitlab.com/sympl.io/sympl/-/issues/189Symbiosis: Update dns.rb to make default dmarc policy safer2019-06-07T14:36:29ZPaul CammishSymbiosis: Update dns.rb to make default dmarc policy saferImported from https://www.github.com/BytemarkHosting/symbiosis/issues/130
The current dmarc policy is dangerous, in that it says to quarantine 100% of unaligned email from the main domain (but oddly not subdomains). This policy is much ...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/130
The current dmarc policy is dangerous, in that it says to quarantine 100% of unaligned email from the main domain (but oddly not subdomains). This policy is much safer for an initial dmarc policy, as recommended for example, by Google at https://support.google.com/a/answer/2466563?hl=en&ref_topic=2759254Backloghttps://gitlab.com/sympl.io/sympl/-/issues/187Symbiosis: testing sites don't get DocumentRoot set2019-04-14T21:44:45ZPaul CammishSymbiosis: testing sites don't get DocumentRoot setImported from https://www.github.com/BytemarkHosting/symbiosis/issues/43
testing sites - e.g. example.com.testing.server.group.user.uk0.bigv.io don't have DocumentRoot set. Not being set by mod_rewriteImported from https://www.github.com/BytemarkHosting/symbiosis/issues/43
testing sites - e.g. example.com.testing.server.group.user.uk0.bigv.io don't have DocumentRoot set. Not being set by mod_rewritehttps://gitlab.com/sympl.io/sympl/-/issues/186Symbiosis: Testing in gitlab2019-04-22T17:57:10ZPaul CammishSymbiosis: Testing in gitlabImported from https://www.github.com/BytemarkHosting/symbiosis/issues/53
At the moment the install / dist-upgrade / upgrade tests get weirdly-far in gitlab-ci then fails. Here's a quick summary of how the tests used to work on maker2 (a...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/53
At the moment the install / dist-upgrade / upgrade tests get weirdly-far in gitlab-ci then fails. Here's a quick summary of how the tests used to work on maker2 (as I understand it), and then later I'll go into detail on why the tests fail in gitlab-ci
The Current Situation
================
1. autotest sets up a VM using schroot magic i don't fully understand
2. the VM boots up with systemd and all that jazz, uses DHCP & SLAAC to configure its networking, and automatically runs all the scripts in the autotest folder, keeping all the output as a log file. Once done, it shuts down
3. autotest cracks open the VM's filesystem and reads the logfile. Somehow it detects failures and fails if there was a failure then it exits with a nonzero error code so that maker2 knows
Some Feelings About The Current Situation
===============================
Patrick said something about autotest using the console to talk to the tests, and something else much scarier about the VM sshing into the host to run something.
This doesn't work on gitlab-ci, and is also kinda hacky, for a few reasons.
* the scripts in the autotest folder aren't particularly focussed. In addition to actually running tests, they do these and probably others:
* add an admin user
* install all the packages needed by symbiosis from a big list of packages
* install symbiosis
* opening up the filesystem of the VM so you can prod it is pretty gross
On the plus side it works, and it would only take a bit of effort to port the whole schroot setup over to gitlab-ci (but would have to run using a shell runner)
Why the tests fail in gitlab-ci
=====================
When gitlab-ci runs a container it starts bash in the context of the container. Effectively, bash is PID 1 for the container. There's no init-system to talk to to get stuff going. I believe the apt-get install step for some packages starts them using /etc/init.d (probably something about the package detecting a lack of systemd and putting a proper sysvinit script in) which would explain why a lot of the tests actually succeed. BUT SOME OF THEM FAIL, and we should really be doing a much more realistic test than running our symbiosis full-system tests in a docker container that isn't a full symbiosis system.
With that in mind:
A More Realistic Test Proposal
======================
We're still going to want to run symbiosis in a VM, I think. To do a realistic full-install / dist-upgrade test we need to have a realistic system, which the docker container environment isn't. We need a systemd to talk to so we can schedule restarts, that sort of thing.
We will need some test-specific configurations (particularly repo URLs) too. And we'll need to be able to orchestrate the testing and fail the build when the tests fail.
We could create an image prior to the testing which would have a user account with passwordless sudo and a .ssh/authorized_keys . The private key would be kept in the [secret variables](https://gitlab.bytemark.co.uk/open-source/symbiosis/settings/ci_cd) section of the project on gitlab, and so would be presented to the gitlab-ci script as an env var.
In the gitlab-ci script we'd start the VM with qemu, as we do for bytemark/bytemark-packer-templates, then use ansible to copy over the tests, install the symbiosis packages, and run the tests. We could write our ansible playbook so that it captures the logs and copies them back to the runner and have the gitlab-ci script spit the logs out, then exit with ansible's exit code.
This would make our test output more readable and shorter, not be quite as weird the current autotest setup on maker2, probably not require also running a DHCP server.
The work we'd need to do:
* add an ansible layer to docker-images/layers
* rewrite the `autotest/` scripts as ansible playbooks
* make a base VM image with the necessary networking & ssh setup
Thoughts @pcherry , @jcarter ?Testing SuitePaul CammishPaul Cammishhttps://gitlab.com/sympl.io/sympl/-/issues/185Symbiosis: symbiosis-ssl can generate SSL config for sites that have no certi...2019-04-17T20:11:54ZPaul CammishSymbiosis: symbiosis-ssl can generate SSL config for sites that have no certificateImported from https://www.github.com/BytemarkHosting/symbiosis/issues/44
symbiosis-ssl can generate SSL config for sites that have no certificate returned by Lets Encrypt. This can lead to invalid configuration, and Apache being unable ...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/44
symbiosis-ssl can generate SSL config for sites that have no certificate returned by Lets Encrypt. This can lead to invalid configuration, and Apache being unable to re-start.
This has been observed both in terms of missing certs that were never returned successfully from Lets Encrypt, or where symbiosis-ssl didn't have permission to write the certificate, but still wrote the SSL config.https://gitlab.com/sympl.io/sympl/-/issues/184Symbiosis: symbiosis-httpd-configure breaks when no certificates are available.2019-06-07T14:36:24ZPaul CammishSymbiosis: symbiosis-httpd-configure breaks when no certificates are available.Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/112
symbiosis-httpd-configure breaks when no certificates are available.
Steps to reproduce:
1. Spin up a new Symbiosis 8 server at panel.bytemark.co.uk
2. remove t...Imported from https://www.github.com/BytemarkHosting/symbiosis/issues/112
symbiosis-httpd-configure breaks when no certificates are available.
Steps to reproduce:
1. Spin up a new Symbiosis 8 server at panel.bytemark.co.uk
2. remove the contents of /etc/ss
3. run "symbiosis-httpd-configure --force --verbose"
4. run "apachectl restart"
Expected behaviour:
* Apache restarts.
Observed behaviour:
* Apache fails to restart. An error message like "Failed to configure at least one certificate and key for symbiosis.foo.bar.uk0.bigv.io:443"
Workaround:
* remove /etc/apache2/sites-enabled/zz-mass-hosting.ssl.conf - Apache will now start, until symbiosis-httpd-configure is run again.
Suggested fix:
* Symbiosis-httpd-configure should test for the presence of some certificate before proceeding to enable zz-mass-hosting.ssl.confBacklog