I have a system with 3 certs (1 setup earlier and 2 newly added),
and when updating them one succeeded, one was skipped, and one failed
(mylacme-jawa is a script on my local system ssh'ing to server jawa
with a tunnel back to local host):
$ mylacme-jawa newOrder
root@jawa.homebase.dk's password:
root@jawa.homebase.dk's password:
Certificate URI: https://acme-v02.api.letsencrypt.org/acme/cert/03a4285ee15d4c432d7351bf2bc35902f78e
Installing X.509 certificate /etc/ssl/shared/homebase.dk.key
Installing X.509 certificate chain /etc/ssl/shared/homebase.dk.chain.pem
SHA256 Fingerprint=B8:83:E0:30:B4:31:A7:F8:19:02:A9:03:92:A6:5C:39:FC:68:51:FF:DA:4A:44:1A:1B:FF:ED:B7:97:A3:45:AF
Serial Number:
03:a4:28:5e:e1:5d:4c:43:2d:73:51:bf:2b:c3:59:02:f7:8e
Signature Algorithm: sha256WithRSAEncryption
Issuer: C = US, O = Let's Encrypt, CN = R3
Validity
Not Before: Dec 7 13:50:55 2020 GMT
Not After : Mar 7 13:50:55 2021 GMT
Subject: CN = homebase.dk
X509v3 extensions:
X509v3 Key Usage: critical
Digital Signature, Key Encipherment
X509v3 Extended Key Usage:
TLS Web Server Authentication, TLS Web Client Authentication
X509v3 Basic Constraints: critical
CA:FALSE
X509v3 Subject Key Identifier:
22:50:90:6B:71:EB:0A:28:FD:A9:49:4E:95:CA:43:A0:21:19:16:22
X509v3 Authority Key Identifier:
keyid:14:2E:B3:17:B7:58:56:CB:AE:50:09:40:E6:1F:AF:9D:8B:14:C2:C6
Authority Information Access:
OCSP - URI:http://r3.o.lencr.org
CA Issuers - URI:http://r3.i.lencr.org/
X509v3 Subject Alternative Name:
DNS:homebase.dk, DNS:wiki.homebase.dk, DNS:www.homebase.dk, DNS:www.wiki.homebase.dk
X509v3 Certificate Policies:
Policy: 2.23.140.1.2.1
Policy: 1.3.6.1.4.1.44947.1.1.1
CPS: http://cps.letsencrypt.org
CT Precertificate SCTs:
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : 94:20:BC:1E:8E:D5:8D:6C:88:73:1F:82:8B:22:2C:0D:
D1:DA:4D:5E:6C:4F:94:3D:61:DB:4E:2F:58:4D:A2:C2
Timestamp : Dec 7 14:50:55.713 2020 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:45:02:21:00:E9:4F:23:6F:4B:A9:D8:B2:E4:EB:C0:
A8:2B:0E:2C:26:0B:90:5F:4A:52:20:D3:6F:22:B9:95:
1E:EF:69:42:CC:02:20:66:2C:9B:F1:6D:9A:A9:4C:54:
5D:DB:16:7A:E9:5C:48:F4:FC:F2:F8:68:84:CB:F5:39:
28:14:DE:BF:D1:3F:C8
Signed Certificate Timestamp:
Version : v1 (0x0)
Log ID : F6:5C:94:2F:D1:77:30:22:14:54:18:08:30:94:56:8E:
E3:4D:13:19:33:BF:DF:0C:2F:20:0B:CC:4E:F1:64:E3
Timestamp : Dec 7 14:50:55.718 2020 GMT
Extensions: none
Signature : ecdsa-with-SHA256
30:46:02:21:00:C9:35:85:2A:C2:2B:F3:E2:4D:87:AB:
97:72:C2:B7:3A:FB:A6:D6:02:F5:5F:A0:03:23:90:52:
99:E2:31:5D:87:02:21:00:FE:36:D3:75:13:8D:09:7D:
17:F7:E6:65:84:7A:E1:CB:E4:1E:3A:4F:23:F2:12:42:
F9:79:61:1D:08:28:63:10
[jawa.homebase.dk] Valid until 2021-01-21 06:36:57 UTC, skipping
Error: Invalid order DNS:mail.homebase.dk, DNS:www.mail.homebase.dk
[mail.homebase.dk] Error: Couldn't issue X.509 certificate!
accept: Invalid argument at /usr/libexec/lacme/webserver line 80.
Connection to jawa.homebase.dk closed.
Now, one issue is that it fails. I guess I made some error in the setup of
mail.homebase.dk config snippet.
Main issue I raise here, however, is more generally that it is not helpful
to the user that an internal-to-lacme call to /usr/libexec/lacme/webserver
passes broken arguments.
I guess it might be more helpful if such internal errors would trace the error
back to the user-facing command.
Better would be if which user-facing error caused the internal error,
obviously :-)
- Jonas
-----BEGIN PGP SIGNATURE-----
iQIzBAEBCgAdFiEEn+Ppw2aRpp/1PMaELHwxRsGgASEFAl/ORS0ACgkQLHwxRsGg
ASGK2BAAhOEB0d6wSqNA1PfZL1vzTaXL7eh6HInli410XGaAcv/aJT4R4BsIuL3k
KFq9Zi8NxlEnQPsdKgW+MzNRIvjsvW8aRk8rU+LowhocGtSIT5C1eDwhJn4/1Wsg
NIeJZTnoRaeGdR8EcRQmORf3mc5XKkmsqfY4tIvkk/fPxgNR+mpYs0ueW+Fy3cea
ueQq9cSrqA30eTuBH5Xv2z5ySIXBtCGVYDYHfebk8vOmskvcgCQIxqf2e3czlEg8
gKPRe0sFSGFsQ3NUy8yTVC3lQeasl9OHttfQtIp9tVIVTADg1nVRm7ojpc2Up5sd
reHF4xniiPAA6m/ZdCuW3EFcpJNYfQkvkdGc4oZHhzNKCnM4pO/V1fBCknGi+O5s
qjgOLxay4aw5mE6vL9twsJW101DhUp7rxGT+23R+jSf/JMhMpLw0d+r/NVLuIhkr
CxRfy/hp5FMtVAwNExfRWZnGkCwcXAc61t9Ngr/o1W9/UFT4UiYUtWtp37A5ih3d
/vSmeOb7QgZcUwsCpkO9CnIQ3pdNyzZDbcUczKtiffwt5my5SkCyJwoTa7r7Qyr0
hyVWXJYCLIBnYv+ZVp/jiq2s9N4WaokPuAxeKUnSj3vFKdIouEstM6rwCHxL44S3
vfpl+k4QcXu7e/ftvCCyDDyM0o9PsN20fsVd3hvis4ay7pYVIXw=
=5aev
-----END PGP SIGNATURE-----
Hi Jonas! received from the server had "status": "invalid", cf. RFC 8555 sec. 7.1.6. AFAICT there is no "detail" field with a human-friendly error message here, so ACME clients don't have much extra reasoning to provide. Shouldn't this be merged with #970458? You mean the order itself? It's indicated inside the brackets :-) The "Error: Invalid order" is spit out by `/usr/libexec/lacme/client`, attaching an order name to it would be doable but require a bit of refactoring as the name is currently not included in the IPC. Could you suggest a better error message here? As for "accept: Invalid argument", this comes from `/usr/libexec/lacme/webserver` , a single instance of which is used for the entire command so it's not specific to a particular order. I agree with you that it's not really useful, but the web server is (intentionally) pretty dumb and will only fail due to socket errors which don't really have a corresponding user-facing command. This particular error comes from the accept(2) syscall failing and setting errno to EINVAL. I *think* it's due to a race condition: lacme(8) shuts down the webserver's listening socket *before* terminating the child, causing any blocking accept(2) call to fail, and the scheduler might allow the error to be spit out before signal handling kicks in. Need to do more tests but I think simply shutting down the socket last would fix this. Cheers,
Quoting Guilhem Moulin (2020-12-08 12:04:15) fatal error, where the other bugreport is about seemingly harmless noise. But see my remark at bottom... I mean that it seems /usr/libexec/lacme/webserver is saying "I don't understand that command" but for a command given not by me but by lacme or some intermediary library, so instead of telling _me_ that error message it should tell it back to whoever called it, who could then compare the error with whatever it was specifically doing at the time and throwing something more meaningful and/or detailed out to me the user who has only the command-line call to compare against. Might not be needed to tie order name - see below... Generally, I think is might help if informed who says what. I.e. when passing on an error message either received from Letsencrypt or captured from stderr or spawned webserver, then a) mention the origin and b) indent the fowarded message to tie it to previous local message: [jawa.homebase.dk] Valid until 2021-01-21 06:36:57 UTC, skipping [mail.homebase.dk] request failed: rejected by Letsenctypt: Error: Invalid order DNS:mail.homebase.dk, DNS:www.mail.homebase.dk [mail.homebase.dk] Error: Couldn't issue X.509 certificate! [internal error] spurious message from internal webserver: accept: Invalid argument at /usr/libexec/lacme/webserver line 80. [internal error] spurious message from internal webserver: Connection to jawa.homebase.dk closed. You might consider using Log::Any - unless it is deliberate (for security reasons?) to limit use of shared modules. More specifically, it might help if lacme could correlate the various parts that me as operator might not be aware of. One of the errors I made was failing to enable the apache2 snippet to the vhost, which means requests initiated from Letscencrypt didn't get received by lacme. I imagine that when lacme sends a request out and waits for a response, then it could mention that no response was received at all. If lacme already does this, but only when debugging is enabled, then I suggest to simply raise verbosity on failure. _This_ part seems to be bug#970458, so I suggest re-posting there to reduce the number of issues discusses in same email thread :-) - Jonas
Quoting Jonas Smedegaard (2020-12-08 13:25:28) I had another failure today (again probably my fault, concretely), where I had a closer look at the debug output. Here is the output from a normal run: jonas@auryn:~$ mylacme-jawa newOrder jawa.homebase.dk root@jawa.homebase.dk's password: root@jawa.homebase.dk's password: Error: Invalid order DNS:jawa.homebase.dk, DNS:www.jawa.homebase.dk, DNS:lists.homebase.dk, DNS:www.lists.homebase.dk, DNS:list.homebase.dk, DNS:mail.homebase.dk [jawa.homebase.dk] Error: Couldn't issue X.509 certificate! accept: Invalid argument at /usr/libexec/lacme/webserver line 80. Connection to jawa.homebase.dk closed. I know that I edited the config to add hosts lists.homebase.dk, www.lists.homebase.dk, and list.homebase.dk - but even then the above is not really helpful on guiding me torwards which of them failed. And if this was an even bigger setup and/or I did not have a prior success to compare against, if would be even harder. I suggest to emit summary info from each response from Letscencrypt in a new --verbose mode - e.g. like this: jonas@auryn:~$ mylacme-jawa newOrder jawa.homebase.dk [[issuer]] Info: valid entry DNS:jawa.homebase.dk [[issuer]] Info: valid entry DNS:list.homebase.dk [[issuer]] Info: valid entry DNS:lists.homebase.dk [[issuer]] Info: valid entry DNS:mail.homebase.dk [[issuer]] Info: valid entry DNS:www.lists.homebase.dk [[issuer]] Info: pending entry DNS:www.jawa.homebase.dk [[issuer]] Error: Invalid order DNS:jawa.homebase.dk, DNS:www.jawa.homebase.dk, DNS:lists.homebase.dk, DNS:www.lists.homebase.dk, DNS:list.homebase.dk, DNS:mail.homebase.dk [mail.homebase.dk] Error: Couldn't issue X.509 certificate! [[internal]] Warning: accept: Invalid argument at /usr/libexec/lacme/webserver line 80. [[internal]] Warning: Connection to jawa.homebase.dk closed. An output like the above would help clue me in on which of the vhosts I might have configured wrongly, causing the whole request to get rejected. While at it, I notice that for certs with many hosts attached it takes a while to complete a request, and (in non-verbose mode) it is not possible to know if it is connections hanging or things are going smooth. It would be nice if in non-verbose mode executed from a real terminal, a dot was emitted for each response from Letsencrypt. - Jonas
Quoting Jonas Smedegaard (2020-12-09 11:22:19)
one.
Here is the output of a similarly failing setup using dehydrated, for
comparison:
# dehydrated --cron
# INFO: Using main config file /etc/dehydrated/config
# INFO: Using additional config file /etc/dehydrated/conf.d/hook.sh
# INFO: Using additional config file /etc/dehydrated/conf.d/secp384r1.sh
Processing boot.homebase.dk with alternative names: www.boot.homebase.dk
+ Checking domain name(s) of existing cert... unchanged.
+ Checking expire date of existing cert...
+ Valid till Dec 6 03:47:30 2020 GMT Certificate will expire
(Less than 30 days). Renewing!
+ Signing domains...
+ Generating private key...
+ Generating signing request...
+ Requesting new certificate order from CA...
+ Received 2 authorizations URLs from the CA
+ Handling authorization for www.boot.homebase.dk
+ Handling authorization for boot.homebase.dk
+ 2 pending challenge(s)
+ Deploying challenge tokens...
+ Responding to challenge for www.boot.homebase.dk authorization...
+ Cleaning challenge tokens...
+ Challenge validation has failed :(
ERROR: Challenge is invalid! (returned: invalid) (result: {
"type": "http-01",
"status": "invalid",
"error": {
"type": "urn:ietf:params:acme:error:unauthorized",
"detail": "Invalid response from http://www.boot.homebase.dk/.well-known/acme-challenge/t6YYZkoSfdJMHc_W1JcylRdlMof-Pe8SoVf0JE8rBrs [94.18.231.212]: \"\u003c!DOCTYPE HTML PUBLIC \\\"-//IETF//DTD HTML 2.0//EN\\\"\u003e\\n\u003chtml\u003e\u003chead\u003e\\n\u003ctitle\u003e404 Not Found\u003c/title\u003e\\n\u003c/head\u003e\u003cbody\u003e\\n\u003ch1\u003eNot Found\u003c/h1\u003e\\n\u003cp\"",
"status": 403
},
"url": "https://acme-v02.api.letsencrypt.org/acme/chall-v3/9182834478/p297vw",
"token": "t6YYZkoSfdJMHc_W1JcylRdlMof-Pe8SoVf0JE8rBrs",
"validationRecord": [
{
"url": "http://www.boot.homebase.dk/.well-known/acme-challenge/t6YYZkoSfdJMHc_W1JcylRdlMof-Pe8SoVf0JE8rBrs",
"hostname": "www.boot.homebase.dk",
"port": "80",
"addressesResolved": [
"94.18.231.212"
],
"addressUsed": "94.18.231.212"
}
]
})
I like how the default output is more verbose, and in case of error it
pukes even more details of the last part.
- Jonas