Ran into an issue today that I couldn't quickly find a useful fix for with Google, so here it is for posterity.
In the Adobe AEM (nee CQ) CMS there are configuration options for the HTTP service that can be set in
/system/console/configMgr. In CQ 5.x the service is called Day CQSE HTTP Service, and in AEM 6.x it's called Apache Felix Jetty Based Http Service.
It's very easy to accidentally lock yourself out of the entire CMS by saving an incorrect configuration for this service, since it controls the all basic HTTP server options - bind IP address/host, HTTP and HTTPS ports, whether HTTPS is enabled, HTTPS keystore, etc.
Unfortunately, the defaults shown in the config are not populated from the running instance, so it's likely they will not be correct. In CQ 5.x this config's default HTTP port is set to 8080, which isn't used for any CQ/AEM setups. In AEM 6.x the HTTP port defaults to 4502, which is correct for author, but if you have dispatcher or firewalls setup for a publish node on 4503 it will no longer be accessible.
By default there's no secure keystore name defined and the HTTPS port is 433 (!?) instead of the standard 443, so simply clicking the "Enable HTTPS" checkbox and saving will result in nonfunctioning HTTPS.
It's of course possible work around these issues if you know what you're doing before you start, but it's certainly not a simple or friendly setup.
Fortunately, backing out of the situation is fairly simple and friendly. It turns out the configuration is saved in the text file
crx-quickstart/launchpad/config/org/apache/felix/http.config. All you need to do is shut down CQ/AEM, remove that file, and restart CQ/AEM. Everything should be back to the way it was before you saved the config.
Thanks to Colin for helping me get this sorted out!