Welcome to Software Development on Codidact!
Will you help us build our independent community of developers helping developers? We're small and trying to grow. We welcome questions about all aspects of software development, from design to code to QA and more. Got questions? Got answers? Got code you'd like someone to review? Please join us.
How to correctly daemonize a Rocket-based app?
Summary
I'm refactoring a Rust-based service/daemon to move away from gRPC and use a Rocket-based API instead. I'm also using the daemonize crate to turn the foreground process into a background process.
The problem I've run into is that when I get the rocket-based service to successfully daemonize itself (i.e. daemon.start() returns Ok() result), it never receives the HTTP requests sent to it. In other words, no HTTP requests are ever logged as having been received by Rocket while the client side (i.e. curl -X GET ... in this case) keeps waiting for a response until it times out.
If I run the Rocket-based service in the foreground, i.e. without daemonize, then it works as expected. Also, my usage of daemonize before refactoring the service to use Rocket was showing correct behavior.
Questions
Given the constraints (see later):
- How can I get Rocket to work properly when turned into a background process with daemonize?
- Is there a better approach than using daemonize to turn a Rocket app into a background service?
- Any suggestions on gathering more data to troubleshoot this issue (e.g. determine if Rocket is failing to receive requests or refusing to process them - i.e. hung)? (I'll be running a few tests later with
tcpdumpto see what I can find out.)
The only thing that stands out here is my attempt to mix Rocket with Daemonize. The Rocket docs don't seem to have info on how to turn run it as a background service.
Constraints
- Launching the service as
./service &will not be possible in production; - This is on an embedded device running Yocto and
systemdis not available, so I cannot replace my usage ofdaemonizeby a service unit
Please note that the above are not my decisions to make, are beyond my control, and are not likely to change. (I'm still looking into it, but, for the sake of this question, we should assume that it won't be possible.)
Code Ref
Here's the code using daemonize for reference:
fn launch_background(figment: Figment, config: Config) -> Rocket<Build> {
let daemon = Daemonize::new()
.pid_file(config.service.files.pid_file.clone())
.chown_pid_file(true)
.working_directory(config.service.context.working_dir.clone())
.user(config.service.context.user.as_str())
.group(config.service.context.group.as_str())
.umask(config.service.context.umask)
.stderr(File::create(config.service.files.log_file.clone()).unwrap());
match daemon.start() {
Ok(_) => launch_foreground(figment, config),
Err(e) => ...
}
}
Here's the code of how Rocket is being launched
fn launch_foreground(figment: Figment, config: Config) -> Rocket<Build> {
rocket::custom(figment)
.manage(...)
.manage(...)
.mount(...)
.mount(...)
}
This is what Rocket shows when launched, regardless of whether it's daemonized or not (shortened for brevity):
[2022-07-09T00:46:00Z INFO rocket::launch] 🔧 Configured for default.
[2022-07-09T00:46:00Z INFO rocket::launch_] address: 127.0.0.1
[2022-07-09T00:46:00Z INFO rocket::launch_] port: 50051
<snip>
[2022-07-09T00:46:00Z INFO rocket::launch] 📬 Routes:
[2022-07-09T00:46:00Z INFO rocket::launch_] (get_config) GET ...
[2022-07-09T00:46:00Z INFO rocket::launch_] (get_downcast_info) GET ...
[2022-07-09T00:46:00Z INFO rocket::launch_] (get_downcast_state) GET ...
<snip>
[2022-07-09T00:46:00Z INFO rocket::launch] 🚀 Rocket has launched from http://127.0.0.1:50051
From the above, there're no obvious issues. There're no warnings or errors either. However, when sending HTTP requests to the mounted endpoints, nothing is ever logged by Rocket, so it's either not receiving any requests or it's hung in some way and it's not processing anything. I think the latter is more likely, but I'm not sure why that would happen either.
Update
This is a packet capture using Wireshark when the Rocket app is running as a background service/daemon. Notice that the 5th packet (#153) is a transport layer TCP ACK packet from the Rocket app side following the client's application layer HTTP GET request packet (#152), but the app layer on the service end (i.e. Rocket itself) did not respond. After a while, Keep-Alive packets start to show up until the connection is, eventually, terminated.
1 answer
I did not find a way to get Rocket and Daemonize to work together without problems. However, I was able to get a more recent version of start-stop-daemon installed in the system (v1.20.11), and was able to get an init script working. (The old version did not recognize some options such as PID file, I/O redirection, etc.) With this in place, I've completely removed the dependency on the daemonize crate and I'm now using Rocket by itself.
Here's the working init script I'm using, but as a template with placeholders for you to adapt it to your own needs:
#!/bin/bash
### BEGIN INIT INFO
# Provides: <daemon>
# Required-Start: $syslog $time $remote_fs
# Required-Stop: $syslog $time $remote_fs
# Default-Start: 2 3 4 5
# Default-Stop: 0 1 6
# Short-Description: Full Daemon Name
# Description: Full description
### END INIT INFO
#
# Author: ghost-in-the-zsh <ghost-in-the-zsh@example.com>
#
# Copy this script to `/etc/init.d/<daemon>` and run:
# $ update-rc.d <daemon> defaults
# $ /etc/init.d/<daemon> {start|stop|restart|status}
#
# Reference man pages:
#
# 1. init-d-script
# 2. update-rc.d
# 3. start-stop-daemon
#
set -e
# `init` consts; these names are meaningful to `init`
# see `man init-d-script`
NAME=<daemon>
DESC="Daemon Description Here"
DAEMON=/path/to-daemon/${NAME}
DAEMON_ARGS="--options for-your-daemon"
PIDFILE=/run/${NAME}.pid
# custom consts; these can be whatever we want
LOGFILE=/var/log/${NAME}.log
ACCOUNT=${NAME}
TIMEOUT=5
## service management functions
start() {
echo "Starting ${NAME} ..."
start-stop-daemon --start \
--background \
--startas ${DAEMON} \
--name ${NAME} \
--user ${ACCOUNT} \
--chuid ${ACCOUNT} \
--pidfile ${PIDFILE} \
--output ${LOGFILE} \
--make-pidfile \
--oknodo \
--quiet \
-- ${DAEMON_ARGS}
return ${?}
}
stop() {
echo "Stopping ${NAME} ..."
start-stop-daemon --stop \
--user ${ACCOUNT} \
--name ${NAME} \
--oknodo \
--pidfile ${PIDFILE} \
--remove-pidfile \
--retry ${TIMEOUT}
return ${?}
}
status() {
echo -n "${NAME} status: "
start-stop-daemon --status \
--name ${NAME} \
--pidfile ${PIDFILE}
local rc=${?}
case ${rc} in
0)
echo "running"
;;
1)
echo "stopped"
;;
*)
echo "unknown"
;;
esac
return ${rc}
}
## case block to call management functions
case "${1}" in
start)
start
retcode=${?}
;;
stop)
stop
retcode=${?}
;;
status)
status
retcode=${?}
;;
restart)
stop && start
retcode=${?}
;;
*)
echo "Usage: ${0} {start|stop|restart|status}"
retcode=1
esac
exit ${retcode}
Note that if you have the service utility installed, then you should be able to run service <daemon> {start|stop|restart|status} instead of using the full /etc/init.d/<daemon> path.
Also, as a bonus, here's the Systemd service unit I played around with in my local system, as an alternative for a possible future. It's also given as a template with placeholders:
# Enable : systemctl enable <daemon>.service
# Control: systemctl {start|stop|...} <daemon>.service
[Unit]
Description=Daemon Description Here
Documentation=https://<somewhere>.<something>/<project>
Before=<before>.service
After=<after-1>.target
After=<after-2>.target
[Service]
Type=simple
ExecStart=/path/to/<daemon>
Restart=always
User=<daemon>
Group=<daemon>
WorkingDirectory=/path/to/daemon/workdir
# systemd v240 and newer; otherwise must rely on `journalctl -eu <daemon>.service`
StandardOutput=append:/var/log/<daemon>.log
StandardError=append:/var/log/<daemon>.log
[Install]
WantedBy=multi-user.target
This last one is controlled via systemctl.
0 comment threads
0 comment threads