Communities

Writing
Writing
Codidact Meta
Codidact Meta
The Great Outdoors
The Great Outdoors
Photography & Video
Photography & Video
Scientific Speculation
Scientific Speculation
Cooking
Cooking
Electrical Engineering
Electrical Engineering
Judaism
Judaism
Languages & Linguistics
Languages & Linguistics
Software Development
Software Development
Mathematics
Mathematics
Christianity
Christianity
Code Golf
Code Golf
Music
Music
Physics
Physics
Linux Systems
Linux Systems
Power Users
Power Users
Tabletop RPGs
Tabletop RPGs
Notifications
Mark all as read
Q&A

How to correctly daemonize a Rocket-based app?

+3
−0

Summary

I'm refactoring a Rust-based service/daemon to move away from gRPC and use a Rocket-based API instead. I'm also using the daemonize crate to turn the foreground process into a background process.

The problem I've run into is that when I get the rocket-based service to successfully daemonize itself (i.e. daemon.start() returns Ok() result), it never receives the HTTP requests sent to it. In other words, no HTTP requests are ever logged as having been received by Rocket while the client side (i.e. curl -X GET ... in this case) keeps waiting for a response until it times out.

If I run the Rocket-based service in the foreground, i.e. without daemonize, then it works as expected. Also, my usage of daemonize before refactoring the service to use Rocket was showing correct behavior.

Questions

Given the constraints (see later):

  1. How can I get Rocket to work properly when turned into a background process with daemonize?
  2. Is there a better approach than using daemonize to turn a Rocket app into a background service?
  3. Any suggestions on gathering more data to troubleshoot this issue (e.g. determine if Rocket is failing to receive requests or refusing to process them - i.e. hung)? (I'll be running a few tests later with tcpdump to see what I can find out.)

The only thing that stands out here is my attempt to mix Rocket with Daemonize. The Rocket docs don't seem to have info on how to turn run it as a background service.

Constraints

  1. Launching the service as ./service & will not be possible in production;
  2. This is on an embedded device running Yocto and systemd is not available, so I cannot replace my usage of daemonize by a service unit

Please note that the above are not my decisions to make, are beyond my control, and are not likely to change. (I'm still looking into it, but, for the sake of this question, we should assume that it won't be possible.)

Code Ref

Here's the code using daemonize for reference:

fn launch_background(figment: Figment, config: Config) -> Rocket<Build> {
    let daemon = Daemonize::new()
        .pid_file(config.service.files.pid_file.clone())
        .chown_pid_file(true)
        .working_directory(config.service.context.working_dir.clone())
        .user(config.service.context.user.as_str())
        .group(config.service.context.group.as_str())
        .umask(config.service.context.umask)
        .stderr(File::create(config.service.files.log_file.clone()).unwrap());

    match daemon.start() {
        Ok(_) => launch_foreground(figment, config),
        Err(e) => ...
    }
}

Here's the code of how Rocket is being launched

fn launch_foreground(figment: Figment, config: Config) -> Rocket<Build> {
    rocket::custom(figment)
        .manage(...)
        .manage(...)
        .mount(...)
        .mount(...)
}

This is what Rocket shows when launched, regardless of whether it's daemonized or not (shortened for brevity):

[2022-07-09T00:46:00Z INFO  rocket::launch] 🔧 Configured for default.
[2022-07-09T00:46:00Z INFO  rocket::launch_] address: 127.0.0.1
[2022-07-09T00:46:00Z INFO  rocket::launch_] port: 50051
<snip>
[2022-07-09T00:46:00Z INFO  rocket::launch] 📬 Routes:
[2022-07-09T00:46:00Z INFO  rocket::launch_] (get_config) GET ...
[2022-07-09T00:46:00Z INFO  rocket::launch_] (get_downcast_info) GET ...
[2022-07-09T00:46:00Z INFO  rocket::launch_] (get_downcast_state) GET ...
<snip>
[2022-07-09T00:46:00Z INFO  rocket::launch] 🚀 Rocket has launched from http://127.0.0.1:50051

From the above, there're no obvious issues. There're no warnings or errors either. However, when sending HTTP requests to the mounted endpoints, nothing is ever logged by Rocket, so it's either not receiving any requests or it's hung in some way and it's not processing anything. I think the latter is more likely, but I'm not sure why that would happen either.

Update

This is a packet capture using Wireshark when the Rocket app is running as a background service/daemon. Notice that the 5th packet (#153) is a transport layer TCP ACK packet from the Rocket app side following the client's application layer HTTP GET request packet (#152), but the app layer on the service end (i.e. Rocket itself) did not respond. After a while, Keep-Alive packets start to show up until the connection is, eventually, terminated.

Rocket/cURL Network Traffic

Why does this post require moderator attention?
You might want to add some details to your flag.
Why should this post be closed?

0 comment threads

1 answer

+2
−0

I did not find a way to get Rocket and Daemonize to work together without problems. However, I was able to get a more recent version of start-stop-daemon installed in the system (v1.20.11), and was able to get an init script working. (The old version did not recognize some options such as PID file, I/O redirection, etc.) With this in place, I've completely removed the dependency on the daemonize crate and I'm now using Rocket by itself.

Here's the working init script I'm using, but as a template with placeholders for you to adapt it to your own needs:

#!/bin/bash
### BEGIN INIT INFO
# Provides:          <daemon>
# Required-Start:    $syslog $time $remote_fs
# Required-Stop:     $syslog $time $remote_fs
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Full Daemon Name
# Description:       Full description
### END INIT INFO
#
# Author:            ghost-in-the-zsh <ghost-in-the-zsh@example.com>
#
# Copy this script to `/etc/init.d/<daemon>` and run:
#   $ update-rc.d <daemon> defaults
#   $ /etc/init.d/<daemon> {start|stop|restart|status}
#
# Reference man pages:
#
#   1. init-d-script
#   2. update-rc.d
#   3. start-stop-daemon
#
set -e

# `init` consts; these names are meaningful to `init`
# see `man init-d-script`
NAME=<daemon>
DESC="Daemon Description Here"
DAEMON=/path/to-daemon/${NAME}
DAEMON_ARGS="--options for-your-daemon"
PIDFILE=/run/${NAME}.pid

# custom consts; these can be whatever we want
LOGFILE=/var/log/${NAME}.log
ACCOUNT=${NAME}
TIMEOUT=5

## service management functions
start() {
    echo "Starting ${NAME} ..."
    start-stop-daemon --start \
        --background \
        --startas ${DAEMON} \
        --name ${NAME} \
        --user ${ACCOUNT} \
        --chuid ${ACCOUNT} \
        --pidfile ${PIDFILE} \
        --output ${LOGFILE} \
        --make-pidfile \
        --oknodo \
        --quiet \
        -- ${DAEMON_ARGS}
    return ${?}
}

stop() {
    echo "Stopping ${NAME} ..."
    start-stop-daemon --stop \
        --user ${ACCOUNT} \
        --name ${NAME} \
        --oknodo \
        --pidfile ${PIDFILE} \
        --remove-pidfile \
        --retry ${TIMEOUT}
    return ${?}
}

status() {
    echo -n "${NAME} status: "
    start-stop-daemon --status \
        --name ${NAME} \
        --pidfile ${PIDFILE}

    local rc=${?}
    case ${rc} in
        0)
            echo "running"
            ;;
        1)
            echo "stopped"
            ;;
        *)
            echo "unknown"
            ;;
    esac
    return ${rc}
}

## case block to call management functions
case "${1}" in
    start)
        start
        retcode=${?}
        ;;
    stop)
        stop
        retcode=${?}
        ;;
    status)
        status
        retcode=${?}
        ;;
    restart)
        stop && start
        retcode=${?}
        ;;
    *)
        echo "Usage: ${0} {start|stop|restart|status}"
        retcode=1
esac

exit ${retcode}

Note that if you have the service utility installed, then you should be able to run service <daemon> {start|stop|restart|status} instead of using the full /etc/init.d/<daemon> path.

Also, as a bonus, here's the Systemd service unit I played around with in my local system, as an alternative for a possible future. It's also given as a template with placeholders:

# Enable : systemctl enable <daemon>.service
# Control: systemctl {start|stop|...} <daemon>.service
[Unit]
Description=Daemon Description Here
Documentation=https://<somewhere>.<something>/<project>
Before=<before>.service
After=<after-1>.target
After=<after-2>.target

[Service]
Type=simple
ExecStart=/path/to/<daemon>
Restart=always
User=<daemon>
Group=<daemon>
WorkingDirectory=/path/to/daemon/workdir
# systemd v240 and newer; otherwise must rely on `journalctl -eu <daemon>.service`
StandardOutput=append:/var/log/<daemon>.log
StandardError=append:/var/log/<daemon>.log

[Install]
WantedBy=multi-user.target

This last one is controlled via systemctl.

Why does this post require moderator attention?
You might want to add some details to your flag.

0 comment threads

Sign up to answer this question »