Author Archives: linuxlizard

About linuxlizard

Embedded Linux engineer working for Cradlepoint (part of Ericsson).

Linux Kernel Module Build Flags

Building out-of-tree Linux kernel modules is pretty easy. The page shows some basic info. There are some great ways to tinker with the CFLAGS (and other build flags).

The kernel module build uses rules/macros in the linux/scripts/Makefile.lib.

# deprecated old-style kernel build flags

The kernel best practice to set build flags is through the ccflags variable.

# these flags added to all C compiles
ccflags-y += -Wall -Werror

File-specific flags can also be set.

# Set DEBUG flag when compiling hello.c
CFLAGS_hello.o += -DDEBUG

A useful feature is also removing flags. I’m currently working with a set of vendor code that has a warning triggered by a newer version of GCC. The compile is failing because the vendor makefiles use -Werror. I can’t modify the code due to license restrictions but I can modify the makefile.

# turn off the -Werror flag
ccflags-remove-y += -Werror

Similarly, can remove flags for a specific file.

# turn off -Werror just for hello.c
CFLAGS_REMOVE_hello.o += -Werror

SSH Over Serial Port

What started as a rant on Mastadon has turned into an actual solution. The problem I need to solve is an out-of-band (non-network) connection between two machines. The problem also was I needed a super secure mechanism to connect to a local Linux box which was VPN’d into a corporate network. (I didn’t want a second network interface because I’m not smart enough to set up security to properly protect a dual-homed machine. The most secure connection is no connection.)

I started thinking: what if I could run a terminal aware protocol such as telnet carried over a serial port. Rather than have a raw character connection over the RS232, let’s carry something that has a terminal control plane. My first thought was hacking telnetd/telnet code to route over serial port. But a colleague (hi, KyleS!) suggested socat.

I’ve used socat before. It’s an amazing tool that can connect disparate socket/file descriptors together.

I prototyped the idea using my Fedora Linux laptop on one side and a Raspberry-Pi on the other. The two machines are connected via USB FTDI serial and a null modem adapter. I am using the R-Pi as the ssh server side. Rather than worry about telnet, I used the already running ssh.

On the Raspberry-Pi (server side):

 socat GOPEN:/dev/ttyUSB0,cfmakeraw,b115200 TCP:localhost:22,reuseaddr

On the Fedora Linux (client side):

 socat TCP-LISTEN:2222,bind=localhost,reuseaddr GOPEN:/dev/ttyUSB0,cfmakeraw,b115200

And then the connection from the client to the server is as simple as:

ssh -p 2222 pi@localhost

And now my serial connection can run Vim with full glorious syntax highlighting. The terminal resize SIGWINCH works. The connection is at 115200 so slightly slower than even 10Mbps Ethernet. But I have an out of band (non-Ethernet) connection between two machines so if (when) I bork the network config (often), I can recover the remote. I can edit code through the serial connection (edit my borked network scripts).

Next logical step is to set up the same socat/Dropbear trick with router firmware.

UPDATE 20230207. The initial connection can be very quirky if there is anything leftover in the serial port. Unlike a file handle or a network socket, closing the serial port will leave detritus available to read. If the ssh connection fails for some reason (in my case, hitting problems with known_hosts and ‘localhost’), the trash on the serial port will confuse the ssh at each ends.

I’m tinkering with some solutions. I’m hoping calls to ‘stty -F /dev/ttyUSB0 sane’ will help. Will update this post when I’ve found a reliable solution.

Static IP Address for Raspberry Pi (Rasbian)

Short story: edit /etc/dhcpcd.conf (man 5 dhcpcd.conf). Add the following block (usually there is a comment “Example static IP configuration”.

interface eth1
static ip_address=
#static ip6_address=fd51:42f8:caae:d92e::ff/64
static routers=
static domain_name_servers=

Long story: when our routers boot, we default to the IP address of When we need to do firmware upgrades using TFTP, uboot will also use When I have multiple routers, all running with the default IP address that need to be upgraded, I cannot connect all the routers to my same Linux box. Confusion will reign.

My solution has been to use a Raspberry Pi connected to each router. The R-Pi main Ethernet port is connected to my cubicle LAN. A second USB Ethernet dongle connects the R-Pi to the router.

The R-Pi acts as a firewall client between the prototype firmware (which can and often does have whacky bugs in my code). The R-Pi is plugged into one of router’s LAN ports. (Usually there’s a single specific port hardwired in uboot that comes up.) The R-Pi runs the TFTP server. The R-Pi runs minicom to the router’s debug serial port. I also run a serial console to the R-Pi itself. If the router testing goes doolally and confuses the R-Pi TCP/IP stack, I can connect over serial (yay, out of band control!) and reboot the R-Pi.

Before the Great R-Pi Shortage during the pandemic, I purchased enough R-Pi that I can have an individual R-Pi for each development router. I find this to be a very reliable solution, except when I forget which R-Pi connects to which router.

I Like Make.

With apologies to Tom T Hall.

In some of my posts I have casually mentioned
The fact that I like to write Make.
This little post is more to the point
Roll out the manual and lend me your ears

I like Make. It makes me a jolly good fellow
I like Make. It helps me unwind
And sometimes it makes me feel mellow
(all) (Makes him feel mellow)

C++’s too rough, Java costs too much
Rust puts my mouth in gear.
This little refrain should help me explain
As a matter of fact I like Make
(all) (he likes Make)

My boss stands mum when we’re having a scrum
And I’m building rules on the fly
She’s writing CMake and thinks I’m a flake
When I yell as devops goes by

I like Make. It makes me a jolly good fellow
I like Make. It helps me unwind
And sometimes it makes me feel mellow
(all) (Makes him feel mellow)

C++’s too rough, Java costs too much
Rust puts my mouth in gear.
This little refrain should help me explain
As a matter of fact I like Make
(all) (he likes Make)

Last night I dreamed and would not awake.
And I went to a place so swell
Oh, the build was clear and written with make
Then they turned it all into shell
(all) (aw)

I like Make. It makes me a jolly good fellow
I like Make. It helps me unwind
And sometimes it makes me feel mellow
(all) (Makes him feel mellow)

C++’s too rough, Java costs too much
Rust puts my mouth in gear.
This little refrain should help me explain
As a matter of fact I like Make
(all) (he likes Make)

Cross Compiling Rust (Notes)

I’ve been fiddling with cross compiling Rust to our ARM-64 platforms. I’m hitting a wall because Rust elfs require the system loader and C libs. I’m writing these notes to document what I’ve found so far. Would like to come back and discover how to build a static ARM-64 executable. Started here:

Step 1. Install Rust.

Step2. Install a C cross compiler. Rust uses the C linker.

% sudo apt install gcc-aarch64-linux-gnu 
% sudo dnf install gcc-aarch64-linux-gnu

Step 3. Add the cross compiler targets to Rust.

rustup target add aarch64-unknown-linux-gnu
rustup target add aarch64-unknown-linux-musl

Step 4. Add to the cargo config

…marvin:hello-rust% cat ~/.cargo/config
linker = "aarch64-linux-gnu-gcc"
linker = "aarch64-linux-gcc"

Step 5. Create a simple new project.

cargo new --bin hello-rust
cd hello-rust

Step 6. Build

cross build --target aarch64-unknown-linux-gnu

Now examining the final executable, I’m finding it requires the glibc loader and libraries. We’re using uClibc because historical reasons.

% file ./target/aarch64-unknown-linux-gnu/debug/hello-rust
./target/aarch64-unknown-linux-gnu/debug/hello-rust: ELF 64-bit LSB shared object, ARM aarch64, version 1 (SYSV), dynamically linked, interpreter /lib/, BuildID[sha1]=f62e92025124bc8018570c700b6e5faeecfdd671, for GNU/Linux 3.7.0, with debug_info, not stripped
% readelf -a ./target/aarch64-unknown-linux-gnu/debug/hello-rust
Program Headers:
Type Offset VirtAddr PhysAddr
FileSiz MemSiz Flags Align
PHDR 0x0000000000000040 0x0000000000000040 0x0000000000000040
0x0000000000000230 0x0000000000000230 R 0x8
INTERP 0x0000000000000270 0x0000000000000270 0x0000000000000270
0x000000000000001b 0x000000000000001b R 0x1
[Requesting program interpreter: /lib/]
Dynamic section at offset 0x3aac8 contains 30 entries:
Tag Type Name/Value
0x0000000000000001 (NEEDED) Shared library: []
0x0000000000000001 (NEEDED) Shared library: []
0x0000000000000001 (NEEDED) Shared library: []
0x0000000000000001 (NEEDED) Shared library: []

I tried the Rust musl cross compiler but hit a wall with some compile errors around, I think, hard/soft floats. The musl cross compiler I found here:

There are notes explaining musl can be static linked.

I should google those errors. Found:

Looks like it might be fixed but not in a released Rust? Need to investigate further.

Update: Installed the nightly build and voila! static link musl elf.

% rustup toolchain install nightly
% rustup default nightly-x86_64-unknown-linux-gnu
% cross build --target aarch64-unknown-linux-musl
% file ./target/aarch64-unknown-linux-musl/debug/hello-rust
./target/aarch64-unknown-linux-musl/debug/hello-rust: ELF 64-bit LSB executable, ARM aarch64, version 1 (SYSV), statically linked, with debug_info, not stripped

Copied the resulting elf to my firmware. IT WORKS! It’s big, but it works.

/var/tmp # ls -l hello-rust
 -rwxr--r--    1 root     root       3878608 Oct 26 12:55 hello-rust*
 /var/tmp # ./hello-rust
 Hello, world!

Message sequence number mismatch in libnl.

I ran into a problem with libnl and sequence numbers while working on my wifi scanner application. My app would subscript to the NL80211_CMD_NEW_SCAN_RESULTS and NL80211_CMD_SCHED_SCAN_RESULTS events. On receiving those events, I would send a NL80211_CMD_GET_SCAN.

Sometimes, I would start getting a -NLE_SEQ_MISMATCH on receiving the scan survey data. Once that occurred, I would receive no more scan data.

The libnl sequence numbers are described briefly here: They’re a simple way to associate a request with a response. They’re not bulletproof and do not claim to be.

The sequence numbers are tracked per socket in the private ‘struct nl_sock’. Both are initialized to time(0) in __alloc_socket().

struct nl_sock
     unsigned int s_seq_next;
     unsigned int s_seq_expect;

The nlmsghdr contains the sequence number. Note the same sequence number can be used multiple times. When there is more data that can fit in a single nl_msg, the data is broken across multiple nl_msg, indicated by flag NLM_F_MULTI (“Multipart message, terminated by NLMSG_DONE”).

struct nlmsghdr {
        __u32           nlmsg_len;      /* Length of message including header */
        __u16           nlmsg_type;     /* Message content */
        __u16           nlmsg_flags;    /* Additional flags */
        __u32           nlmsg_seq;      /* Sequence number */
        __u32           nlmsg_pid;      /* Sending process port ID */

The nlmsghdr->nlmsg_seq is assigned in nl_complete_msg() which is called before the nl_msg is sent to the nl_sock. The socket ‘next’ is incremented at this time.

        if (nlh->nlmsg_seq == NL_AUTO_SEQ) { 
                nlh->nlmsg_seq = sk->s_seq_next++;
                NL_DBG(3, "nl_complete_msg(%p): Increased next " \
                           "sequence number to %d\n",
                           sk, sk->s_seq_next);

The sequence number is checked in recvmsgs(), which is the core of libnl’s nl_msg receive handling. The recvmsgs() is responsible for calling several callbacks and for checking sequence numbers.

                if (hdr->nlmsg_type == NLMSG_DONE ||
                    hdr->nlmsg_type == NLMSG_ERROR ||
                    hdr->nlmsg_type == NLMSG_NOOP ||
                    hdr->nlmsg_type == NLMSG_OVERRUN) {
                        /* We can't check for !NLM_F_MULTI since some netlink
                         * users in the kernel are broken. */
                        NL_DBG(3, "recvmsgs(%p): Increased expected " \
                               "sequence number to %d\n",
                               sk, sk->s_seq_expect);

When the NLMSG_DONE is received, the expected sequence number is increased. If that DONE or ERROR aren’t received, the expected sequence number is never incremented.

The sequence number seq_next is advanced when a new nl_msg is created. The sequence number seq_expect is advanced when an incoming nl_msg is DONE or ERROR (or NOOP or OVERRUN, which I haven’t encountered yet).

The sequence number checking occurs also in recvmsgs(). In my code, I’m not using the NL_CB_SEQ_CHECK callback and leaving auto-ack mode enabled, so the sequence number checking in recvmsgs() is enforced. (As I’m still learning libnl, I was using the same pattern as the iw library.) Note this check happens before the DONE+ERROR check which increments the seq_expect.

                /* Sequence number checking. The check may be done by
                 * the user, otherwise a very simple check is applied
                 * enforcing strict ordering */
                if (cb->cb_set[NL_CB_SEQ_CHECK]) {
                        NL_CB_CALL(cb, NL_CB_SEQ_CHECK, msg);

                /* Only do sequence checking if auto-ack mode is enabled */
                } else if (!(sk->s_flags & NL_NO_AUTO_ACK)) {
                        NL_DBG(3, "recvmsgs(%p) : nlmsg_seq=%d s_seq_expect=%d\n", 
                                        sk, hdr->nlmsg_seq, sk->s_seq_expect);
                        if (hdr->nlmsg_seq != sk->s_seq_expect) {
                                if (cb->cb_set[NL_CB_INVALID])
                                        NL_CB_CALL(cb, NL_CB_INVALID, msg);
                                else {
                                        err = -NLE_SEQ_MISMATCH;
                                        nl_msg_dump(msg, stdout);
                                        goto out;

A simple transaction could look like:

7296872968new nl_msg; assigned 72968; seq_next++
729697296872968 (MULTI)
729697296872968 (MULTI)
729697296872968 (MULTI+DONE); seq match! seq_expect++

Moving on to the problem I encountered. The trigger of the problem is a 2nd CMD_NEW_SCAN_RESULTS or CMD_SCHED_SCAN_RESULTS received while already reading a CMD_GET_SCAN. The kernel doesn’t like interleaving the get-scan-results apparently so refuses with an -EBUSY error. The request increments the sk->seq_next but the error response nl_msg->seq doesn’t match the sk->seq_expect (which is tracking the previous request) and so the nl_msg is dropped before hitting the DONE check that would increment sk->seq_expect.

Once the sequence numbers get into this state, there is no exit. The nl_socket is perpetually at the wrong sequence number. The only solution is to close/re-open the socket on receiving a -NLE_SEQ_MISMATCH.

A better solution would be to avoid getting into this state in the first place. Perhaps not send a new CMD_GET_SCAN while a previous fetch is already running. I’m still tinkering with solutions.

IEEE 802.11 Standards, Alphabetically.

IEEE802.11 Standards.

One Letter, Alphabetical Order. IEEE Std 802.11a-1999: High-speed Physical Layer in the 5 GHz Band (Amendment 1) IEEE Std 802.11b-1999: Higher-Speed Physical Layer Extension in the 2.4 GHz Band (Amendment 2)

IEEE Std 802.11b-1999/Corrigendum 1-2001: Higher-speed Physical Layer (PHY) extension in the 2.4 GHz band (Corrigendum 1 to Amendment 2) IEEE Std 802.11d-2001: Specification for operation in additional regulatory domains (Amendment 3) IEEE Std 802.11e-2005: Medium Access Control (MAC) Quality of Service Enhancements (Amendment 8) IEEE Std 802.11g-2003: Further Higher Data Rate Extension in the 2.4 GHz Band (Amendment 4) IEEE Std 802.11h-2003: Spectrum and Transmit Power Management Extensions in the 5 GHz band in Europe (Amendment 5) IEEE Std 802.11i-2004: Medium Access Control (MAC) Security Enhancements (Amendment 6) TKIP. IEEE Std 802.11j-2004: 4.9 GHz–5 GHz Operation in Japan (Amendment 7) IEEE Std 802.11k-2008: Radio Resource Measurement of Wireless LANs (Amendment 1) RMM Radio Resource Management. IEEE Std 802.11n-2009: Enhancements for Higher Throughput (Amendment 5). AKA WiFi-5 IEEE Std 802.11p-2010: Wireless Access in Vehicular Environments (Amendment 6) IEEE Std 802.11r-2008: Fast Basic Service Set (BSS) Transition (Amendment 2). Fast Roaming IEEE Std 802.11s-2011: Mesh Networking (Amendment 10) IEEE Std 802.11u-2011: Internetworking with External Networks (Amendment 9). QoS. IEEE Std 802.11v-2011: Wireless Network Management (Amendment 8). WNM. IEEE Std 802.11w-2009: Protected Management Frames (Amendment 4). PMF. IEEE Std 802.11y-2008: 3650–3700 MHz Operation in USA (Amendment 3) Extra spectrum. Never caught on, no radios support it. IEEE Std 802.11z-2010: Extensions to Direct-Link Setup (DLS) (Amendment 7)

Two Letter ‘a’, Alphabetical Order. 

IEEE Std 802.11aa-2012: MAC Enhancements for Robust Audio Video Streaming (Amendment 2) 10 IEEE Std 802.11ac-2013: Enhancements for Very High Throughput for Operation in Bands below 6 GHz (Amendment 4)  IEEE Std 802.11ad-2012: Enhancements for Very High Throughput in the 60 GHz Band (Amendment 3)

IEEE Std 802.11ae-2012: Prioritization of Management Frames (Amendment 1)   IEEE Std 802.11af-2013: Television White Spaces (TVWS) Operation (Amendment 5)  HaLow  900 MHz  FILS Fast Initial Link Setup China Millimeter Wave (CMMW) (45GHz) pre-association discovery of services (extends 802.11u)  HE (High Efficiency)  aka WiFi6 Second WiGig standard (60 GHz) (cf. 802.11ad)

Two Letter ‘b’, Alphabetical Order.  Wake-up Radio (WUR) Extremely High Throughput (EHT) aka WiFi7

Command Line JSON

I just stumbled across a wonderful tool: command line JSON validation/pretty-print.

I often work with JSON with our routers. I use curl to read from our API endpoints which return JSON. Getting back large blobs of JSON is useful but hard to read all jumbled together.

% curl –basic –user admin:${CP_PASSWORD}

Command line output of router wifi survey api output.

Now instead I can pipe to python3 -m json.tool and the JSON will be cleanly formatted and humanly readable.

C++ filesystem vs experimental::filesystem

I’m tinkering with C++ again.  I’m trying to use as much standard C++17 as possible because it’s a good reason to learn the C++17 standards.

Boost has a filesystem library. The library evolved into a core C++ library standard. GCC 7 has the filesystem module in the std::experimental namespace. GCC 7 moved the filesystem into the top level std:: namespace.

I’m trying to write a program that works in both recent Fedora and Ubuntu. Fedora29 uses GCC8, Ubuntu18.04 uses GCC7. Dang it. The experimental filesystem module still exists in GCC8 (/usr/include/c++/8/experimental/filesystem) but I’d like to use the latest and greatest available, if possible (/usr/include/c++/8/filesystem).

I can rename the namespace via the “namespace fs = ” which is really useful. Is a bit like Pythons “import module as newname” but I need to know which header and which namespace to rename. My first thought was duplicating the __cplusplus macro used in the filesystem header in the two aforementioned paths in Fedora.

#if __cplusplus >= 201703L
#include <filesystem> // gcc8 (Fedora29+)
namespace fs = std::experimental::filesystem;
#elif __cplusplus >= 201103L
#include <experimental/filesystem> // gcc7 (Ubuntu 18.04)
namespace fs = std::experimental::filesystem;

No such luck. The __cplusplus macro is set based on the -stdc++NN command line arg. So my build under Ubuntu also uses -stdc++17 and would hit the __cplusplus >= 201703L.

I don’t want to specifically test for GCC version in an #ifdef. I’d also like to eventually build under OSX (Clang+XCode) and Windows (VC++).

I’m using CMake. I think I’m going to have to do some test compiles with CMake to determine which header to use.