Fix for rngd: read error

[neoaeon]

rngd was getting a lot of fips errors and read errors, eventually shutting down. This appears to be due to errant EOFs and kill commands showing up in the random stream. Using stty to turn off these characters fixed it, now rngd runs normally.

I modified the udev rule as follows to fix it on-plug (and on-boot).
SUBSYSTEM=="tty", ATTRS{product}=="TrueRNG", SYMLINK+="TrueRNG", RUN+="/bin/stty -icanon -isig -F /dev/%k speed 3000000"

You can use stty -F /dev/TrueRNG to show it’s current settings, if it’s missing -icanon and -isig you may experience the same.

You can test for the errant configuration using dd to read directly from TrueRNG, you’ll notice it die within seconds in most cases (CTRL+C to abort if it doesn’t die).

[root@localhost ~]# stty -F /dev/TrueRNG
speed 9600 baud; line = 0;
-brkint -imaxbel
[root@localhost ~]# dd if=/dev/TrueRNG bs=64 of=/dev/null
3+49 records in
3+49 records out
1315 bytes (1.3 kB) copied, 0.172402 s, 7.6 kB/s

Versus when it’s functioning, CTRL+C’d after ~30 seconds:

[root@localhost ~]# stty -F /dev/TrueRNG
speed 3000000 baud; line = 0;
min = 1; time = 0;
-brkint -imaxbel
-isig -icanon
[root@localhost ~]# dd if=/dev/TrueRNG bs=64 of=/dev/null
^C15380+13728 records in
15379+13728 records out
1606549 bytes (1.6 MB) copied, 33.1986 s, 48.4 kB/s

My logs showed the following prior to the fix.

Mar  7 18:29:50 localhost rngd: read error
Mar  7 18:29:50 localhost rngd: failed fips test
Mar  7 18:29:51 localhost rngd: failed fips test
Mar  7 18:29:51 localhost rngd: read error
Mar  7 18:29:51 localhost rngd: read error
Mar  7 18:47:12 localhost rngd: read error
Mar  7 18:47:12 localhost rngd: read error
Mar  7 18:47:12 localhost rngd: read error
Mar  7 18:47:12 localhost rngd: read error
Mar  7 18:47:13 localhost rngd: read error
Mar  7 18:47:13 localhost rngd: failed fips test
Mar  7 19:14:02 localhost rngd: read error
Mar  7 19:18:35 localhost rngd: read error
Mar  7 19:20:55 localhost rngd: read error
Mar  7 19:27:30 localhost rngd: read error
Mar  7 19:27:30 localhost rngd: failed fips test
Mar  7 19:27:30 localhost rngd: read error
Mar  7 19:27:30 localhost rngd: failed fips test
Mar  7 19:37:58 localhost rngd: failed fips test
Mar  7 19:37:58 localhost rngd: read error
Mar  7 19:37:59 localhost rngd: failed fips test
Mar  7 19:37:59 localhost rngd: read error
Mar  7 19:37:59 localhost rngd: failed fips test
Mar  7 19:38:13 localhost rngd: read error
Mar  7 19:38:14 localhost rngd: read error
Mar  7 19:38:14 localhost rngd: failed fips test
Mar  7 19:38:15 localhost rngd: read error
Mar  7 19:38:15 localhost rngd: read error
Mar  7 19:38:16 localhost rngd: failed fips test
Mar  7 19:38:16 localhost rngd: failed fips test
Mar  7 19:38:16 localhost rngd: read error
Mar  7 19:38:16 localhost rngd: read error
Mar  7 19:38:16 localhost rngd: failed fips test
Mar  7 19:48:51 localhost rngd: failed fips test
Mar  7 19:48:51 localhost rngd: read error
Mar  7 19:48:55 localhost rngd: read error
Mar  7 19:48:55 localhost rngd: failed fips test
Mar  7 19:48:55 localhost rngd: too many FIPS failures, disabling entropy source
Mar  7 19:48:55 localhost rngd: No entropy sources working, exiting rngd

system: CentOS 6 x86_64
credit: Arduino related sites.

• This topic was modified 11 years, 1 month ago by neoaeon. Reason: formatting

[Ubld.it Staff]

I would be lying if I said I wasn’t embarassed by not explicitly setting the termios settings in the udev rules. We’ve been testing the TrueRNG on mostly debian based distros for months now and never ran into this issue. I would think the default port states would be identical. In fact we are running a long entropy test as we speak capturing 100gigs of data to analyze, and it hasn’t hiccuped 1 bit.

While inspecting the rng-tools daemon init code just now, they don’t set any termios settings (and why should they?, they don’t know anything about the source file descriptor). While in canonical mode, and with other line mode features enabled I can picture this happening. My knee jerk reaction says we want to use raw (which is basically -everything). But I will experiment tomorrow starting with the same distro as you.

I really appreciate your patience and effort thus far. I’m going to spend the next few days installing a few major distros to inspect this case further and update our udev rules with your fix. I will say setting the baud rate shouldn’t make any difference at all, as the serial port is virtualized it’s at the mercy of the usb bus… but it doesn’t hurt anything to set it to a high number.

[neoaeon]

Based on your input I switched to stty -echoe -echok -F /dev/%k raw, which seems to be working better.

Do you have any thoughts on -echoe -echok? I can’t tell if those apply to input or output. If applied to output then lacking them could create predictable patterns, if applied to input then having them flagged won’t hurt anything as there is no input.

Thank you very much for the follow up.

[Ubld.it Staff]

So good news is I was able to reproduce your problem exactly with Centos 6. Here’s some initial observations;

Centos 6.5 – 2.6.32-431 kernel – default termios settings

speed 9600 baud; line = 0;
-brkint -imaxbel

Mint 15 – 3.8.0-33 kernel

speed 57600 baud; line = 0;
eof = ^A; min = 1; time = 0;
-brkint -icrnl ixoff ixany -imaxbel
-opost -onlcr
-isig -icanon -iexten -echo -echoe -echok -echoctl -echoke

First thing to notice is Mint 15 is doing -everything (which is what we want, and explains why I wasn’t seeing this problem initially). The next thing to notice is 3.xxx kernel vs 2.6.xx kernel which is where I think the biggest disconnect between distros is.

So I have the advantage of using a USB analyzer to see what exactly is going where to see what effect these things have. First thing I notice is on the CentOS side, it’s actually sending characters upstream to the microcontroller. This isn’t an issue, the micro just drops then, but this is wasted bandwidth (maybe about 1k/s difference if that). Adding stty -echo seems to fix that and characters are no longer being sent upstream.

The second part is the stream stopping. So when the stream stops the DTR flag is killed, this means linux actually closed the port. It appears that linux sees the EOF/KILL/Whatever character in the stream randomly (which you can see different stop points to prove that) it closes the port and thinks its done reading data. After doing stty -icanon, the stream is perfect (from the analyzer side) and doesn’t do anything odd any more.

Now looking at -echoe and -echok these seem to only affect what happens when data is sent to the microcontroller from linux, such as adding a nl after cr.

So what we really need is stty raw -echo

Testing this on both distros produces the following.

CentOS 6.5

speed 9600 baud; line = 0;
min = 1; time = 0;
-brkint -icrnl -imaxbel
-opost
-isig -icanon -echo

The delta between the initial state is -icrnl -opost -sig -icanon -echo

On Mint 15 we get

speed 57600 baud; line = 0;
eof = ^A; min = 1; time = 0;
-brkint -icrnl -imaxbel
-opost -onlcr
-isig -icanon -iexten -echo -echoe -echok -echoctl -echoke

The delta here is exactly the same, except the initial state also had ixoff ixany. Stty shows this for those flags

 * [-]ixany      let any character restart output, not only start character
   [-]ixoff      enable sending of start/stop characters

So I decided to go with stty raw -echo -ixoff

I’m in the process of updating the udev rules on the web site with the changes.

Again I appreciate your help, please let me know if there is anything we can do for you in return.

[rcs]

USE CASE: Newly purchased TrueRNGpro with current rule on MINT 19 with a 4.15.0-38-generic kernel.

PROBLEM: Using rng-tools 5-0ubuntu4 from the default MINT repositories execution of /etc/init.d/rng-tools start posts rngd read errors to /var/log/syslog and rngd halts.

DIAGNOSIS: TrueRNGpro functions properly (e.g. ACT light comes on briefly when a command such as head -c 512K /dev/TrueRNG > /tmp/foo.dat is executed). The default stty configuration shown below on MINT 19 or a modified rule to match the configuration given earlier in this thread for MINT 15 both exhibit rngd read errors.

CURE: rngd does not produce errors. Any ideas? Thank you in advance!

==== RULE ====

SUBSYSTEM=="tty", ATTRS{product}=="TrueRNGpro", SYMLINK+="TrueRNG", RUN+="/bin/stty raw -echo -ixoff -F /dev/%k speed 3000000"
ATTRS{idVendor}=="16d0", ATTRS{idProduct}=="0aa0", ENV{ID_MM_DEVICE_IGNORE}="1"

==== STTY CONFIGURATION ====

# stty -F /dev/TrueRNG
speed 3000000 baud; line = 0;
min = 1; time = 0;
-brkint -icrnl -imaxbel
-opost
-isig -icanon -echo

==== RNGD ERROR FROM SYSLOG ====

rngd: read error
rngd: message repeated 99 times: [ read error]
rngd: No entropy sources working, exiting rngd

• This reply was modified 6 years, 5 months ago by rcs.

[rcs]

This was an error on my part. When I redid the setup from scratch I failed to set up rng-tools correctly.

CURE: Uncomment HRNGDEVICE specifier in /etc/default/rng-tools file.

...
# Set to the input source for random data, leave undefined
# for the initscript to attempt auto-detection.  Set to /dev/null
# for the viapadlock and tpm drivers.
#HRNGDEVICE=/dev/hwrng
#HRNGDEVICE=/dev/null
HRNGDEVICE=/dev/TrueRNG
...

• This reply was modified 6 years, 5 months ago by rcs.

[Author]

Posts