feat: Integrate BME680 Bosch BSEC support for RAK4631

[NickDunklee]

I'm on a wild tear today! Trying to take advantage of it.

This is a consolidation of my changes for BME680 on RAK4631 nodes. I will close my other PRs related to this and link back to this one.

Background on change:

This change replaces the Adafruit BME680 driver on RAK4631 with the Bosch BSEC library. Other boards continue to use the existing Adafruit path via ENV_INCLUDE_BME680.

This makes the IAQ portion of the sensor functional, and more accurate. It also contains the math and/or CayenneLPP fixes from my other PRs. The Bosch code also appears to handle calibrating sensor aging as well, whereas the Adafruit code is just looking at blind values that can drift with time. Pretty cool to see this shooting out useful data!

RAK4631 platform.io is set to override to ENV_INCLUDE_BME680_BSEC while leaving the Adafruit code for other node types. (If this becomes applicable for other node types in future, awesome! I just don't have hardware to test against.)

Using the BSEC library introduces IAQ sensor calibration, and saves the calibration state periodically so it does not have to calibrate again later.

At startup the IAQ sensor takes 30 minutes to heat and to hit a baseline, then starts calibrating. Once calibrated, it will save those settings and will only write settings again if calibration falls back and restores back to state 3.

This fix also has the gas resistance math fix that was in pull 2146 so the adafruit path also can at least show accurate values instead of looping negative.

Also includes the fix from pull 2149 so the pressure output isn't truncated to 1hPa steps.

Fixes/Changes:

• Add bsec_config_iaq[] with the 3.3V/3s-LP/28d calibration profile
• BSEC init applies setConfig() for voltage-correct heater targeting
• IAQ, heat-compensated temperature/humidity, pressure, and altitude reported over CayenneLPP
• IAQ accuracy reported as analog input over CayenneLPP (0,1,2,3)
• Calibration state persisted to /bsec_state.bin on nRF52 internal flash; written only when iaqAccuracy improves to >= 2, should keep write frequency well within flash endurance over device lifetime
• Fix non-BSEC query_bme680: float pressure division, addGenericSensor for gas resistance (was addAnalogInput, overflows at > 327 Ohm)
• loop() correctly gated for both GPS and BSEC-only builds
• Add fix_bsec_lib.py extra_script to resolve nRF52840 hard-float ABI mismatch in Bosch's PlatformIO packaging, silly Bosch

One general note outside of this code change: I noticed while BME680 functions in companion nodes, since companion nodes run Bluetooth, BLE preempts the CPU, and can do so mid-I2C-transaction.

This can cause the BME680 to see an anomaly and drop calibration and start a recalibrate. This is behavior that will exist (and has existed) regardless of using the Adafruit or Bosch paths.

This particular companion behavior does not seem to occur in sensor or repeater nodes since their BLE is off. Probably affects other I2C devices as well.

Tests:

• RAK19003
• RAK19007
• RAK19001
• repeater, sensor, companion

24 Hour Grafana Stats Example: (Graph gap is from a service restart, not the node.)

[NickDunklee]

My other two PRs are closed and this is the source of truth for my BME680 stuff now, trying to not make too many GitHub messes 😂

🎉

[cwichura]

There doesn't appear to be any backoff here. If accuracy is updating every loop(), you will be doing a lot of filesystem writes. And there are concerns in other threads about RAKs corrupting their filesystem during writes under various trigger conditions. Maybe update this to include a millis() check as well, so that it isn't constantly overwriting the state file in rapid succession? How important is it that it update the state persistence? Could it limit it to every minute? five minutes? Or does the BSEC library itself already gate accuracy checks and so you have an inherent backoff from the library?

[NickDunklee]

Oooh, that's a good observation, I like it. Once the sensor gets into a calibrated state (except for on companions) it just remains there for hours on end. If the sensor gets into a calibration flux like on companions, it would otherwise be writing very frequently as you said. I will check into that.

[NickDunklee]

Did some interesting reading through the datasheet and integration guide and looked at the loop code a bit more versus my dashboard data going back a few weeks.

The Bosch example saves just over every 8 hours, some home assistant sites save every 6 by default.

Looking through my code loop, it seems two behaviors happen:

• My implementation actually writes less - in fact too less. On a properly operating node, it will save once hitting state 3 and never save again since the state transitions cease. This will actually result in the sensor having to do a full recalibrate when the node is rebooted as the calibration data will be very stale.
• On a node not functioning proper like a companion, however, it will write more because the state keeps jumping around due to I2C being interrupted.

Companion flickering example:

It also looks like the sensor has four power states, from the Bosch doc:

• Ultra low power (ULP) mode is designed for battery-powered and/or frequency-coupled devices over extended periods of time. This mode features an update rate of 300 seconds and an average current consumption of <0.1 mA
• Quick Ultra-low power (q-ULP) has a 3 s data rate for Temprature, pressure and humidity w/o significantly increasing the power consumption compared to ULP.
• Low power (LP) mode that is designed for interactive applications where the air quality is tracked and observed at a higher update rate of 3 seconds with a current consumption of <1 mA
• Continuous (CONT) mode provides an update rate of 1 Hz and shall only be used short-term for use cases that incorporate very fast events or stimulus.

Updating every 5 minutes (ULP) seems a reasonable trade-off for battery-powered nodes, and in general.

So I am thinking:

• Save state upon first successful calibration, and then every 8 hours from there on out
• Will check into what power mode it runs by default and if ULP can be triggered upon initialization if it is not already in that state by default

My angle being "lets be useful but also conserve as much power as possible."

Any thoughts there?

[cwichura]

Seems reasonable to me.

Saving state every eight hours makes sense, considering you say it's what Bosch's own examples do. Even companions tend to not get turned on/off a lot, methinks. Problem is, the companions that do get turned off, you don't really get a "last gasp" notification to save state when the power switch is flipped and just instantly kills the board. I also don't know how many people would put a BME680 in a companion device -- I expect they are primarily used in infrastructure nodes that don't move around. So maybe just don't worry about it, and stick with eight hours on companions as well...

My BME680 nodes are primarily solar powered, so definitely interested in conserving power so long as it doesn't make the sensor readings drift unnecessarily. You could always go the route of adding a cli config variable and let the operator decide if they want faster readout when the node has hardline power (or at least has it most of the time). Mostly people would probably be selecting between ULP or q-ULP. MeshCore firmware telemetry doesn't really lend itself towards LP or CONT (except maybe in the sensor role firmware if a timeseries is created over the sensor?).

[cwichura]

Or just go with ULP like you suggest and if someone wants q-ULP, let them add the CLI config variable at that time... :)

[NickDunklee]

It looks like to add mode-switching on-node is a no-go. From the integration guide:

To achieve best gas sensor performance, the user shall not switch between LP and ULP modes during the life-time of a given sensor.

I'm also finding conflicting information as to ULP possibly taking hours to days to calibrate vs LP, and also that ULP measurement cycles may be longer and/or take considerable more power due to extended heating, but the answers there also seem a bit vague. I had considered trying a mode-switch, calibrate in LP and then switch to ULP once calibrated and reload the calibration file, but the guide blurb above makes that a no-go as well.

So probably just one mode, no reason to CLI, and if ULP has the two negatives of higher power hit during measurement (which could trigger a brownout in low battery states), and of calibration taking forever, it might not be worth the effort.

Also found out q-ULP is an on-demand measurement mode of ULP to use in conjunction with ULP, so yeah probably not worth it either.

I'm going to flash a node with ULP configured and let it cook for a bit and see what comes out. Really curious how long calibration takes, and probably faster to see with my eyes than sort through the Internet. Power logging may end up showing the different behavior as well.

[NickDunklee]

ULP made it to calibration state 0 -> 1 in 30 minutes, but I imagine now the transition -> 2 -> 3 is going to be the slowness. Will be curious if it will recover faster from cold boot once the calibration file is written.

Although, from reading the operational description, if there is a significant environmental change like sudden dirty air or a quick change in humidity/temperature, it could trigger a recalibration event, and the calibration file won't help with that.

Will probably punt on ULP if it takes longer than 6-8 hours to calibrate, as that's data accuracy being lost whenever a sensor thinks it isn't reading right.

[cwichura]

Based on your comments that they don't recommend switching modes on a given device, I agree with figuring out which one is "best" compromise across accuracy and power and just go with that, and skip trying to do any "smartness".

You are already going to be better than what is in current firmware!

[NickDunklee]

I pushed an update with LP, compiled against a few targets to validate it wasn't broken. If you or anyone else wants to play with the LP flavor, this has it ready to go. I'll do final revision once I see results from ULP with whatever ends up making sense.