BrewPi Remix – What’s Old is New Again

[duncan.brown]

Got it, I can reproduce it. It happens when a tilt has been sending data but then goes offline. With the lines above commented out, it doesn’t crash the script, but you get NAN for the temperatures from the tilt. If the line that calls an int() on HWVer isn’t commented out, then it will call int(null) which triggers the error. So @Thorrak is off the hook, as I think TiltBridge is going the right thing if a tilt goes offline. Let me see if I can find a fix.

 

[duncan.brown]

OK, here's a temporary fix that should work. The code is getting unhappy when null gets into the previous JSON. Need to figure out what the underlying issue is, but this will stop it from choking:

diff --git a/brewpi.py b/brewpi.py
index c2ed372..82d1246 100755
--- a/brewpi.py
+++ b/brewpi.py
@@ -1342,9 +1342,10 @@ def loop():  # Main program loop
                                                     prevTempJson[config['tiltColor'] + 'Temp'] = round(_temp)
 
                                                 # Get battery value from anything >= Tilt v2
-                                                if int(prevTempJson[config['tiltColor'] + 'HWVer']) >= 2:
-                                                    if (checkKey(api['tilts'][config['tiltColor']], 'weeks_on_battery')):
-                                                        prevTempJson[config["tiltColor"] + 'Batt'] = int(api['tilts'][config['tiltColor']]['weeks_on_battery'])
+                                                if prevTempJson[config['tiltColor'] + 'HWVer']:
+                                                    if int(prevTempJson[config['tiltColor'] + 'HWVer']) >= 2:
+                                                        if (checkKey(api['tilts'][config['tiltColor']], 'weeks_on_battery')):
+                                                            prevTempJson[config["tiltColor"] + 'Batt'] = int(api['tilts'][config['tiltColor']]['weeks_on_battery'])
 
                                                 # Set time of last update
                                                 lastTiltbridge = timestamp = time.time()

 

[LBussy]

That’s awesome. As I said we can probably use the last good value there (probably up in Tilt.py) since the HW version is unlikely to change.

The underlying issue is probably the dual-use of that part of the beacon. I’d have to find the emails I exchanged with the Tilt dev but I’m pretty sure that’s the case.

If you want to add a PR I’ll get it merged into dev and then we can push to main if we can get a tester.

Thank you!

 

[duncan.brown]

That stops the script from crashing, but you still get NAN in the temperatures. I’ll try and fix that tonight and then put a pull request in for both issues.

This also might be the cause of something else that has been bothering me. Sometimes the web page has a hard time displaying the graph once a beer has been fermenting for a while. It take ages to render the plot. If the data set has NANs in it, that that might explain it. Will investigate that further as well.

 

[LBussy]

It take ages to render the plot. If the data set has NANs in it, that that might explain it.

That’s an interesting take on it. Easy enough to test I guess. I always assumed it was just JS being a PIA.

 

[balrog]

I always assumed it was taking longer to parse two weeks of 10s interval (~120,000 points) because it had to find min/max values.

 

[LBussy]

I always assumed it was taking longer to parse two weeks of 10s interval (~120,000 points) because it had to find min/max values.

If you set 10s intervals, you get what you deserve. If you think you need them, you have different issues.

 

[athistle]

That’s awesome. As I said we can probably use the last good value there (probably up in Tilt.py) since the HW version is unlikely to change.

The underlying issue is probably the dual-use of that part of the beacon. I’d have to find the emails I exchanged with the Tilt dev but I’m pretty sure that’s the case.

If you want to add a PR I’ll get it merged into dev and then we can push to main if we can get a tester.

Thank you!

I am happy to test if needed.

 

[LBussy]

As soon as I have a test branch for you to test, I'll be sure to give you instructions on how to change.

 

[athistle]

As soon as I have a test branch for you to test, I'll be sure to give you instructions on how to change.

 

[duncan.brown]

OK, I’ve had this running for a few days now and I haven’t seen any issues. I took the tilt bridge upstairs to simulate the loss of a tilt and all seems to be working fine. The tiltbridge times out after 300s and the values disappear rather than becoming NaN.

fixed bug when tiltbridge tilt disappears by duncan-brown · Pull Request #197 · brewpi-remix/brewpi-script-rmx

When a tilt monitored by tiltbridge goes offline, the HWVer gets set to None which causes int() to fail. Fix is to check that HWVer is not None before casting it to an integer.

github.com

 

[LBussy]

TY, sir! I’ll check that in tomorrow morning and let @athistle give it a go.

 

[LBussy]

Okay, @athistle here is what you can do to test the fix. Please note anyone else reading this please do NOT execute these steps. These may not even work by the time you read this; it is only to allow one person to test a change.

Do not do this with an active brew running; execute the following in order:

cd /home/brewpi/
sudo git restore .
sudo git branch -r | grep -v '\->' | sed "s,\x1B\[[0-9;]*[a-zA-Z],,g" | while read remote; do sudo git branch --track "${remote#origin/}" "$remote" 2>/dev/null; done
sudo git fetch --all
sudo git pull --all
sudo git checkout tiltfix
sudo systemctl restart brewpi

Also note that I am unable to test the above process right now, so you may get an error. If you do, please respond back and tell me which command and what error, and we'll get you hooked up.

 

[duncan.brown]

@athistle if the git restore fails for you, your pi may have an older version of git (mine has 2.20.1 and restore was introduced in 2.23.0). The commands below do the same thing:

cd /home/brewpi
sudo git reset --hard HEAD
sudo git fetch -a
sudo git checkout -b tiltfix origin/tiltfix
sudo systemctl restart brewpi

 

[balrog]

Okay, @athistle here is what you can do to test the fix. Please note anyone else reading this please do NOT execute these steps.

Okay, I tried this out and...oh...right...sorry.

JK

 

[Quigongentoo]

That stops the script from crashing, but you still get NAN in the temperatures. I’ll try and fix that tonight and then put a pull request in for both issues.

This also might be the cause of something else that has been bothering me. Sometimes the web page has a hard time displaying the graph once a beer has been fermenting for a while. It take ages to render the plot. If the data set has NANs in it, that that might explain it. Will investigate that further as well.

I fixed this by turning the sample time for the data to like 15+ minutes. It seemed to do ok

 

[athistle]

Okay, @athistle here is what you can do to test the fix. Please note anyone else reading this please do NOT execute these steps. These may not even work by the time you read this; it is only to allow one person to test a change.

Do not do this with an active brew running; execute the following in order:

cd /home/brewpi/
sudo git restore .
sudo git branch -r | grep -v '\->' | sed "s,\x1B\[[0-9;]*[a-zA-Z],,g" | while read remote; do sudo git branch --track "${remote#origin/}" "$remote" 2>/dev/null; done
sudo git fetch --all
sudo git pull --all
sudo git checkout tiltfix
sudo systemctl restart brewpi

Also note that I am unable to test the above process right now, so you may get an error. If you do, please respond back and tell me which command and what error, and we'll get you hooked up.

Ok,
I have executed all the above and am now on the tiltfix branch.
Still have the N/AN/AN/AN/A displayed for sg and temp values

 

[duncan.brown]

Ah, I thought there might be one more edge case... if you restart brewpi with bad tilt data in the JSON from prior to the bug, it will do this. Let me see if I can come up with a fix for that.

 

[duncan.brown]

@athistle in the mean time, please can you look at the stdout and stderr logs and send a screen shot? It shouldn't be crashing now, at least.

 

[athistle]

@athistle in the mean time, please can you look at the stdout and stderr logs and send a screen shot? It shouldn't be crashing now, at least.

 

[LBussy]

That seems to indicate TiltBridge is not sending data. Have you restarted it?

 

[athistle]

That seems to indicate TiltBridge is not sending data. Have you restarted it?

I have restarted both BPR and TiltBridge, Now it displays 12C for both temp and SG

 

[LBussy]

Uh …. That’s interesting. Can you share an excerpt of stdout.txt?

 

[athistle]

Uh …. That’s interesting. Can you share an excerpt of stdout.txt?

 

[LBussy]

Well, that's curious as hell. @duncan.brown I didn't see any opportunity for that to happen in your changes. Any ideas?

 

[athistle]

Well, that's curious as hell. @duncan.brown I didn't see any opportunity for that to happen in your changes. Any ideas?

Just our of curiosity I rebooted the pi, now its back to "N/AN/AN/AN/A" for both SG and temp.

 

[duncan.brown]

I’ve gone back to the old version of the code and moved my tiltbridge upstairs to try and reproduce the error. So far, it’s cycling between displaying the previous good values and nothing each time it gets an empty update from TiltBridge:

2022-05-09 16:01:59 [N] API TB JSON Recvd: {"mdns_id": "tiltbridge", "tilts": null}
2022-05-09 16:02:29 [N] API TB JSON Recvd: {"mdns_id": "tiltbridge", "tilts": null}
2022-05-09 16:02:40 [N] Turned off Tiltbridge.
2022-05-09 16:02:40 [N] Update: {"BeerTemp": 71.26, "FridgeTemp": 71.26, "BeerAnn": null, "FridgeAnn": null, "RoomTemp": null, "State": 0, "BeerSet": 72.0, "FridgeSet": 72.0, "GreenHWVer": 5, "GreenSWVer": 0, "GreenSG": 1.011, "GreenTemp": 71, "GreenBatt": 97}
2022-05-09 16:02:59 [N] Turned on Tiltbridge.
2022-05-09 16:02:59 [N] Stopping Tilt.
2022-05-09 16:02:59 [N] API TB JSON Recvd: {"mdns_id": "tiltbridge", "tilts": null}

I think I need to wait for the data to expire, as that’s when the bug is triggered.

 

[duncan.brown]

Ah, ha, figured out the bug! To trigger it, you need to:

1. Have TiltBridge sending data correctly for you tilt color for the brew.
2. Have TiltBridge loose the tilt connection and send a JSON containing

   {"mdns_id": "tiltbridge", "tilts": null}

3. Restart the BrewPi script while the contents of the internal JSON for that tilt contain nulls

   Update: {"BeerTemp": 71.71, "FridgeTemp": 71.71, "BeerAnn": null, "FridgeAnn": null, "RoomTemp": null, "State": 0, "BeerSet": 72.0, "FridgeSet": 72.0, "GreenHWVer": null, "GreenSWVer": null, "GreenSG": null, "GreenTemp": null, "GreenBatt": null}

Then you get the problem, which is originally caused by the error:

2022-05-09 18:47:49 [E] Failed to parse Green Tilt from Tiltbridge payload.
2022-05-09 18:47:50 [E] Caught an unexpected exception.
2022-05-09 18:47:50 [E] Error info:
2022-05-09 18:47:50 [E]     Type: <class 'UnboundLocalError'>
2022-05-09 18:47:50 [E]     Filename: brewpi.py
2022-05-09 18:47:50 [E]     LineNo: 1600
2022-05-09 18:47:50 [E]     Error: local variable 'color' referenced before assignment
2022-05-09 18:47:53 [E] Starting BrewPi.
2022-05-09 18:48:03 [E] Failed to retrieve Green Tilt value, restarting Tilt.
2022-05-09 18:48:18 [E] Failed to parse Green Tilt from Tiltbridge payload.

The failure on int()-ing a None is a bit of a red herring. This seems to be the real issue. Working on a fix.

 

[duncan.brown]

OK, caught it in action...

2022-05-09 18:55:42 [N] 0.7.6 (master) [b144360]
2022-05-09 18:55:42 [N] Starting '044-ill-have-an-otter.'
2022-05-09 18:55:42 [N] Opening serial port.
2022-05-09 18:55:42 [N] Waiting 10 seconds for board to restart.
2022-05-09 18:55:52 [N] Checking software version on controller.
2022-05-09 18:55:52 [N] Found BrewPi v0.2.12 build 3a7d428, running on an Arduino Uno with an I2C shield on port /dev/chronical.
2022-05-09 18:55:52 [N] Turned on Tiltbridge.
2022-05-09 18:55:52 [N] Stopping Tilt.
2022-05-09 18:55:52 [N] API TB JSON Recvd: {"mdns_id": "tiltbridge", "tilts": null}
2022-05-09 18:55:52 [N] Turned off Tiltbridge.
2022-05-09 18:55:52 [N] prevTempJson = {'BeerTemp': 71.6, 'FridgeTemp': 71.6, 'BeerAnn': None, 'FridgeAnn': None, 'RoomTemp': None, 'State': 0, 'BeerSet': 72.0, 'FridgeSet': 72.0, 'GreenHWVer': 0, 'GreenSWVer': 0, 'GreenSG': 0, 'GreenTemp': 0, 'GreenBatt': 0}
2022-05-09 18:55:52 [N] Caught an unexpected exception, exiting.
2022-05-09 18:55:52 [N] Stopping background serial processing.
2022-05-09 18:55:52 [N] Closing port.
2022-05-09 18:55:52 [N] Closing open sockets.
2022-05-09 18:55:52 [N] Exiting.

2022-05-09 18:55:42 [E] Starting BrewPi.
2022-05-09 18:55:52 [E] Failed to parse Green Tilt from Tiltbridge payload.
2022-05-09 18:55:52 [E] Caught an unexpected exception.
2022-05-09 18:55:52 [E] Error info:
2022-05-09 18:55:52 [E]         Type: <class 'UnboundLocalError'>
2022-05-09 18:55:52 [E]         Filename: brewpi.py
2022-05-09 18:55:52 [E]         LineNo: 1601
2022-05-09 18:55:52 [E]         Error: local variable 'color' referenced before assignment

FOTD pick time... will debug in a bit.

 

[duncan.brown]

@LBussy I know what the real bug is. There is a loop that start on line 1892 for color in Tilt.TILT_COLORS that tries to parse through and get the information from the tilt that matches config['tiltColor']. When this loop is done, color contains Pink, since that's the last color in TILT_COLORS.

This means that when the code gets to line 1893 to expire the old TiltBridge values, color contains Pink rather than the color of the tilt configured for the brew. The simplest fix seems to be to add

color = config['tiltColor']

after tiltbridge is set to False to make sure that the tilt configured for this brew is expired.

Or buy a pink tilt...

 

[LBussy]

Doh!

You da man!

 

[DuncB]

I follow this thread and admire your work, I'm not really sure what's going on in detail, but I see progress.

 

[duncan.brown]

OK, pull request is in. Just tested it with the procedure I used to trigger the bug the first time and it looks good.

@LBussy you needs both this patch and the original one to fix the int() bug. I just realized that I based this pull request on your main branch, not the tiltfix branch. GitHub should be able to merge this on top with no conflict, but if not, let me know.

 

[duncan.brown]

Hold off merging... there is another piece to this...

 

[duncan.brown]

OK, I couldn't let this go... I blame this for giving me nightmares when I was a kid about not fixing things properly...



The last bug was in the parsing of HWVer. The code was doing nested if loops to check the hardware version. The way it was implemented was:

if ( key1 exists ):
    if ( key1 is true ):
        hw version equals value for key1
elif ( key2 exists ):
    if ( key2 is true ):
        hw version equals value for key2
else:
    hw version equals zero

The problem is that if key1 exists, the code never looks for the existence of key2 (regardless of whether key1 is true or false) so it can never set that hardware version. If key1 exists, but is false, then the code defaults to the existing value. This is either None, causing the code to crash on int(None), or zero which screws up the parsing the data for a newer tilt resulting in the NAN values. The way to do this is

if ( key1 exists and key1 is true ):
    hw version equals value for key1
elif ( key2 exists and key2 is true ):
    hw version equals value for key2
else:
    hw version equals zero

I added this fix to Expire the correct tilt color by duncan-brown · Pull Request #198 · brewpi-remix/brewpi-script-rmx and rebased against the tilt fix branch, so it should be good now.

I think that's it, so I'm going to call it a night and celebrate by putting a rug on a polished floor...

 

[duncan.brown]

Also @Thorrak is definitely off the hook

 

[Thorrak]

Also @Thorrak is definitely off the hook

Thank god.

 

[LBussy]

Okay @athistle:

cd /home/brewpi/
sudo git pull
sudo systemctl restart brewpi

Don't break it this time.

 

[athistle]

Okay @athistle:

cd /home/brewpi/
sudo git pull
sudo systemctl restart brewpi

Don't break it this time.

Wooo Hoooo! It works like a champ, however while it now displays TiltBridge info in the banner it is not logging or plotting tilt SG in the chart.

 

[LBussy]

There's still something wrong - I thought I had the Tilt color in the header. I mean good it's not crashing, and yes it should be plotting the SG.

I wish I could explain how much this sucks living in an RV - at least when it comes to things like this.