btmon3.py waiting for blocking reads?

[shane128]

Hello,

I have just recently installed the btmon3.py in "server" mode on a local server with packet forwarding set up from my dashbox so that btmon3.py can provide a more frequent MQTT update than what the built-in MQTT module on the Dashbox provides. While the Dashbox is set to transmit real-time data to the IP address of the server hosting the btmon3.py script (every 5 seconds - and I can confirm that it's arriving every 5 seconds with tcpdump on that server), the btmon3.py seems to be stuck in blocking read mode twice: once the incoming data is received and again before processing the incoming data.

With two terminal windows open, one watching tcpdump to see the incoming packets every 5 seconds and the other watching the debug output from btmon3.py, the btmon3.py output only jumps in tandem with the incoming data. This suggests that it's getting stuck in a blocking read before continuing on to process incoming data.

It will start with "SOCKET: waiting for connection" and sits there until a packet comes in (this is fine). Once the packet arrives it will display "waiting for data from device" and "reading 1 of 1 packets" together and then just sits there stuck until the next packet arrives 5 seconds later. Once that arrives, it will send the data to the MQTTprocessor and the sit and wait with "SOCKET: waiting for connection" again. The next cycle has to wait 5 seconds for another packet to arrive, then 5 seconds for the packet to be read, then another 5 seconds to send it to processing to then start the cycle again. The effect is min 15 seconds between MQTT updates.

I would expect that once the data is received, the script should process it without delay and then sit and wait for data again. I am consistently missing two out of every three update packets sent by the dashbox. For my need, I was hoping to catch the updates every 5 seconds and send them to the MQTT server.

Would more familiar eyes on the btmon3.py script be able to pinpoint the bottleneck?

Thank you.

Shane.

[ben]

Hello,

I have just recently installed the btmon3.py in "server" mode on a local server with packet forwarding set up from my dashbox so that btmon3.py can provide a more frequent MQTT update than what the built-in MQTT module on the Dashbox provides. While the Dashbox is set to transmit real-time data to the IP address of the server hosting the btmon3.py script (every 5 seconds - and I can confirm that it's arriving every 5 seconds with tcpdump on that server), the btmon3.py seems to be stuck in blocking read mode twice: once the incoming data is received and again before processing the incoming data.

With two terminal windows open, one watching tcpdump to see the incoming packets every 5 seconds and the other watching the debug output from btmon3.py, the btmon3.py output only jumps in tandem with the incoming data. This suggests that it's getting stuck in a blocking read before continuing on to process incoming data.

It will start with "SOCKET: waiting for connection" and sits there until a packet comes in (this is fine). Once the packet arrives it will display "waiting for data from device" and "reading 1 of 1 packets" together and then just sits there stuck until the next packet arrives 5 seconds later. Once that arrives, it will send the data to the MQTTprocessor and the sit and wait with "SOCKET: waiting for connection" again. The next cycle has to wait 5 seconds for another packet to arrive, then 5 seconds for the packet to be read, then another 5 seconds to send it to processing to then start the cycle again. The effect is min 15 seconds between MQTT updates.

I would expect that once the data is received, the script should process it without delay and then sit and wait for data again. I am consistently missing two out of every three update packets sent by the dashbox. For my need, I was hoping to catch the updates every 5 seconds and send them to the MQTT server.

Would more familiar eyes on the btmon3.py script be able to pinpoint the bottleneck?

Thank you.

Shane.

Can you post your config script (redact anything you don't want public)?

[shane128]

Thank you for the quick response!

Config file is posted here. Beyond what is listed below, the remaining items are set to "false" and their options are commented out (I can attach the rest if it's pertient).

Code: Select all

# sample configuration file for btmon

[source]
device_type = gem
#device_list = fc, fd, fe
#packet_format = ecm1240bin
#db_schema = counters

#serial_read = true
#serial_port = COM1
#serial_poll_interval = 10     # use 0 when monitor is in real-time mode

[mqtt]
mqtt_out=true
mqtt_host=10.x.x.x
mqtt_port=1883
mqtt_clientid=btmon
mqtt_base_topic=house/energy_rt
mqtt_user=xxxxx
mqtt_passwd=xxxxx
#mqtt_map=399999_ch1_w,solar,399999_ch1_wh,solar_wh,399999_t1,garage
mqtt_upload_period=0

ip_read = true
#ip_host = 10.x.x.x
#ip_host = 127.0.0.1
#ip_port = 8001
ip_port = 8083
#ip_mode = client
ip_mode = server
ip_poll_interval = 0
# use 0 when monitor is in real-time mode
reverse_polarity=1

Thank you again.

Shane.

[ben]

I should've asked this yesterday along with that, are you using persistent or non-persistent mode on the DashBox?

[shane128]

I have "Persistent Connection" set to "ON" in the Packet Forwarder on the Dashbox.

Thank you.

[ben]

I'll get this running locally on Monday and take a look. I cleaned up the script today a bit (removed the dead services) but didn't have enough time to get it running.

[ben]

Turn off Persistent Mode on the DashBox (update to the latest if non-persistent doesn't work).

Btmon kills the server after every read. The DashBox tries to send a packet, fails, then reconnects.

[shane128]

That worked beautifully. Thank you. My apologies for not trying that - I had figured that a persistent connection would make more sense.

Thanks again.

Shane.

[ben]

That worked beautifully. Thank you. My apologies for not trying that - I had figured that a persistent connection would make more sense.

Thanks again.

Shane.

Yeah I thought the same.