[BUG] [Errno -3] Temporary failure in name resolution when using Button entity with own MQTT client

[simonthechipmunk]

Describe the bug
After I ran into #194 I tried using the own MQTT client approach and handle connecting the MQTT broker myself but this doesn't seem to work with the button entity.
The script stops with a bogus DNS error message:

socket.gaierror: [Errno -3] Temporary failure in name resolution

The network is fine though and Home Assistant is reachable via the hostname (I tried using the IP address directly too). If I use a sensor entity instead of the button the code runs as expected.
It looks like the button entity is trying to connect the already connected client again, but I'm at a loss on why this is happening.

To Reproduce
Here's a minimal example to reproduce:

import logging
from paho.mqtt import client as MQTTClient
from ha_mqtt_discoverable import Settings, DeviceInfo
from ha_mqtt_discoverable.sensors import Button, ButtonInfo

# Entity instances
button1: Button

def connect_mqtt() -> MQTTClient.Client:

    # For paho-mqtt 2.0.0, you need to add the properties parameter.
    def on_connect(client, userdata, flags, rc, properties):
        if rc == 0:
            logging.info("Connected to MQTT Broker!")
        else:
            logging.error("Failed to connect, return code %d\n", rc)

    # For paho-mqtt 2.0.0, you need to set callback_api_version.
    client = MQTTClient.Client(client_id="ButtonTestClient", callback_api_version=MQTTClient.CallbackAPIVersion.VERSION2)

    client.username_pw_set("mqtt", "hamqtt")
    client.on_connect = on_connect
    client.connect(host="homeassistant.local", port=1883)
    return client

# Main
if __name__ == "__main__":

    # Connect to MQTT broker
    client = connect_mqtt()
    client.loop_start()
   
    # Define the device. At least one of `identifiers` or `connections` must be supplied
    device_info = DeviceInfo(name="ButtonTestDevice", identifiers="buttontestdevice1")

    # Configure the required parameters for the MQTT broker
    mqtt_settings = Settings.MQTT(client=client)

    # Information about the entities
    button_info = ButtonInfo(name="Button", unique_id="button12345", device=device_info)

    # Create entity settings
    button_settings = Settings(mqtt=mqtt_settings, entity=button_info)

    # Instantiate the entities
    button1 = Button(button_settings, None, None)

    # Publish the button's discoverability message to let HA automatically notice it
    button1.write_config()

    while(True):
        pass

And the full output:

~/Code/boobox/software/BooBox_Audio/src$ python Button.py 
Traceback (most recent call last):
  File "/home/simon/Code/boobox/software/BooBox_Audio/src/Button.py", line 52, in <module>
    button1 = Button(button_settings, None, None)
  File "/home/simon/Code/boobox/.venv/lib/python3.10/site-packages/ha_mqtt_discoverable/__init__.py", line 889, in __init__
    self._connect_client()
  File "/home/simon/Code/boobox/.venv/lib/python3.10/site-packages/ha_mqtt_discoverable/__init__.py", line 719, in _connect_client
    result = self.mqtt_client.connect(self._settings.mqtt.host, self._settings.mqtt.port or 1883)
  File "/home/simon/Code/boobox/.venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 1435, in connect
    return self.reconnect()
  File "/home/simon/Code/boobox/.venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 1598, in reconnect
    self._sock = self._create_socket()
  File "/home/simon/Code/boobox/.venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 4609, in _create_socket
    sock = self._create_socket_connection()
  File "/home/simon/Code/boobox/.venv/lib/python3.10/site-packages/paho/mqtt/client.py", line 4640, in _create_socket_connection
    return socket.create_connection(addr, timeout=self._connect_timeout, source_address=source)
  File "/usr/lib/python3.10/socket.py", line 824, in create_connection
    for res in getaddrinfo(host, port, 0, SOCK_STREAM):
  File "/usr/lib/python3.10/socket.py", line 955, in getaddrinfo
    for res in _socket.getaddrinfo(host, port, family, type, proto, flags):
socket.gaierror: [Errno -3] Temporary failure in name resolution

Expected behavior
The button entity should be advertised to Home Assistant via the existing MQTT client connection

Environment:

 - Ubuntu 20.04
 - Python 3.10.12
 - Packages
      ha-mqtt-discoverable  0.18.0
      paho-mqtt              2.1.0

[simonthechipmunk]

I'm suspecting this line is the culprit, since connect_client() is always called without checking whether the client already exists.

ha-mqtt-discoverable/ha_mqtt_discoverable/__init__.py

Line 889 in 8feb496

self._connect_client()

Which would mean currently any entity of class subscriber would be affected by this if I understand the code correctly.
In the parent discoverable class there's a guard around the connect_client() call which explains why the sensor entity is working fine:

ha-mqtt-discoverable/ha_mqtt_discoverable/__init__.py

Lines 648 to 651 in 8feb496

	# If there is a callback function defined, the user must manually connect
	# to the MQTT client
	if not (on_connect or self._settings.mqtt.client is not None):
	self._connect_client()

[kratz00]

@simonthechipmunk Thanks for your issue report and your initial analysis. I agree and I even see more issues, e.g. the instances of Subscriber rely on the on_connect callback, which is never set and therefor subscribing to the command topic will never happen, when using an external MQTT client. In regards to #194 and this issues all entities have to use the same client and subscribing to a entities' command topic happens successfully. Please give me some time to come up with a fix.

[simonthechipmunk]

@kratz00 Thanks for your help. It is much appreciated.
I feel like solving this issue might also resolve parts of the problems described in #194. When I initially tried using a button and sensor entity with the internal MQTT client, they cyclically disconnected and reconnected to the broker. So I got about 5s of sensor data while the button was unresponsive, then the sensor data stopped coming in but the button started working for a few seconds until the loop repeated. That would also match your assessment that entities can't use individual clients. With the current design they seem to disconnect each other's connection.

[kratz00]

#332 - First draft of a possible solution (just a first try, tests are missing etc.).

With these changes I can create multiple entities (Discoverable and/or Subscriber based) using an existing MQTT client.

• connecting the MQTT client before or after entity creation should be supported
• if the MQTT client is not connected already, an outside defined on_connect callback is overwritten and therefor never called

I could not reproduce #192 - all entities create their own MQTT clients and on a first glance everything looks good. Might be a scheduling issue as each MQTT client starts one communication loop thread, which might overwhelm low end hardware.
On which hardware platform did you experience this problem?

[simonthechipmunk]

I could not reproduce #192 - all entities create their own MQTT clients and on a first glance everything looks good. Might be a scheduling issue as each MQTT client starts one communication loop thread, which might overwhelm low end hardware. On which hardware platform did you experience this problem?

I'll start with this one as I'm not sure what's going on myself. The target hardware is a Raspberry pi 3b+ but I experience this behavior on my regular PC too. Here's a basic example again that reproduces the problem. Maybe I'm also just doing something completely wrong since I'm just starting out with threading and MQTT in python:

import threading
import time
from paho.mqtt import client as MQTTClient
from ha_mqtt_discoverable import Settings, DeviceInfo
from ha_mqtt_discoverable.sensors import BinarySensor, BinarySensorInfo, Button, ButtonInfo

# Entity instances
sensor: BinarySensor
button: Button

# Define an optional object to be passed back to the callback
user_data = "Button was pressed!"

# To receive button commands from HA, define a callback function:
def my_callback(client: MQTTClient.Client, user_data, message: MQTTClient.MQTTMessage):
    print(user_data)
    time.sleep(6)


# Sensor test task
def sensortest():
    while True:
        global sensor
        sensor.update_state(True)
        time.sleep(1)
        sensor.update_state(False)
        time.sleep(1)

# Main
if __name__ == "__main__":
  
    # Define the device. At least one of `identifiers` or `connections` must be supplied
    device_info = DeviceInfo(name="ButtonSensorTestDevice", identifiers="buttonsensortestdevice1")

    # Configure the required parameters for the MQTT broker
    mqtt_settings = Settings.MQTT(host="homeassistant.local", client_name="ButtonSensorTestClient", username="mqtt", password="hamqtt")

    # Information about the entities
    button_info = ButtonInfo(name="Button", unique_id="button12346", device=device_info)
    sensor_info = BinarySensorInfo(name="Sensor", unique_id="sensor12347", device_class="motion", device=device_info)
    # Create entity settings
    sensor_settings = Settings(mqtt=mqtt_settings, entity=sensor_info)
    button_settings = Settings(mqtt=mqtt_settings, entity=button_info)

    # Instantiate the entities
    sensor = BinarySensor(sensor_settings)
    button = Button(button_settings, my_callback, user_data)

    # Publish the button's discoverability message to let HA automatically notice it
    button.write_config()

    sensorThread = threading.Thread(target=sensortest)
    sensorThread.start()

    sensorThread.join()

What I see in HA and MQTT explorer is, that the sensor is only sending every 2s, so the state remains either on or off continuously.
The button isn't working correctly too, so often button presses in HA are not recognized and no console output is generated from the python script as if the MQTT messages are not picked up by the client.

Interestingly enough if you keep pressing the button eventually a message will come through and as long as the callback is inside the sleep routine, the sensor will start sending its correct values.
It looks like the two internal clients are somehow influencing each other. Here's a screenshot from MQTT explorer:

[simonthechipmunk]

#332 - First draft of a possible solution (just a first try, tests are missing etc.).

With these changes I can create multiple entities (Discoverable and/or Subscriber based) using an existing MQTT client.

* connecting the MQTT client before or after entity creation should be supported

* if the MQTT client is not connected already, an outside defined _on_connect_ callback is overwritten and therefor never called

I tested your changes and it worked flawlessly with the external MQTT client and button + sensor. Thank you.
Sensor data is reported continuously and the button presses are also recognized without message losses.

The weird behavior with the internal clients described above still persists, but maybe I'm making a rookie mistake on my end?

[kratz00]

@simonthechipmunk Thanks for testing the feature branch, glad it's solving the issue - I will clean up the PR and merge it then.

Regarding the other problem, I guess the problems are in regards to all the sleep calls, which basically pause the execution of the thread. You call sleep twice, each with one second - which corresponds to what you see - sensors get updated every 2 seconds only. Can you try reducing the duration e.g. using 0.1 and completely removing the sleep call in the my_callback?

[simonthechipmunk]

The sleep call in my_callback was just to show the effect that the callback is having on the sensor thread so it can be removed of course. Between the two sleep calls is another sensor update so the expected behavior is a status change of the sensor every second. However something seems to be blocking it.
The same code is working fine using the external MQTT client btw.

If I remove the sleep call in my_callback completely and replace the other two with a 0.1s sleep, it's getting somewhere: The sensor is rapidly updating as expected, but there's a disruption every other second with a complete stop of updates for exactly one second.

It looks like the connection is briefly dropped and reestablished. I would need to make a Wireshark recording to confirm this, but there is clearly something going on with the callback. Is it maybe blocking the MQTT event loop?

This is an entirely different issue though, so this issue regarding the external MQTT client can be closed.

[Ginden]

I did some very hacky stuff to use externally managed MQTT client for devices with multiple buttons: https://github.com/Ginden/home-assistant-remote-commands

[hue8i]

Hallo I tested it too. But it is not working if you had more than one button or number.They all uses the same callback funktion from the last created Button and have the same user data.

Maybe you can fix that somehow.

One question left: When I use not an existing MQTT Client, do I start with every new created Button a new mqtt thread? So when I had 10 Buttons I‘m Running 10 threads?

Thanks for answering.

[kratz00]

Hi @hue8i

It's still on the list, I will look into it.

Regarding your question, yes, you will end up will as many threads as your created entities.
So better use one MQTT client for all your entities - see #332

[Ginden]

@hue8i @kratz00 Even with #322, the problem persists - creation of Subscriber instances always overwrites on_connect and on_message callbacks with newest version.

message_callback_add must be used instead of on_message.

There is no such method for on_connect, though.