AsyncSSH Hanging on Cisco UCS #730
Comments
|
I think the biggest issue here is that you are using If you pass in an integer argument, it will read up to that many bytes, but it will also return early if some bytes are received but the number is less than the value you pass in, without needing either the client or the server to send an EOF. That said, it may take multiple reads to get all the data you want, but that's where you can do things like use Something else to think about is that some devices may not handle writing input to them while they're still processing a previous command. So, ideally, you'd find some way to send a single command and then read the output in its entirety from that before you write the next command, if you want to send multiple commands in a single SSH session. You also shouldn't send EOF until you're completely done, or alternately finish with an "exit" command (or equivalent for your device) and let it be the one to end the session. |
|
Hi Ron, Thanks for your detailed response. I managed to get it working using
I do not handle this issue perfectly here but it does get me the output I need. I will keep it in mind if I do run it that issue in the future. It also works without having to input exit commands. If you have any more improvements, please tell me. Otherwise, you can close this issue. Thank You! commands = ["show service-profile status", "connect nxos b", "show mac address", "show interface description"]
async def run_client_multiple_commands(
self, host, username, password, commands, command_name_key=[]
):
if not commands:
return None
if isinstance(commands, str):
commands = [commands]
logger.debug(f"Running commands on host {host}: {commands}")
try:
asyncssh.set_log_level('DEBUG')
asyncssh.set_sftp_log_level('DEBUG')
asyncssh.set_debug_level(2)
async with await asyncio.wait_for(
asyncssh.connect(
host,
username=username,
password=password,
port=22,
known_hosts=None,
),
timeout=10,
) as conn:
async with await asyncio.wait_for(
conn.create_process(stderr=asyncssh.STDOUT), timeout=10
) as process:
res = {}
for i, command in enumerate(commands):
logger.debug(f'--- Running "{command}" ---')
process.stdin.write(f"{command}\n")
process.stdin.write("show show\n") # Delimiter between commands
for command in commands:
if command not in ["connect nxos a", "connect nxos b"]:
if command_name_key:
if isinstance(command_name_key, str):
command_name_key = [command_name_key]
res[command_name_key[i]] = await process.stdout.readuntil("show show")
else:
res[command] = await process.stdout.readuntil("show show")
process.stdin.write_eof()
return res
except asyncio.TimeoutError as e:
logger.warning(f"Timeout while connecting to {host}: {e}") |
After some more tinkering, this is not true. The main ucs terminal doesn't not echo back the delimiter on a syntax error, but the nxos terminal does. This leads to the delimiter/readuntil mismatch. So I have to figure out a different solution for that but at least I understand what it doing better now. I can add a regex to |
Yes, that's right. If you log in to one of the devices with OpenSSH and do a "connect", does it echo your input in that case? If so, it seems kind of odd that it wouldn't echo it for an AsyncSSH connection, but it could have something to do with whether you try to set up a pseudo-TTY or not. If you add something like I'm also wondering if maybe the "show" command is getting dropped because you are sending it before the previous command has been fully processed, especially in the case of something like the "connect". You could try adding a delay before sending that just to see what happens, though that's not a great final solution. |
|
I recommend to use shell integration (vscode example) to detect several kind of command events (e.g. command start/end) with |
Yes, it does echo back the input using OpenSSH. Sorry if I misspoke earlier but it does not show up in the AsyncSSH stdout unless term_type='ansi' is set which is my problem.
With "-T" it hangs after I enter the password when trying to connect.
This does work in echoing back the input. I have to add a delay for the commands to be processed correctly. Otherwise, I get this: {
'show service-profile status': 'show service-profile status\r\nshow show',
'connect nxos b': '\r\nconnect nxos b\r\nshow show',
'show mac address': '\r\nshow mac address\r\nshow show',
'show interface description': '\r\nshow interface description\r\nshow show'
}I think this is a okay solution, I should be able to use terminal length 0 and it should work with a small time delay. It is supposed to be used in an internal tool so there is not a lot of user input and a bit of time delay does not matter. Without ansi and a delay, the output I get from SSH is without any input echoes: Output from show service-profile status
% Invalid Command at '^' marker\n\n
Output from show fabric-interconnect b
Output from connect nxos b
% Invalid command\nSyntax error while parsing 'show show"
Output from show mac address
Syntax error while parsing 'show show"
Output from show interface description
Syntax error while parsing 'show show"What is weird is how The other problem is if I do a regex with like It would be nice to get the stdin to output here as well without using ansi. I will try to mess around with it some more. @FindDefinition |
|
@FindDefinition, how would this apply in the case of talking to something like a network switch or router with a CLI which isn't a typical Linux shell (and which might not be Linux-based at all)? |
Yeah, I was going to ask you about the "More" prompt issue, and the potential need to send "terminal length 0" or equivalent.
This looks pretty much like I'd expect.
Is there any command which just takes a string and echoes that string back (independent of the input echoing)? Alternately, if you do have input echoing, is there any "comment" character you can put at the beginning of a command you send, so you get only the input echo but no other output? This may still not be enough without a delay depending on when the input is echoed, but on some devices this might be enough to get the input to show up only after the output of the previous command, and by using a comment string you don't have to worry about Invalid Command errors.
Unfortunately, this behavior is coming from the target device. There isn't really anything you can do in AsyncSSH (or any other SSH client) to change this behavior. Basically, the pseudo-TTY you request when setting the terminal type is what is responsible on the SSH server for doing the input echoing. |
I do not think there is one, unfortunately. The closest thing would be sending a newline but that does not output any newlines in the output. I wrote this workaround so I have to deal with messing around with timers for now. I am surprised that it works, at least for my purposes of dumping the device data based on the command. I most likely won't have any invalid commands entered so it works for me. commands = ["show service-profile status", "show fabric-interconnect inventory", "connect nxos a", "show mac address", "show interface description", "exit", "connect nxos b", "show mac address", "show interface description"]
async def run_client_multiple_commands(
self, host, username, password, commands, command_name_key=[]
):
if not commands:
return None
if isinstance(commands, str):
commands = [commands]
logger.debug(f"Running commands on host {host}: {commands}")
try:
asyncssh.set_log_level('DEBUG')
asyncssh.set_sftp_log_level('DEBUG')
asyncssh.set_debug_level(2)
async with await asyncio.wait_for(
asyncssh.connect(
host,
username=username,
password=password,
port=22,
known_hosts=None,
),
timeout=10,
) as conn:
async with await asyncio.wait_for(
conn.create_process(stderr=asyncssh.STDOUT), timeout=10
) as process:
res = {}
for i, command in enumerate(commands):
logger.debug(f'--- Running "{command}" ---')
process.stdin.write(f"{command}\n")
process.stdin.write("show show\n") # Delimiter between commands
for command in commands:
if command not in ["connect nxos", "connect nxos a", "connect nxos b"]:
if command_name_key:
if isinstance(command_name_key, str):
command_name_key = [command_name_key]
res[command_name_key[i]] = res.get(command_name_key[i], '') + await process.stdout.readuntil(re.compile("Invalid Command|show show"), 15)
else:
res[command] = res.get(command, '') + await process.stdout.readuntil(re.compile("Invalid Command|show show"), 15)
process.stdin.write_eof()
return res
except asyncio.TimeoutError as e:
logger.warning(f"Timeout while connecting to {host}: {e}") |
|
Regarding the command output boundary, does it work to start a command with an '!'? Many switches will treat everything from there to the end of the line as a comment and ignore it, but I don't know if it works on a line by itself with no command preceding it. The other thought I had was to maybe set the CLI prompt that the switch outputs to something unique that you don't expect to see in the output of the commands you are running. If you search for that, you might not need to send the "show show" and match on invalid command errors. You could just directly readuntil that prompt. This might also avoid the need for a timeout, though you may still need the PTY depending on whether the switch outputs prompts when no PTY is allocated. On another note, if you are allocating a PTY, you don't need to worry about redirecting stderr to stdout. With a PTY allocated, that happens automatically and the two streams are merged into one. So, the |
After testing, "!" does work in the nxos terminal but not the main ucs terminal.
I do not know if my read access would allow me to do this. Also, the main ucs terminal does not have a setting to change the prompt.
This is good to know. The main ucs terminal is very limited in what it can do and also does it differently than the nxos terminal. The only way you could do this is by sending a multiple \n and have the regex pick up if multiple CLI prompts have happened in succession. This works in both environments, plus you can send extra newlines to increase the chances to match in case some of the newlines get corrupted or get interrupted by switch logs. Its a bit costly, but programmatically no one should be sending an empty newline without a command. |
I don't really have any experience with UCS, but looking online for a CLI reference, I don't see any mention of supporting comments the way other Cisco CLIs do, so this may not be possible.
Yeah - maybe not. Are the existing prompts already unique enough to serve this purpose, though? Perhaps you could parse out the first prompt and then remember that and use it for finding the end of the output for future commands. Sending an extra newline here to get two prompts in a row is an interesting idea, if you think the single prompt isn't unique enough by itself. |
They should be unique enough.
The main problem is not the uniqueness, rather network devices also contain logging that disrupts the commands being sent. I don't know how they fully work with ssh output but I am worried if someone shuts down a port, it will send a log followed by a new command prompt line. Edit: |
|
Yeah - if there is such output, it may get captured as part of some command you are running, but it shouldn't change the prompts. The only possible issue I could imagine is if someone the log output came out in the middle of the prompt, but I'm not even sure if that's possible. It would depend on where (if at all) buffering was happening on the output. |
|
Unfortunately, my workaround started hanging on a Cisco device running a maintenance release of NXOS in a similar fashion to #628. Even using a |
|
Do you have a sample of the debug log from when the problem occurred (preferable at debug level 2)? Also, what specifically are you calling A call to If you are calling A hang could also be triggered by an exception you are not catching, but you should be able to see that show up once you enable debug logging. |
|
I am calling It should not be an exception that is causing it, there is no indication that it is happening. The function still works for most Cisco NXOS device in the environment so I don't know what it is causing it to break here. @staticmethod
async def ssh(
host: str,
username: str,
password: str,
commands: List[str],
command_name_key: List[str] = [],
ssh_debug: bool = False,
exception_traceback: bool = False,
timeout: int = 10,
) -> Optional[Dict[str, Any]]:
if not commands:
return None
if isinstance(commands, str):
commands = [commands]
if ssh_debug:
asyncssh.set_log_level("DEBUG")
asyncssh.set_sftp_log_level("DEBUG")
asyncssh.set_debug_level(2)
logger.debug(
f"Running command on host {BGB}{host}{RESET}: {BGB}{commands}{RESET}"
)
# Connect to the host using AsyncSSH
try:
async with await asyncio.wait_for(
asyncssh.connect(
host,
username=username,
password=password,
port=22,
known_hosts=None,
),
timeout=timeout,
) as conn:
# Create interactive SSH session on host
async with await asyncio.wait_for(
conn.create_process(stderr=asyncssh.STDOUT), timeout=timeout
) as process:
res = {}
# Run each command on the host
for i, command in enumerate(commands):
logger.debug(f'--- Running "{BGB}{command}{RESET}" ---')
process.stdin.write(f"{command}\n")
# Delimiter between commands
process.stdin.write("show show\n")
# Read the output of each command
for command in commands:
# Workaround to ignore the connect commands on UCS chassis
if command not in [
"connect nxos",
"connect nxos a",
"connect nxos b",
]:
# Set output using command_name_key
if command_name_key:
if isinstance(command_name_key, str):
command_name_key = [command_name_key]
res[command_name_key[i]] = res.get(
command_name_key[i], ""
) + await asyncio.wait_for(process.stdout.readuntil(
# Read until stderr containing "Invalid Command" or "show show"
re.compile("Invalid Command|show show"),
max_separator_len=15,
), timeout=timeout)
# Else set output using actual command
else:
res[command] = res.get(
command, ""
) + await asyncio.wait_for(process.stdout.readuntil(
# Read until stderr containing "Invalid Command" or "show show"
re.compile("Invalid Command|show show"),
max_separator_len=15,
), timeout=timeout)
# Force the process to close
process.stdin.write_eof()
return res
except Exception as e:
logger.error(f"Failed to connect to host {host}: {e}")
raise CommandExecutionError(host, f"{e.__class__.__module__}.{e.__class__.__name__}: {e}")
finally:
if exception_traceback:
logger.error(traceback.format_exc()) |
|
I can't be sure, but it looks like this may be a case where the server is ignoring the EOF you are sending and keeping the session open despite that. If that's the case, you may need to send something like an "exit" command at the end instead of sending EOF. If that triggers the server to send EOF (or close) back to you, the implicit |
|
I just tried adding an |
|
Did you add it inside the "async with block for the process, at the same place where the write_eof() call is now? Do you see anything different in the debug output as a result of that change? If the exit isn't triggering a close, I wonder if somehow that's getting sent before the remote device is back at a command prompt. You could try adding a small delay before sending the "exit" to see if that makes any difference. |
|
Yes I put it at the same place where the write_eof() is. I have tried it with and without the eof also. The only difference I see is the output sends 5 bytes ("exit\n") before logging Closing Channel. If I add a delay before the exit, Closing Channel is logged and no further logs happen even after the delay. |
|
I don't really understand why the Cisco device isn't closing the channel. Even if it ignored both the "exit" and the EOF, it really should send us back a close when we send our close. Without it, it's expected for A workaround here would be to not use a context manager for the call to |
|
Okay it looks that workaround did work correctly. Is there any downside to using this workaround? |
|
Since you are exiting the outer context manager (for the connection) immediately after doing so for the SSH session/process, there's really no downside in this case. In fact, you can probably get by without even calling The place where this might be an issue would be where you are opening a long-lived connection object and then opening multiple sessions on it. If the server never responds with a close for those sessions, you could end up building up multiple session/process objects that don't get cleaned up even though you've called |
Hi Ron,
I am working on Cisco UCS Fabric Interconnects and I am running into issues with it hanging when awaiting for the stdout.
Cisco UCS requires me to use the
connect nxos aandconnect nxos bcommands to access the commands I need to run. I found a solution to getconnect nxos ato work but notconnect nxos b.I am currently using an interactive shell to run all of the commands at once and dump them at the end of the session. I have also been using write_eof() to get sessions to close on normal Cisco and Arista switches since they would hang without it. Please tell me if doing this is a bad idea or not.
with process.write_eof()
Without process.write_eof()
I believe that the
connect nxos bcreates a ssh tunnel to its pair and that is why it is not working as intended. When run in the CLI it displaysOperating in CiscoSSL FIPS modewhich does not show when runningconnect nxos a. Is there a way I can get this to work without having to connect directly to the other host?Another thing that is weird is that I have to write the exit command for it to not hang while using
connect nxos a. If I am connected to a normal Cisco or Arista device using exit is not necessary andcommands = ["show mac address"]would return and not hang for this code. How can I better grasp what AsyncSSH is doing here?Originally, I was using Paramiko and did get this to work so hopefully it is possible in AsyncSSH as well.
Please let me know your thoughts on this.
The text was updated successfully, but these errors were encountered: