Skip to content

Fix UTF-8 input handling in interactive CLI - #11

Merged
Aaryanverma merged 2 commits into
Aaryanverma:mainfrom
gilsonolegario:fix/utf8-interactive-input
Aug 8, 2026
Merged

Fix UTF-8 input handling in interactive CLI#11
Aaryanverma merged 2 commits into
Aaryanverma:mainfrom
gilsonolegario:fix/utf8-interactive-input

Conversation

@gilsonolegario

Copy link
Copy Markdown
Contributor

Summary

Typing accented characters (e.g. água) in the interactive CLI input stores them without the accent (gua).

Root cause

  1. _getch() reads a single byte and decodes it in isolation with decode("utf-8", "ignore"). Multi-byte UTF-8 characters (e.g. á = 0xC3 0xA1) arrive as separate bytes; each lone byte is invalid UTF-8 and is silently discarded.
  2. tty.setraw() uses TCSAFLUSH, which discards queued input on every keypress — breaking pasted or fast-typed multi-byte text.

Fix

  • Assemble full UTF-8 sequences in _getch (_utf8_sequence_len + _decode_key) before decoding.
  • Switch to raw mode with tty.setraw(fd, termios.TCSANOW) so pending input is not flushed between reads.

Tests

Added tests/test_cli_input.py with unit tests for the UTF-8 helpers plus a pty-based integration test that drives _interactive_input with água para as plantas and asserts the full string is returned.

Full suite: 211 passed.

@Aaryanverma

Copy link
Copy Markdown
Owner

@gilsonolegario Thanks for the PR.

Before merging:
Can you confirm this has been tested manually on Linux/macOS with a few non-ASCII characters? The CI doesn't appear to have run any jobs, so I'd like one manual verification.

Also: The Windows path still uses msvcrt.getch() and isn't covered by the new integration test. It would be good to either verify that path or open a follow-up issue/PR to improve UTF-8 handling there.

@gilsonolegario

Copy link
Copy Markdown
Contributor Author

Thanks for the review — both points addressed.

Manual verification (macOS): Drove the real CLI capture flow through a pty (the same _getch_interactive_inputcmd_capture path) and typed água do coração 日本語 🪴 — covering 2-byte (á/ç), 3-byte (CJK), and 4-byte (emoji) UTF-8. The stored inbox item preserves the string exactly:

água do coração 日本語 🪴

Also confirmed paste/fast-typed input survives (whole string written in one shot, exercising the TCSANOW change). Full suite locally: 211 passed.

Windows path: Correct — msvcrt.getch() still decoded a single byte with "ignore", so accented input had the same loss as the original bug. I applied the same fix to that branch: _getch now assembles the full UTF-8 sequence via the already-tested _utf8_sequence_len/_decode_key helpers, gated on msvcrt.kbhit(), mirroring the termios path. It is not covered by any running CI job and I have no Windows box here, so it needs a real-Windows smoke test — happy to open a follow-up if you want it tracked.

@Aaryanverma
Aaryanverma merged commit 9c18ec5 into Aaryanverma:main Aug 8, 2026
4 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants