Skip to content

Implement utf8 support and solve #8 issue #47

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Melnytskyi wants to merge 11 commits into
base: master
Choose a base branch
from
Open

Implement utf8 support and solve #8 issue #47

Show file tree
Hide file tree
Changes from 1 commit
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e58cbb1
Implement utf8 support and solve #8 issue
Melnytskyi Jul 1, 2024
be67d3f
Fix linux build
Melnytskyi Jul 6, 2024
f47a2f3
FIx UTF8 length handling in `current_view`
Melnytskyi Jul 6, 2024
d356a1e
Change cstdint to cstddef
Jul 6, 2024
41e33e9
Merge branch 'lionkor:master' into master
Melnytskyi May 21, 2025
0a4dc01
Update cmake required
May 23, 2025
4c77ec6
Bump c++ version
Melnytskyi Sep 15, 2025
48d3592
fix UB for empty buffer
Melnytskyi Sep 20, 2025
748f92c
Update src/backends/utf8.hpp
Melnytskyi Sep 20, 2025
791bebe
Revert CMake changes
Melnytskyi Sep 20, 2025
6eba2a7
Bump cmake to 3.10
Melnytskyi Sep 23, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
129 changes: 76 additions & 53 deletions src/backends/InteractiveBackend.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2,6 +2,44 @@

#include "impls.h"

size_t make_left(const char* arr, size_t pos) {
size_t res = 1;
--pos;
while (pos) {
if ((arr[pos] & 0xc0) == 0x80) {
++res;
} else {
break;
}
--pos;
}
return res;
}
size_t make_right(const char* arr, size_t pos) {
size_t res = 1;
++pos;
if ((arr[pos] & 0xc0) == 0x80) {
while ((arr[pos] & 0xc0) == 0x80) {
++res;
++pos;
}
}
return res;
}

size_t utf8_length(const char* str, size_t siz = -1) {
Copy link
Owner

@lionkor lionkor Jul 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

would it be reasonable to use a library to handle UTF-8?

Copy link
Author

@Melnytskyi Melnytskyi Jul 4, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, there's a header-only library that might suit this repo. At first, I thought this library was too big for only three functions. Mb just add this lib as a Git submodule? Or some other lib?

Copy link
Author

@Melnytskyi Melnytskyi Jul 6, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I tried this library and changed my mind, because it checks every character and throws exceptions on non-valid input and that needs to be handled. I have no idea how to handle such things in a separate thread for console reading and whether they make sense in the console. That's why I decided to improve the current functions so that they just return values that will definitely not lead to an out of range.

If it is not suitable, I can redo it.

Copy link
Owner

@lionkor lionkor Jul 8, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, thanks for looking into it! I'll read up on UTF-8 so I can be sure that this is conformant, but it looks good. Thank you!

Copy link
Author

@Melnytskyi Melnytskyi Aug 1, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any news?

Copy link
Owner

@lionkor lionkor Feb 3, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I genuinely have not had the time or motivation to understand the unicode utf 8 spec to a point where i can merge this code without concerns :/

size_t len = 0;

while (*str && len != siz) {
len += (*str++ & 0xc0) <= 0x80; // normal ascii or codepoint
}
return len;
}

size_t utf8_length(const std::string& str) {
return utf8_length(str.c_str(), str.size());
}

lk::InteractiveBackend::InteractiveBackend(const std::string& prompt)
: Backend()
, m_prompt(prompt) {
Expand All @@ -27,6 +65,14 @@ std::string lk::InteractiveBackend::prompt() const {
void lk::InteractiveBackend::add_to_current_buffer(char c) {
m_current_buffer.insert(m_cursor_pos, 1, c);
++m_cursor_pos;
++m_cursor_pos_view;
update_current_buffer_view();
m_history_temp_buffer = m_current_buffer;
}
void lk::InteractiveBackend::add_to_current_buffer_multibyte(char* c, size_t count) {
m_current_buffer.insert(m_cursor_pos, c, count);
m_cursor_pos += count;
++m_cursor_pos_view;
update_current_buffer_view();
m_history_temp_buffer = m_current_buffer;
}
Expand All @@ -48,6 +94,7 @@ void lk::InteractiveBackend::go_back() {
m_current_buffer = m_history.at(m_history_index);
}
m_cursor_pos = (int)m_current_buffer.size();
m_cursor_pos_view = utf8_length(m_current_buffer);
update_current_buffer_view();
}

Expand All @@ -63,32 +110,37 @@ void lk::InteractiveBackend::go_forward() {
m_current_buffer = m_history.at(m_history_index);
}
m_cursor_pos = (int)m_current_buffer.size();
m_cursor_pos_view = utf8_length(m_current_buffer);
update_current_buffer_view();
}

void lk::InteractiveBackend::go_left() {
if (m_cursor_pos > 0 && !m_current_buffer.empty()) {
--m_cursor_pos;
--m_cursor_pos_view;
m_cursor_pos -= make_left(m_current_buffer.c_str(), m_cursor_pos);
update_current_buffer_view();
}
}

void lk::InteractiveBackend::go_right() {
if (size_t(m_cursor_pos) < m_current_buffer.size()) {
++m_cursor_pos;
++m_cursor_pos_view;
m_cursor_pos += make_right(m_current_buffer.c_str(), m_cursor_pos);
update_current_buffer_view();
}
}

void lk::InteractiveBackend::go_to_begin() {
if (m_cursor_pos > 0 && !m_current_buffer.empty()) {
m_cursor_pos = 0;
m_cursor_pos_view = 0;
update_current_buffer_view();
}
}

void lk::InteractiveBackend::go_to_end() {
m_cursor_pos = (int)m_current_buffer.size();
m_cursor_pos_view = utf8_length(m_current_buffer);
update_current_buffer_view();
}

Expand Down Expand Up @@ -140,17 +192,21 @@ bool lk::InteractiveBackend::cancel_autocomplete_suggestion() {

void lk::InteractiveBackend::handle_backspace() {
if (!cancel_autocomplete_suggestion() && !m_current_buffer.empty()) {
if (--m_cursor_pos < 0) {
size_t remove_count = make_left(m_current_buffer.c_str(), m_cursor_pos);
--m_cursor_pos_view;
if ((m_cursor_pos -= remove_count) < 0) {
m_cursor_pos = 0;
m_cursor_pos_view = 0;
}
m_current_buffer.erase(m_cursor_pos, 1);
m_current_buffer.erase(m_cursor_pos, remove_count);
update_current_buffer_view();
}
}

void lk::InteractiveBackend::handle_delete() {
if (!m_current_buffer.empty() && m_cursor_pos < int(m_current_buffer.size())) {
m_current_buffer.erase(m_cursor_pos, 1);
size_t remove_count = make_right(m_current_buffer.c_str(), m_cursor_pos);
m_current_buffer.erase(m_cursor_pos, remove_count);
update_current_buffer_view();
}
}
Expand All @@ -160,8 +216,6 @@ void lk::InteractiveBackend::handle_escape_sequence(std::unique_lock<std::mutex>
if (m_key_debug) {
fprintf(stderr, "c2: 0x%.2x\n", c2);
}

#if defined(UNIX)
int c3 = impl::getchar_no_echo();
if (m_key_debug) {
fprintf(stderr, "c3: 0x%.2x\n", c3);
Expand Down Expand Up @@ -195,34 +249,9 @@ void lk::InteractiveBackend::handle_escape_sequence(std::unique_lock<std::mutex>
// SHIFT+TAB
handle_tab(guard, false);
}
#elif defined(WINDOWS)
if (c2 == 'H') {
// up / back
go_back();
} else if (c2 == 'P') {
// down / forward
go_forward();
} else if (c2 == 'K') {
// left
go_left();
} else if (c2 == 'M') {
// right
go_right();
} else if (c2 == 0x47) {
// HOME
go_to_begin();
} else if (c2 == 0x4f) {
// END
go_to_end();
} else if (c2 == 0x53) {
// DEL
handle_delete();
#endif
} else {
add_to_current_buffer(c2);
#if defined(UNIX)
add_to_current_buffer(c3);
#endif
}
}

Expand All @@ -231,47 +260,41 @@ void lk::InteractiveBackend::input_thread_main() {
int c = 0;
while (c != '\n' && c != '\r' && !m_shutdown.load()) {
update_current_buffer_view();
c = impl::getchar_no_echo();
char utf8_c[7];
size_t read = 0;
if (!impl::getchar_utf8_no_echo(utf8_c, read))
continue;
c = utf8_c[0];
if (m_key_debug) {
fprintf(stderr, "c: 0x%.2x\n", c);
}
std::unique_lock<std::mutex> guard(m_current_buffer_mutex);
if (c != '\t') {
}
if (c == '\b' || c == 127) { // backspace or other delete sequence
if (utf8_c[0] == '\b' || c == 127) {
handle_backspace();
clear_suggestions();
} else if (c == '\t') {
handle_tab(guard, true);
} else if (isprint(c)) {
add_to_current_buffer(c);
clear_suggestions();
} else if (c == 0x1b) { // escape sequence
#if defined(UNIX)
handle_escape_sequence(guard);
#else
cancel_autocomplete_suggestion();
#endif
} else if (c == 0xe0) { // escape sequence
#if defined(WINDOWS)
handle_escape_sequence(guard);
#endif
} else {
if (m_key_debug) {
fprintf(stderr, "unhandled: 0x%.2x\n", c);
}
add_to_current_buffer_multibyte(utf8_c, read);
clear_suggestions();
}
}
bool shutdown = m_shutdown.load();
// check so we dont do anything on the last pass before exit
if (!shutdown) {
m_current_buffer.erase(m_current_buffer.size() - 1);

if (history_enabled() && m_current_buffer.size() > 0) {
add_to_history(m_current_buffer);
}
std::lock_guard<std::mutex> guard(m_to_read_mutex);
m_to_read.push(m_current_buffer);
m_current_buffer.clear();
m_cursor_pos = 0;
m_cursor_pos_view = 0;
update_current_buffer_view();
}
if (on_command && !shutdown) {
Expand Down Expand Up @@ -403,18 +426,18 @@ std::string lk::InteractiveBackend::current_view() {
size_t lk::InteractiveBackend::current_view_cursor_pos() {
const auto view = current_view_size();
if (current_view_offset() + view > view) {
return m_cursor_pos + m_prompt.size() - current_view_offset() + 2;
return m_cursor_pos_view + m_prompt.size() - current_view_offset() + 2;
} else {
return m_cursor_pos + m_prompt.size() - current_view_offset() + 1;
return m_cursor_pos_view + m_prompt.size() - current_view_offset() + 1;
}
}

size_t lk::InteractiveBackend::current_view_offset() {
const auto view_size = current_view_size();
if (m_cursor_pos < view_size) {
if (m_cursor_pos_view < view_size) {
return 0;
}
return m_cursor_pos - view_size;
return m_cursor_pos_view - view_size;
}
size_t lk::InteractiveBackend::current_view_size() {
const auto w = impl::get_terminal_width() - m_prompt.size() - 1;
Expand Down
2 changes: 2 additions & 0 deletions src/backends/InteractiveBackend.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ class InteractiveBackend : public Backend {
void go_back_in_history();
void go_forward_in_history();
void add_to_current_buffer(char c);
void add_to_current_buffer_multibyte(char* c, size_t);
void update_current_buffer_view();
void handle_escape_sequence(std::unique_lock<std::mutex>& guard);
void handle_backspace();
Expand Down Expand Up @@ -87,6 +88,7 @@ class InteractiveBackend : public Backend {
std::mutex m_current_buffer_mutex;
std::string m_current_buffer;
int m_cursor_pos = 0;
int m_cursor_pos_view = 0;
std::vector<std::string> m_autocomplete_suggestions;
size_t m_autocomplete_index = 0;
std::string m_buffer_before_autocomplete;
Expand Down
1 change: 1 addition & 0 deletions src/impls.h
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,7 @@ bool is_interactive();
void init_terminal();
void reset_terminal();
int getchar_no_echo();
bool getchar_utf8_no_echo(char (&character)[7], size_t& used);
bool is_shift_pressed(bool forward);
int get_terminal_width();
}
Expand Down
22 changes: 22 additions & 0 deletions src/linux_impl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -56,6 +56,28 @@ int impl::getchar_no_echo() {
return detail::getch_(false);
}

bool impl::getchar_utf8_no_echo(char (&character)[7], size_t& used) {
used = 1;
int first = character[0] = getchar_no_echo();
if (first & 0x80) {
int count = 0;
if ((first 0xE0) == 0xC0) {
count = 1;
} else if ((first 0xF0) == 0xE0) {
count = 2;
} else if ((first 0xF8) == 0xF0) {
count = 3;
} else {
return false;
}
for (int i = 0; i < count; i++) {
character[used++] = getchar_no_echo();
}
}
character[used] = 0;
return true;
}

bool impl::is_shift_pressed(bool forward) {
return forward;
}
Expand Down
47 changes: 42 additions & 5 deletions src/windows_impl.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -12,11 +12,22 @@ bool impl::is_interactive() {
}

void impl::init_terminal() {
HANDLE hConsole_c = GetStdHandle(STD_OUTPUT_HANDLE);
DWORD dwMode = 0;
GetConsoleMode(hConsole_c, &dwMode);
dwMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
SetConsoleMode(hConsole_c, dwMode);
SetConsoleOutputCP(CP_UTF8);
{
HANDLE hConsole_c = GetStdHandle(STD_OUTPUT_HANDLE);
DWORD dwMode = 0;
GetConsoleMode(hConsole_c, &dwMode);
dwMode |= ENABLE_VIRTUAL_TERMINAL_PROCESSING;
SetConsoleMode(hConsole_c, dwMode);
}
{
HANDLE hConsole_c = ::GetStdHandle(STD_INPUT_HANDLE);
DWORD dwMode = 0;
::GetConsoleMode(hConsole_c, &dwMode);
dwMode &= ~(ENABLE_ECHO_INPUT | ENABLE_LINE_INPUT);
dwMode |= ENABLE_VIRTUAL_TERMINAL_INPUT;
::SetConsoleMode(hConsole_c, dwMode);
}
}

void impl::reset_terminal() {
Expand All @@ -26,6 +37,32 @@ int impl::getchar_no_echo() {
return _getch();
}

bool impl::getchar_utf8_no_echo(char (&character)[7], size_t& used) {
used = 0;
HANDLE hConsole_c = GetStdHandle(STD_INPUT_HANDLE);

wchar_t input[3];
DWORD dummy = 0;
DWORD lpNumberOfCharsRead = 1;

if (!ReadConsoleW(hConsole_c, input, 1, &dummy, nullptr)) {
return false;
}

if (IS_HIGH_SURROGATE(input[0])) {
if (!ReadConsoleW(hConsole_c, input + 1, 1, &dummy, nullptr)) {
return false;
}
lpNumberOfCharsRead += 1;
}
input[lpNumberOfCharsRead] = 0;
used = WideCharToMultiByte(CP_UTF8, 0, input, lpNumberOfCharsRead, character, 7, nullptr, nullptr);
if (!used) {
return false;
}
return true;
}

bool impl::is_shift_pressed(bool forward) {
auto x = uint16_t(GetKeyState(VK_SHIFT));
if (x > 1) {
Expand Down