Skip to content

[CLIENT-4836] Add support for string operations and expressions - #1072

Merged
juliannguyen4 merged 261 commits into
devfrom
CLIENT-4836-add-string-ops-and-expressions
Jul 1, 2026
Merged

[CLIENT-4836] Add support for string operations and expressions#1072
juliannguyen4 merged 261 commits into
devfrom
CLIENT-4836-add-string-ops-and-expressions

Conversation

@juliannguyen4

@juliannguyen4 juliannguyen4 commented May 20, 2026

Copy link
Copy Markdown
Collaborator
  • Deprecate using aerospike_helpers.operations.operations.{append,prepend} with strings.
  • Add cross references from string expressions to string operations since the latter has more detailed documentation about behavior.

Extra changes

  • CLIENT-4944 Replace precedence rules for the "key" policy option with a union-type rule.
  • Fix potential memory leak found when Query.where() runs successfully.
  • CLIENT-4820 Disable TLSv1.3 support for now as the server does not currently support it.
  • Disable running tests with flags that convert DeprecationWarning's into errors. This is because aerospike_helpers.operations.operations.append() and aerospike_helpers.operations.operations.prepend() are deprecated but are still used throughout the test suite. We still to support these calls since the new string ops append() and prepend() are only supported in server version >= 8.1.3. If we enable errors for DeprecationWarnings, all the operations.append()/prepend() calls will fail out with ClientError; we need these operations to succeed.

TODO

  • Python docstrings in VS code are not formatted. So the placeholders still show up in type hinting. This problem may also exist with extended metrics / NodeStats in aerospike_helpers. There's no well known solution for this, so I replaced all the placeholders manually.
  • Ops dictionaries now have extra variables we don't want (found via GDB). Need to omit those... only way is to filter out locals() using dictionary comprehension, but this is O(n) time. Resolved by just returning a dictionary with only the arg
  • Expressions need to state return value type since they are strongly typed Ignoring since it makes the return description too long IMO
  • Cross reference for expressions to corresponding operations doesn't show up in IDE type hints Cross reference generated at runtime. This isn't a big deal since the online API docs will show the cross references
  • Align with other clients re: snip. Make end argument required
  • Add copyright notices to all new files
  • Regression if deprecating CmpRegex? I'm guessing only for non-valid UTF-8 strings
  • Test failures on server 8.1.3 with test_prepend.py, test_operate.py. I also tested with Python client's dev and also seeing test failures Fixed by changing server setup to support setting ttl

Out of scope

  • Find way to automate copyright notices
  • add_expr_macros - wondering if a failing negative codepath can cause memory leaks due to returning err->code immediately. Leaving this alone for now since this is an existing problem in dev that is out of scope for this PR.
  • Document when query.where() memory leak was introduced. Won't do this for now because I couldn't narrow down the test case that was causing this leak in this feature branch. I'm assuming it's from the new tests which I haven't run against dev.
  • This is currently missing the string ops code examples to be used downstream by the aerospike.com website. I'll add this in a future PR before the next client release

Docs

@codecov-commenter

codecov-commenter commented May 20, 2026

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 92.83920% with 57 lines in your changes missing coverage. Please review.
✅ Project coverage is 84.97%. Comparing base (0e88abb) to head (64b7667).
⚠️ Report is 1 commits behind head on dev.

Files with missing lines Patch % Lines
src/main/query/where.c 50.00% 16 Missing ⚠️
src/main/convert_expressions.c 92.53% 15 Missing ⚠️
src/main/client/list_and_string_operate.c 94.73% 10 Missing ⚠️
src/main/client/cdt_operation_utils.c 66.66% 8 Missing ⚠️
src/main/conversions.c 77.77% 4 Missing ⚠️
src/main/client/operate.c 81.81% 2 Missing ⚠️
aerospike_helpers/expressions/string.py 99.54% 1 Missing ⚠️
aerospike_helpers/operations/string_operations.py 98.76% 1 Missing ⚠️
Additional details and impacted files
@@            Coverage Diff             @@
##              dev    #1072      +/-   ##
==========================================
+ Coverage   84.46%   84.97%   +0.51%     
==========================================
  Files          99      102       +3     
  Lines       14090    14839     +749     
==========================================
+ Hits        11901    12610     +709     
- Misses       2189     2229      +40     

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

…reasing numeric order) instead of the default alphabetical order
…o define set of _AS_STRING_OP* codes separate from the C client's AS_STRING_OP* codes. The _AS_STRING* codes need to be unique from the other op codes exposed via aerospike_helpers. Then use X macro to expose string op codes from aerospike module
…e type stubs don't show any constants with value 127, so I believe _AS_EXP_CODE_CALL is safe to expose for aerospike_helpers.expressions to use
…s for class defined in this module and not the imported classes.
@juliannguyen4

Copy link
Copy Markdown
Collaborator Author
PYTHONMALLOC=malloc valgrind --log-file=CLIENT-4836.log --leak-check=full python3 -m pytest new_tests/test_string_operations.py -k "test_concat[expect_earlier_than_server_version_to_fail0-kwargs_policy0-no_ctx_arg-value_list0]"
...
==130221== 101 (56 direct, 45 indirect) bytes in 1 blocks are definitely lost in loss record 1,995 of 3,020
==130221==    at 0x4846828: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
==130221==    by 0x86588EC: as_arraylist_new (as_arraylist.c:57)
==130221==    by 0x85B7F89: pyobject_to_list (conversions.c:745)
==130221==    by 0x859FF65: get_val_list (cdt_operation_utils.c:122)
==130221==    by 0x85A71DF: add_list_or_string_op (list_and_string_operate.c:219)
==130221==    by 0x85A9263: add_op (operate.c:366)
==130221==    by 0x85AB309: AerospikeClient_Operate_Invoke (operate.c:974)
==130221==    by 0x85ABE7A: AerospikeClient_Operate (operate.c:1067)
==130221==    by 0x5516B7: ??? (in /usr/bin/python3.12)
==130221==    by 0x549CB4: PyObject_Vectorcall (in /usr/bin/python3.12)
==130221==    by 0x5D7388: _PyEval_EvalFrameDefault (in /usr/bin/python3.12)
==130221==    by 0x54CEB3: ??? (in /usr/bin/python3.12)

@juliannguyen4

Copy link
Copy Markdown
Collaborator Author

TODO - only will run valgrind / massif but not build artifacts on server 8.1.3 since we don't have JFrog images yet.

Merging this in ASAP to give QE a build with string ops

@juliannguyen4

juliannguyen4 commented Jun 25, 2026

Copy link
Copy Markdown
Collaborator Author

Memory leak no longer shows up on github actions (with server 8.1.2.x)

But now getting these test failures.

=========================== short test summary info ============================
FAILED new_tests/test_dynamic_config.py::TestDynamicConfig::test_dyn_config_file_has_highest_precedence[False] - assert 1 is None
FAILED new_tests/test_dynamic_config.py::TestDynamicConfig::test_dyn_config_file_has_highest_precedence[True] - assert 1 is None
FAILED new_tests/test_max_error_rate.py::TestMaxErrorRate::test_max_error_rate[100] - exception.ClientError: (-1, 'Dynamic config interval 1000 must be greater or equal to the tend interval 100000', 'src/main/aerospike/as_cluster.c', 1530, False)
FAILED new_tests/test_max_error_rate.py::TestMaxErrorRate::test_max_error_rate[101] - exception.ClientError: (-1, 'Dynamic config interval 1000 must be greater or equal to the tend interval 100000', 'src/main/aerospike/as_cluster.c', 1530, False)
FAILED new_tests/test_max_error_rate.py::TestMaxErrorRate::test_max_error_rate[99] - exception.ClientError: (-1, 'Dynamic config interval 1000 must be greater or equal to the tend interval 100000', 'src/main/aerospike/as_cluster.c', 1530, False)
FAILED new_tests/test_metrics.py::TestMetrics::test_setting_metrics_policy_custom_settings[None] - assert False is True
FAILED new_tests/test_metrics.py::TestMetrics::test_setting_metrics_policy_custom_settings[my_app] - assert False is True
FAILED new_tests/test_metrics.py::TestMetrics::test_disable_metrics_throwing_exc - Failed: DID NOT RAISE <class 'exception.AerospikeError'>
= 8 failed, 3745 passed, 236 skipped, 25 xfailed, 18 xpassed, 1 warning in 716.09s (0:11:56) =

The failing dynamic config tests here use the send key policy, so I suspect the send key changes in C client commit f0e030590eb33f7182c04e35fa692edf2f1de530 are causing these failures

…tion memory leak (found in happy path test)."

This reverts commit a894ed4.
…ound for not using Shannon's memory leak fix in the C client because it comes after the send key changes which I suspect are causing multiple test failures unrelated to string ops.
@juliannguyen4
juliannguyen4 force-pushed the CLIENT-4836-add-string-ops-and-expressions branch from 6e69f26 to 0ac4e2e Compare June 26, 2026 16:12
@juliannguyen4

juliannguyen4 commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator Author

Currently all string write op tests are failing against latest server master commit with string ops support

Waiting for Nate to respond today or Monday.

Caught memory error at 0ac4e2e

echo 28251888648 | xargs gh run view --log | cut -f 3- | less

Rerunning valgrind on send key PR & latest HEAD here to see if memory error is still there.

Valgrind here: https://github.com/aerospike/aerospike-client-python/actions/runs/28267287865/job/83757294667

2026-06-26T22:13:44.5462725Z new_tests/test_dynamic_config.py::TestDynamicConfig::test_config_provider_class_invalid_args PASSED
2026-06-26T22:13:46.7099438Z ==4796== Syscall param write(buf) points to uninitialised byte(s)
2026-06-26T22:13:46.7100320Z ==4796==    at 0x4D2893F: __libc_write (write.c:26)
2026-06-26T22:13:46.7100822Z ==4796==    by 0x4D2893F: write (write.c:24)
2026-06-26T22:13:46.7102080Z ==4796==    by 0x97D4024: sock_write (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libcrypto-ad2cd6c1.so.3)
2026-06-26T22:13:46.7103818Z ==4796==    by 0x97CAF66: bwrite_conv (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libcrypto-ad2cd6c1.so.3)
2026-06-26T22:13:46.7105494Z ==4796==    by 0x97C991D: BIO_write (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libcrypto-ad2cd6c1.so.3)
2026-06-26T22:13:46.7107271Z ==4796==    by 0x9673C7F: ssl3_write_pending (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libssl-7d9e47d9.so.3)
2026-06-26T22:13:46.7109286Z ==4796==    by 0x9674EAA: do_ssl3_write (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libssl-7d9e47d9.so.3)
2026-06-26T22:13:46.7111035Z ==4796==    by 0x967519E: ssl3_write_bytes (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libssl-7d9e47d9.so.3)
2026-06-26T22:13:46.7112339Z ==4796==    by 0x96560E4: SSL_write (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libssl-7d9e47d9.so.3)
2026-06-26T22:13:46.7113102Z ==4796==    by 0x9315F01: as_tls_write (as_tls.c:1295)
2026-06-26T22:13:46.7113624Z ==4796==    by 0x930DD73: as_socket_write_deadline (as_socket.c:317)
2026-06-26T22:13:46.7114168Z ==4796==    by 0x92DBD01: as_command_execute (as_command.c:760)
2026-06-26T22:13:46.7114687Z ==4796==    by 0x92DC63B: as_command_send (as_command.c:662)
2026-06-26T22:13:46.7115268Z ==4796==  Address 0xc4d51fc is 28 bytes inside a block of size 16,584 alloc'd
2026-06-26T22:13:46.7116356Z ==4796==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
2026-06-26T22:13:46.7117326Z ==4796==    by 0x9676D66: ssl3_setup_write_buffer (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libssl-7d9e47d9.so.3)
2026-06-26T22:13:46.7118832Z ==4796==    by 0x9676E4E: ssl3_setup_buffers (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libssl-7d9e47d9.so.3)
2026-06-26T22:13:46.7120638Z ==4796==    by 0x9687A50: state_machine (in /opt/hostedtoolcache/Python/3.10.20/x64/lib/python3.10/site-packages/aerospike.libs/libssl-7d9e47d9.so.3)
2026-06-26T22:13:46.7121871Z ==4796==    by 0x93151D9: as_tls_connect (as_tls.c:1017)
2026-06-26T22:13:46.7122699Z ==4796==    by 0x930D526: as_socket_start_connect (as_socket.c:193)
2026-06-26T22:13:46.7123598Z ==4796==    by 0x92FADF7: as_node_try_family_connections (as_node.c:446)
2026-06-26T22:13:46.7124483Z ==4796==    by 0x92FB47A: as_node_create_socket (as_node.c:482)
2026-06-26T22:13:46.7125362Z ==4796==    by 0x92FD0B6: as_node_create_connection (as_node.c:527)
2026-06-26T22:13:46.7126213Z ==4796==    by 0x92FD552: as_node_get_connection (as_node.c:687)
2026-06-26T22:13:46.7127387Z ==4796==    by 0x92DBAC5: as_command_execute (as_command.c:742)
2026-06-26T22:13:46.7128440Z ==4796==    by 0x92DC63B: as_command_send (as_command.c:662)
2026-06-26T22:13:46.7129088Z ==4796== 

Valgrind in dev branch (baseline) doesn't show this memory error: https://github.com/aerospike/aerospike-client-python/actions/runs/28343357687/job/83962184787

C client change from dev..HEAD involving SSL/TLS: aerospike/aerospike-client-c@49e7042...e844942#diff-c3b743a146519d28f7c3c6dde3fb464fde3deeac8ada5ab1901f437da06ec032

…to align with PRD and be consistent with the other expressions.
…incorporating the send_key union changes in this PR, so need to cherrypick the test changes to here. Also use C client memory leak fix instead of the workaround in the python client code
@juliannguyen4

Copy link
Copy Markdown
Collaborator Author

Noticed that memory usage curve changed:

    MB
99.58^                                                                   #    
     |                                                 @                 # :: 
     |                                                 @          @:     #::::
     |                                                 @      ::::@::::::#::::
     |                                                 @      ::::@::::::#::::
     |                                                 @   :::::::@::::::#::::
     |                                       @@        @::::: ::::@::::::#::::
     |                                       @      :::@: ::: ::::@::::::#::::
     |                                      :@ ::::::::@: ::: ::::@::::::#::::
     |                           @:::::::::::@ :::: :::@: ::: ::::@::::::#::::
     |      :::@::::::@@:::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     |     :: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     |     :: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     |  ::::: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     |  :: :: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     |  :: :: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     | @:: :: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     | @:: :: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     | @:: :: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
     | @:: :: :@: ::::@ :::::::@:@:: ::::: ::@ :::: :::@: ::: ::::@::::::#::::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   228.0

…gative and happy path code paths together to prevent this from happening
@juliannguyen4
juliannguyen4 force-pushed the CLIENT-4836-add-string-ops-and-expressions branch from 02a75d5 to 4cf37d1 Compare June 30, 2026 16:40
@juliannguyen4

Copy link
Copy Markdown
Collaborator Author
    MB
99.72^                                                                    #   
     |                                                  @                 #:: 
     |                                                  @       :  @      #:::
     |                                                  @    @:::::@:::::@#:::
     |                                                  @    @:::::@:::::@#:::
     |                                                  @  ::@:::::@:::::@#:::
     |                              @                   @::::@:::::@:::::@#:::
     |                              @       ::::    ::::@::::@:::::@:::::@#:::
     |                              @       : :::::@::: @::::@:::::@:::::@#:::
     |   @                       :::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     |   @   ::::::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     |   @:::::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     |   @:: ::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     |   @:: ::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     |  :@:: ::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     |  :@:: ::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     | @:@:: ::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     | @:@:: ::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     | @:@:: ::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
     | @:@:: ::: ::::::::::::::@::::@:::::::: :::::@::: @::::@:::::@:::::@#:::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   226.1

@juliannguyen4

juliannguyen4 commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator Author

Culprit test - test_expression_write

2026-06-30T19:46:42.9749925Z ==4630== 494 (224 direct, 270 indirect) bytes in 4 blocks are definitely lost in loss record 3,089 of 3,482
2026-06-30T19:46:42.9750252Z ==4630==    at 0x4848899: malloc (in /usr/libexec/valgrind/vgpreload_memcheck-amd64-linux.so)
2026-06-30T19:46:42.9750475Z ==4630==    by 0x9317A93: as_arraylist_new (as_arraylist.c:57)
2026-06-30T19:46:42.9750698Z ==4630==    by 0x927EC11: pyobject_to_list (conversions.c:745)
2026-06-30T19:46:42.9750955Z ==4630==    by 0x927EA2A: as_val_new_from_pyobject (conversions.c:1431)
2026-06-30T19:46:42.9751179Z ==4630==    by 0x92681A4: get_asval (cdt_operation_utils.c:103)
2026-06-30T19:46:42.9751423Z ==4630==    by 0x92914A7: add_expr_macros (convert_expressions.c:2010)
2026-06-30T19:46:42.9751792Z ==4630==    by 0x9283A4E: as_exp_new_from_pyobject (convert_expressions.c:2335)
2026-06-30T19:46:42.9752058Z ==4630==    by 0x9269A07: add_op_expr_write (expression_operations.c:90)
2026-06-30T19:46:42.9752298Z ==4630==    by 0x9269A07: add_new_expr_op (expression_operations.c:56)
2026-06-30T19:46:42.9752500Z ==4630==    by 0x9270CA5: add_op (operate.c:391)
2026-06-30T19:46:42.9752750Z ==4630==    by 0x9272B40: AerospikeClient_Operate_Invoke (operate.c:974)
2026-06-30T19:46:42.9752987Z ==4630==    by 0x9273593: AerospikeClient_Operate (operate.c:1067)
2026-06-30T19:46:42.9753274Z ==4630==    by 0x49B2A02: method_vectorcall_VARARGS_KEYWORDS (descrobject.c:344)

…ct heap allocated values passed to the Val expression, but need to confirm this
…stead of freeing every as_exp_entry that represents a value to prevent double free
@juliannguyen4

Copy link
Copy Markdown
Collaborator Author
    MB
99.65^                                                                    #   
     |                                                 @                 @#   
     |                                                 @       :: :      @#:::
     |                                                 @    :::: ::::::::@#:::
     |                                                 @   :: :: ::: ::::@#:::
     |                                                 @   :: :: ::: ::::@#:::
     |                                        @@       @::::: :: ::: ::::@#:::
     |                                        @     :::@: ::: :: ::: ::::@#:::
     |               @@                      :@ :::@:: @: ::: :: ::: ::::@#:::
     |               @            @:::::::::::@ : :@:: @: ::: :: ::: ::::@#:::
     |       ::::::::@ :::::::::::@:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |   : ::: ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |   ::: : ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |   ::: : ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |  :::: : ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |  :::: : ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |:::::: : ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |: :::: : ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |: :::: : ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
     |: :::: : ::::: @ ::::: :: : @:: : : :: :@ : :@:: @: ::: :: ::: ::::@#:::
   0 +----------------------------------------------------------------------->Gi
     0                                                                   227.8

@juliannguyen4

juliannguyen4 commented Jun 30, 2026

Copy link
Copy Markdown
Collaborator Author
  • Build artifacts passes
  • Valgrind shows no memory errors or leaks from these changes
  • Massif graph looks ok. There's a slight memory usage increase but it's not that much.

@juliannguyen4
juliannguyen4 marked this pull request as ready for review June 30, 2026 23:41
@juliannguyen4
juliannguyen4 merged commit 5c6294d into dev Jul 1, 2026
137 of 138 checks passed
@juliannguyen4
juliannguyen4 deleted the CLIENT-4836-add-string-ops-and-expressions branch July 1, 2026 16:34
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants