Performance optimization

[mzueva]

• Introduce fast mode for sequence match
• Support XLSX file as assay data input

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly enhances the application's data processing capabilities by introducing a performance-optimized 'fast mode' for sequence alignment and expanding data input options to include XLSX files. It also provides users with greater control over computational resources and refines internal data handling for improved accuracy.

Highlights

• Performance Optimization (Fast Mode): Introduced a 'Fast mode' for sequence matching, which prioritizes speed over sensitivity by reducing prefiltering precision. This mode is configurable via a new UI checkbox and adjusts MMseqs2 parameters like comp-bias-corr, mask, exact-kmer-matching, and k-mer size.
• XLSX File Support: Added support for importing XLSX files as assay data input. A new Python-based software package xlsx-to-csv was integrated into the workflow to convert XLSX files to CSV format before further processing.
• Resource Allocation Configuration: Implemented UI controls and workflow logic to allow users to specify memory (GiB) and CPU (cores) for the alignment process, providing more granular control over resource utilization.
• Improved XSV Type Detection: Enhanced the UI's file import mechanism to detect the actual delimiter (CSV or TSV) from the file content, even if the file extension is misleading, ensuring correct parsing.
• Workflow Data Axis Correction: Corrected the mapping of 'query' and 'target' columns in the workflow's data import and aggregation steps to ensure accurate data processing and joining.

Changelog

• .changeset/hip-years-cut.md
  • Added a new changeset file to document the minor version bumps for several packages due to the introduction of fast mode and XLSX support.
• model/src/index.ts
  • Updated the BlockArgs interface to include new optional properties: detectedXsvType for storing the detected delimiter type, lessSensitive for enabling fast mode, and mem and cpu for resource allocation.
  • Initialized the lessSensitive property to false in the default model state.
• pnpm-lock.yaml
  • Added software/xlsx-to-csv to the devDependencies section for the software/xlsx-to-csv package.
  • Added software/xlsx-to-csv as a workspace dependency for the workflow package.
• pnpm-workspace.yaml
  • Included the new software/xlsx-to-csv package in the workspace configuration.
• software/coverage-mode-calc/src/main.py
  • Modified the initial coverage_mode value from '2' to '1'.
  • Swapped the conditional assignment of coverage_mode from '1' to '2' when assay_avg is less than clones_avg.
• software/xlsx-to-csv/package.json
  • Added a new package.json file for the xlsx-to-csv software, defining its name, version, scripts, dependencies, and block-software entrypoints for Python execution.
• software/xlsx-to-csv/src/main.py
  • Added a new Python script main.py that provides functionality to convert XLSX files to CSV format, including a find_header_row utility and command-line argument parsing.
• software/xlsx-to-csv/src/requirements.txt
  • Added openpyxl as a required Python dependency for the xlsx-to-csv software.
• ui/src/importFile.ts
  • Set app.model.args.detectedXsvType to undefined at the start of file import.
  • Added logic to detect the actual XSV delimiter (CSV or TSV) from the file content if the extension is 'csv' or 'tsv', storing the result in app.model.args.detectedXsvType.
• ui/src/pages/MainPage.vue
  • Imported new UI components: PlAccordionSection, PlCheckbox, and PlTooltip.
  • Updated the PlFileInput component to include 'xlsx' in the list of supported file extensions.
  • Added a new PlAccordionSection for 'Advanced settings' containing a PlCheckbox for 'Fast mode' with a tooltip, and PlNumberField components for 'Memory (GiB)' and 'CPU (cores)' resource allocation.
• workflow/package.json
  • Added @platforma-open/milaboratories.immune-assay-data.xlsx-to-csv as a dependency to the workflow package.
• workflow/src/build-outputs.tpl.tengo
  • Changed the column for the first axis in xsv.importFile from 'query' to 'target'.
  • Swapped the column values for the first and second axes in the xsv.importFile call for bestAlignmentTsv from 'query'/'target' to 'target'/'query'.
  • Modified the column used in the for loop for cols from 'target' to 'query'.
  • Changed the groupBy column from 'query' to 'target'.
  • Adjusted the groupBy column for assayDf from 'target' to 'query' and updated the rightOn join condition from 'target' to 'query'.
  • Modified the leftOn join condition for clonesDf from 'target' to 'query'.
• workflow/src/main.tpl.tengo
  • Imported the xlsxToCsvSw software.
  • Added a new runXlsxToCsv function to execute the xlsx-to-csv software, configuring its memory, CPU, and file arguments.
  • Implemented logic to use args.detectedXsvType if available to override the file extension for XSV type detection.
  • Added a conditional block to convert XLSX files to CSV using runXlsxToCsv before further processing.
  • Passed lessSensitive, mem, and cpu arguments from the workflow to the runMmseqs function.
• workflow/src/run-alignment.tpl.tengo
  • Introduced variables mem and cpu with default values, allowing them to be overridden by args.mem and args.cpu.
  • Updated the mmseqs execution builder to use the configurable mem and cpu values.
  • Reordered arguments for easy-search to assay.fasta then clones.fasta.
  • Added --threads and --max-seqs arguments to the mmseqs command, using the configurable cpu for threads.
  • Implemented a conditional block to apply specific mmseqs arguments (--comp-bias-corr 0, --mask 0, --exact-kmer-matching 1, -k 7) when lessSensitive (fast mode) is enabled.

Activity

• No human activity has been recorded on this pull request yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request introduces two main features: a 'fast mode' for sequence matching to improve performance, and support for XLSX files as assay data input. The changes look good overall. I've added a few comments:

• A suggestion to improve the memory efficiency of the new XLSX to CSV conversion script.
• A fix for the delimiter detection logic in the UI to make it more robust.
• A recommendation to improve code clarity in the Tengo workflow by using a more descriptive variable name to avoid confusion.
  Please take a look at the detailed comments.

_{Note: Security Review did not run due to the size of the PR.}

[gemini-code-assist]

The current delimiter detection logic firstLine.includes('\t') is not very robust. It will incorrectly identify a CSV file as a TSV if the first line contains a tab character within a field's value. A more reliable approach would be to count the occurrences of tabs and commas and choose the delimiter that appears more frequently.

Suggested change

	app.model.args.detectedXsvType = firstLine.includes('\t') ? 'tsv' : 'csv';
	app.model.args.detectedXsvType = (firstLine.match(/\t/g) || []).length > (firstLine.match(/,/g) || []).length ? 'tsv' : 'csv';

[gemini-code-assist]

Loading all rows into memory with list(ws.iter_rows(values_only=True)) can be inefficient for large XLSX files, potentially causing high memory consumption. This is likely why the workflow allocates 16GiB for this step. To improve performance and reduce memory usage, consider processing the file as a stream. You can iterate over ws.iter_rows() directly and write to the CSV row-by-row, after finding the header by inspecting the first few hundred rows.