[Codegen] Introduce a generic & adjustable C header parser in Rust (#…

…3065) * init codegen-v2 * play around with possible implementation * impl derive for GSeparator * add GSeparators with wrapped Vec<GSeparator> * impl ParseTree for GSeparator * add Driver and DriverUsed convenient structs * add read_amt function, use in ParseTree impl * add DerivationResult and Error type, continue impl * update existing code to DerivationResult changes * small cleanup * rename revert to rollback * implement basic tests for GSeparator * fix target slice in Driver impl * track test module * make a GStruct test in GType::derive * handle EOF in ParseTree * check for EOF in tests * init basic types for function parameters * impl ParseTree for GParamItemWithMarker * impl ParseTree for GMarker and GParamName * complete ParseTree impl for GParamItemWithMarker * expand tests, derive Eq and PartialEq for relevant types * implement ParseTree for GParamItemWithoutMarker, write unit tests * add Continuum and ContinuumNext structs * implement ParseTree for Continuum * introduce WithEof struct for checking EOF * add EitherOr struct * deprecate WithEof * make internal changes to Driver * update ParseTree impls to Driver changes * complete new Driver implementation * update unit tests * rename DriverScope to Driver * update scoping mechanism * return DriverFinalized from derive impl * add test for individual separator items * add test_eof * fix position when applying scope * rename various types and methods * move Reader types into reader module * move types into grammar module * rename ReaderToMerge to ReaderBranch * introduce GFuncParams type * add GNonAlphanumeric type * add wipe and ensure convenience functions * expand usage of ensure * use convenience functions in GFuncParams * check explicitly for brackets * complete function definition parsing * convert GMarker to enum, set explicit variants * expand tests for function decleration * reorder impls in grammar * cleanup * add comments * rename GType to GPrimitive * add additional type categories * handle parsing of pointers, add unit tests * handle struct parsing * test Unknown type variant * handle markers in function declaration * add test * cleanup warnings * start ParseTree impl for GHeaderInclude * complete ParseTree impl for GHeaderInclude * deprecate GParamItemWithMarker * rename GParamItemWithoutMarker to GParamItem * write simple parse_file test, add GNewline and GAnyLine to grammar * expand * implement ParseTree for GEndOfLine * introduce GHeaderFileItem * introduce GCommentLine and GCommentBlock types * handle comments in GHeaderFileItem * start with impl of handling of structs * expand tests for struct parsing * split tests into smaller files * implement ParseTree for GType, convert function params to GType * fix mutability indicator * handle struct declarations * implement GEnum * fix some warnings * get rid of remaining warnings * handle marker after return value * introduce GReturnValue * update unit test * handle marker for struct * handle #pragma * handle makers directly * reorder grammar module * return in parse_file * implement serde support * rename variant to g_variant * add ASCII only for now * ignore attribute for test_parse_file * handle 'define' attributes, write unit tests * track attributes tests * implement GMarkers type * introduce GMarkers into other G types * convert inner GMarkers field to tuple * remove unused imports * remove double comment * support typedef, add unit test * track typedef unit test module * test with const void TWData * handle typedef and define in GHeaderFileItem * rename GHeaderDefine to GDefine * check for more separators * be more precise with GDefine parsing * more characters tests for define * handle GDefine better * add must_ok and must_err convenience functions * convert must_ok and must_err to macros * add comment * add unit tests for GKeyword * convert From<String> to From<&str> for test cases * cleanup warnings * use must_ok! in type categories unit tests * migrate functions unit tests to macros too * check for separator handling in functions tests * additional tests in functions module * cleanup * rename var * update unit tests for typedef * deprecate some tests * use GKeyword for GTypeCategory internally * make use of GKeyword in GStruct * use GKeyword in GEnum impl * deprecate GTypeCategory::Unknown * fix warnings * rename GStruct to GStructName * migrate primitives tests to must_ok! * migrate other unit tests to must_ok! * update in mod.rs * poc for reading includes * formatting * complete parse_file and parse_dir impl * print unrecognized items in test * remove unused import * add Unrecognized variant for types, used to handle structs as params * explicitly check for GTypeCategory::Unrecognized * add parser for inline comments * start with ParseTree impl for GEnumDecl * adjust comma check for enums * additional unit tests for enum decls * handle enum decl as part of GHeaderFileItem * track tests * expand tests for enum declaration * handle HEX values in enum variant values * implement ParseTree for GStaticVar * add support for static variables * use macro for single character definitions * move macro to root * wipe comment line in enums * write test for number in static var * handle TW_EXPORT_ENUM marker * cleanup * add test for non-parameter functions * typo * rename GStructDecl to GStructInd * implement ParseTree for GStructDecl * add unit tests for GStructDecl * handle struct declarations in primary parser * handle 'extern' functions * track .devcontainer * update .devcontainer * add extension to devcontainer * add README * add minimal main() method * update README * add workflow for codegen-v2 tests * update workflow * add manifest path to cargo test * deprecate GParamName * deprecate GFuncName * removed example.include_manifest.json * GStructName wraps GKeyword * adjust GStruct and GEnum * deprecate wipe, just use optional * rename Continuum types to plural * move define_char! decl to grammar module * rename EitherOr to Either * update comment * remove unnecessary check on GKeyword derive impl * rename GEnum to GEnumName * add more tests for primitives * fix devcontainer * add copyright notice to each Rust file * track Cargo lock and ignore target
trustwallet · Apr 21, 2023 · 1005f72 · 1005f72
1 parent 45fe29a
commit 1005f72
Show file tree

Hide file tree

Showing 22 changed files with 3,279 additions and 0 deletions.
diff --git a/.devcontainer/Dockerfile b/.devcontainer/Dockerfile
@@ -0,0 +1,73 @@
+FROM mcr.microsoft.com/devcontainers/base:ubuntu-22.04
+
+ENV DEBIAN_FRONTEND=noninteractive
+
+# Install some basics
+RUN apt-get update \
+    && apt-get install -y \
+        wget \
+        curl \
+        git \
+        vim \
+        unzip \
+        xz-utils \
+        software-properties-common \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+# Add latest cmake
+SHELL ["/bin/bash", "-o", "pipefail", "-c"]
+RUN wget -O - https://apt.kitware.com/keys/kitware-archive-latest.asc | apt-key add - \
+    && apt-add-repository "deb https://apt.kitware.com/ubuntu/ $(lsb_release -sc) main"
+
+# Install required packages for dev
+RUN apt-get update \
+    && apt-get install -y \
+        build-essential \
+        libtool autoconf pkg-config \
+        ninja-build \
+        ruby-full \
+        clang-14 \
+        llvm-14 \
+        libc++-dev libc++abi-dev \
+        cmake \
+        libboost-all-dev \
+        ccache \
+        # Swift dependencies
+        binutils \
+        git \
+        gnupg2 \
+        libc6-dev \
+        libcurl4-openssl-dev \
+        libedit2 \
+        libgcc-9-dev \
+        libpython3.8 \
+        libsqlite3-0 \
+        libstdc++-9-dev \
+        libxml2-dev \
+        libz3-dev \
+        pkg-config \
+        tzdata \
+        unzip \
+        zlib1g-dev \
+    && apt-get clean && rm -rf /var/lib/apt/lists/*
+
+ENV CC=/usr/bin/clang-14
+ENV CXX=/usr/bin/clang++-14
+
+# Instal Swift
+RUN curl -sSL \
+	https://download.swift.org/swift-5.8-release/ubuntu2204/swift-5.8-RELEASE/swift-5.8-RELEASE-ubuntu22.04.tar.gz \
+	-o swift-5.8.tar.gz && \
+    mkdir -p swift-5.8 && \
+    tar -xzf swift-5.8.tar.gz -C swift-5.8 --strip-components=1 && \
+    mv swift-5.8 /usr/share/swift
+
+ENV PATH="/usr/share/swift/usr/bin:${PATH}"
+
+USER vscode
+
+# Install rust
+RUN curl https://sh.rustup.rs -sSf | sh -s -- --no-modify-path -y
+ENV PATH="/home/vscode/.cargo/bin:${PATH}"
+RUN cargo install --force cbindgen \
+    && rustup target add wasm32-unknown-emscripten
diff --git a/.devcontainer/devcontainer.json b/.devcontainer/devcontainer.json
@@ -0,0 +1,31 @@
+// For format details, see https://aka.ms/devcontainer.json. For config options, see the
+// README at: https://github.com/devcontainers/templates/tree/main/src/debian
+{
+	"name": "Debian",
+	// Use a Dockerfile or Docker Compose file. More info: https://containers.dev/guide/dockerfile
+	"build": {
+		"dockerfile": "Dockerfile"
+	},
+
+	// Features to add to the dev container. More info: https://containers.dev/features.
+	// "features": {},
+
+	// Use 'forwardPorts' to make a list of ports inside the container available locally.
+	// "forwardPorts": [],
+
+	// Configure tool-specific properties.
+	"customizations": {
+		"vscode": {
+			"extensions": [
+				"rust-lang.rust-analyzer",
+				"sswg.swift-lang",
+				"ms-vscode.cpptools",
+				"ms-vscode.cpptools-extension-pack",
+				"rebornix.Ruby"
+			]
+		}
+	},
+
+	// Uncomment to connect as root instead. More info: https://aka.ms/dev-containers-non-root.
+	"remoteUser": "vscode"
+}
diff --git a/.github/workflows/codegen-v2.yml b/.github/workflows/codegen-v2.yml
@@ -0,0 +1,26 @@
+name: Codegen-v2 Tests
+
+on:
+  push:
+    branches:
+      - master
+    paths:
+      - 'codegen-v2/**'
+  pull_request:
+    branches:
+      - master
+    paths:
+      - 'codegen-v2/**'
+
+jobs:
+  test:
+    runs-on: ubuntu-latest
+    steps:
+    - uses: actions/checkout@v2
+    - uses: actions-rs/toolchain@v1
+      with:
+        toolchain: stable
+    - uses: actions-rs/cargo@v1
+      with:
+        command: test
+        args: --manifest-path codegen-v2/Cargo.toml
diff --git a/.gitignore b/.gitignore
@@ -64,3 +64,6 @@ samples/cpp/*.cmake
 
 # built binary
 samples/cpp/sample
+
+# Rust target build
+**/target/
diff --git a/codegen-v2/Cargo.lock b/codegen-v2/Cargo.lock
diff --git a/codegen-v2/Cargo.toml b/codegen-v2/Cargo.toml
@@ -0,0 +1,18 @@
+[package]
+name = "codegen-v2"
+version = "0.1.0"
+edition = "2021"
+
+# See more keys and their definitions at https://doc.rust-lang.org/cargo/reference/manifest.html
+
+[lib]
+name = "libparser"
+path = "src/lib.rs"
+
+[[bin]]
+name = "parser"
+path = "src/main.rs"
+
+[dependencies]
+serde = { version = "1.0.159", features = ["derive"] }
+serde_json = "1.0.95"
diff --git a/codegen-v2/README.md b/codegen-v2/README.md
@@ -0,0 +1,23 @@
+# About
+
+This is a _work-in-progress_ parser meant to deprecate the existing Ruby parser
+in `codegen/`. This project will progess over multiple stages (PRs).
+
+- [x] Write a (minimal) parser with full coverage of the C headers in `include/` ([#3065](https://github.com/trustwallet/wallet-core/pull/3065)).
+- [ ] Implement a templating engine, write easy to read templates.
+	- [ ] Start with Swift codegen, replicate existing API (no breakage).
+	- [ ] Complete codegen for remaining languages.
+- [ ] Extend the parser to handle all valid C header syntax.
+	- [ ] Add explicit error types (right now it's just `Error::Todo`).
+	- [ ] Seperate parser logic from C header grammer (no refactoring required, just reorganization).
+- [ ] Update the entire building system.
+
+## Execution
+
+```bash
+$ cd codegen-v2
+$ cargo run
+```
+
+This command reads the C headers in `include/` and generates a
+`include_manifest.json` file.