Skip to content

Commit 947de0f

Browse files
JakeOShannessyLatrasis
authored andcommitted
Validation (#151)
* Change nightly version. The previous nightly version specified (nightly-2019-03-05) had an issue where components where not supported on Windows. 2018 versions had an issue with some wasm memory functions not being finalised (see rust-lang/rust#56292). This could all be wrong but I found this version worked for me where others didn't. * Add batch equivalent to build.sh * Update windows build command * Rough validation testing This tests the proofs-of-concept for the validation code. It is not currently integrated into the test suites or CI. * Fix contract name * Add cap9-build to link in sycalls * Validate syscalls. * Functioning syscall linking and validation (albeit very messy) * Change toolchain version * Use cross platform CWD. * Use no_std in native validator * Clean up cap9-build logging * Add toolchain to native-validator * Move validation code into separate lib * Switch to HTTP transport for tests This is what I'd used previously and it had always worked well for me. * Experiment with circleci * Remove SolC * Only build ewasm * Change parity command * Fix directory * Build wasm-build 0.6.0 * Fix parity url * Fix directory to run parity * Fix location of cargo.lock * Fix parity path * Don't need to kill database * Fix Cargo.lock path * Remove reference to Cargo.lock, which is not commited * Correct specification of dev dependencies * Fix no_std build * Fix std build * Add get_code_size to the kernel * Use custom parity node in CI * Fix installation of parity * circleci: install cmake * circleci: use the right package manager * circleci: needed privileges * circleci: add more dependencies * circleci: remove gflags * circleci: try different gflags package name * circleci: add perl and yasm * circleci: see if we can use parity's image * circleci: remove sudo * circleci: remove install line * circleci: try building parity's branch * Copy example from parity's docker image * circleci: actually clone parity * circleci: switch to stable parity * circleci: build syn first * circleci: build syn from git * circleci: dev build * circleci: build 1 package at a time * circleci: save cache after parity build * Revert "circleci: save cache after parity build" This reverts commit df48168. * circleci: save cache after parity build * circleci: checkout code * circleci: restore cache * circleci: cache parity builds * circleci: version bump deps * circleci: install parity stage * circleci: don't update rust * circleci: set default toolchain * circleci: reorder rustup * circleci: check for the existence of cargo * circleci: only install not build then install * circleci: install this package * circleci: formatting * Don't update rust * circleci: cache rustup * circleci: update PATH * circleci: checkout code * circleci: fix .profile * circleci: force overwrite of parity node * circleci: wrong line * circleci: add log line * circleci: more log lines * circleci: fix cache path * circleci: properly namespace directories * circleci: fix yaml syntax error * circleci: install nodejs * circleci: add repo for nodejs * circleci: add missing -y * circleci: install newer version of node * circleci: use .xz * circleci: switch parity to master * Build custom parity as part of ci (#152) * circleci: add more dependencies * circleci: remove gflags * circleci: try different gflags package name * circleci: add perl and yasm * circleci: see if we can use parity's image * circleci: remove sudo * circleci: remove install line * circleci: try building parity's branch * Copy example from parity's docker image * circleci: actually clone parity * circleci: switch to stable parity * circleci: build syn first * circleci: build syn from git * circleci: dev build * circleci: build 1 package at a time * circleci: save cache after parity build * Revert "circleci: save cache after parity build" This reverts commit df48168. * circleci: save cache after parity build * circleci: checkout code * circleci: restore cache * circleci: cache parity builds * circleci: version bump deps * circleci: install parity stage * circleci: don't update rust * circleci: set default toolchain * circleci: reorder rustup * circleci: check for the existence of cargo * circleci: only install not build then install * circleci: install this package * circleci: formatting * Don't update rust * circleci: cache rustup * circleci: update PATH * circleci: checkout code * circleci: fix .profile * circleci: force overwrite of parity node * circleci: wrong line * circleci: add log line * circleci: more log lines * circleci: fix cache path * circleci: properly namespace directories * circleci: fix yaml syntax error * circleci: install nodejs * circleci: add repo for nodejs * circleci: add missing -y * circleci: install newer version of node * circleci: use .xz * circleci: switch parity to master * validation: update whitelist and reorder to match parity. * validation: add EXTCODECOPY and simple test * Validation merge (#154) * circleci: add more dependencies * circleci: remove gflags * circleci: try different gflags package name * circleci: add perl and yasm * circleci: see if we can use parity's image * circleci: remove sudo * circleci: remove install line * circleci: try building parity's branch * Copy example from parity's docker image * circleci: actually clone parity * circleci: switch to stable parity * circleci: build syn first * circleci: build syn from git * circleci: dev build * circleci: build 1 package at a time * circleci: save cache after parity build * Revert "circleci: save cache after parity build" This reverts commit df48168. * circleci: save cache after parity build * circleci: checkout code * circleci: restore cache * circleci: cache parity builds * circleci: version bump deps * circleci: install parity stage * circleci: don't update rust * circleci: set default toolchain * circleci: reorder rustup * circleci: check for the existence of cargo * circleci: only install not build then install * circleci: install this package * circleci: formatting * Don't update rust * circleci: cache rustup * circleci: update PATH * circleci: checkout code * circleci: fix .profile * circleci: force overwrite of parity node * circleci: wrong line * circleci: add log line * circleci: more log lines * circleci: fix cache path * circleci: properly namespace directories * circleci: fix yaml syntax error * circleci: install nodejs * circleci: add repo for nodejs * circleci: add missing -y * circleci: install newer version of node * circleci: use .xz * circleci: switch parity to master * validation: update whitelist and reorder to match parity. * validation: add EXTCODECOPY and simple test * cap9-build: increase the amount of available memory in kernel * kernel: link in basic validation code for testing * example_contact: expand * cap9-build: properly pass the number of memory pages * kernel: LTO needs to be turned off for mem access bug workaround * kernel: unlock more tests * kernel: switch to git version of parity-wasm * kernel: use wasm module type exposed by validator * kernel: set number of memory pages to 3 * kernel: panic on inability to parse wasm * kernel: test simple procedure contract * Switch to manual parsing for validation (#157) * wasm-parser: remove parity-wasm dependency * wasm-parser: proof of basic idea * wasm-parser: import instruction mappings from parity-wasm * wasm-parser: build and link with kernel * wasm-parser: fix parsing of import entries * wasm-parser: add missing file * kernel: add test script * wasm-parser: minor cleanup * circleci: don't force rebuild of parity-ethereum * kernel: update tests to be more accurate * kernel: update nightly version for alloc crate * circleci: fix error with parity installation * circleci: fix error in previous commit * wasm-parser: reinclude alloc * wasm-parser: merge Cursor and CodeCursor * circleci: add example contract 2 * circleci: set-up environment for test * circleci: fix example_contract_2 build * wasm-parser: properly validate instructions in syscall * wasm-parser: fix import cursor progression * circleci: remove config step from parity * circleci: clear cache * remove and gitignore build files * validation: rustfmt and add tests * wasm-parse: refactor * wasm-parser: docs * validation: forbid indirect calls * validation: correct invalidation of bad dcall * wasm-parser: minor cleanup * kernel: skip unimplemented test * wasm-parser: check for varuint32 issues * wasm-parser: additional comments * whitespace
1 parent 33b8398 commit 947de0f

Some content is hidden

Large Commits have some content hidden by default. Use the searchbox below for content that may be hidden.

49 files changed

+5337
-256
lines changed

.circleci/config.yml

+135-34
Original file line numberDiff line numberDiff line change
@@ -1,44 +1,145 @@
1-
# Javascript Node CircleCI 2.0 configuration file
2-
#
3-
# Check https://circleci.com/docs/2.0/language-javascript/ for more details
4-
#
51
version: 2
62
jobs:
7-
build:
8-
docker:
9-
# specify the version you desire here
10-
- image: circleci/node:9
11-
12-
# Specify service dependencies here if necessary
13-
# CircleCI maintains a library of pre-built images
14-
# documented at https://circleci.com/docs/2.0/circleci-images/
15-
# - image: circleci/mongo:3.4.4
3+
# build-evm:
4+
# docker:
5+
# # specify the version you desire here
6+
# - image: circleci/node:9
167

17-
working_directory: ~/repo
8+
# # Specify service dependencies here if necessary
9+
# # CircleCI maintains a library of pre-built images
10+
# # documented at https://circleci.com/docs/2.0/circleci-images/
11+
# # - image: circleci/mongo:3.4.4
1812

19-
steps:
20-
- checkout
13+
# working_directory: ~/repo
2114

22-
# Download and cache dependencies
23-
- restore_cache:
24-
keys:
25-
- v1-dependencies-{{ checksum "evm1/package.json" }}
26-
# fallback to using the latest cache if no exact match is found
27-
- v1-dependencies-
15+
# steps:
16+
# - checkout
17+
18+
# # Download and cache dependencies
19+
# - restore_cache:
20+
# keys:
21+
# - v1-dependencies-{{ checksum "evm1/package.json" }}
22+
# # fallback to using the latest cache if no exact match is found
23+
# - v1-dependencies-
24+
25+
# - run: cd evm1 && npm install
26+
# - run: cd evm1 && chmod +x ./scripts/test.sh
2827

29-
- run: cd evm1 && npm install
30-
- run: cd evm1 && chmod +x ./scripts/test.sh
28+
# - save_cache:
29+
# paths:
30+
# - evm1/node_modules
31+
# key: v1-dependencies-{{ checksum "evm1/package.json" }}
3132

33+
# # run tests!
34+
# - run:
35+
# name: Running Local EVM Instance
36+
# command: ./evm1/node_modules/.bin/ganache-cli
37+
# background: true
38+
# - run: cd evm1 && ./node_modules/.bin/truffle test
39+
build:
40+
docker:
41+
- image: centos:latest
42+
working_directory: /tmp/my-project
43+
steps:
44+
- restore_cache:
45+
keys:
46+
- deps7-{{ .Branch }}-{{ .Revision }}
47+
# - deps7-{{ .Branch }}-cargo-{{ checksum "kernel-ewasm/Cargo.lock" }}
48+
- deps7-{{ .Branch }}-
49+
- deps7-
50+
- run:
51+
name: Install native build prequisites
52+
command: |
53+
yum -y update
54+
yum install -y systemd-devel git make gcc-c++ gcc file binutils
55+
curl -L "https://cmake.org/files/v3.12/cmake-3.12.0-Linux-x86_64.tar.gz" -o cmake.tar.gz
56+
tar -xzf cmake.tar.gz
57+
cp -r cmake-3.12.0-Linux-x86_64/* /usr/
58+
echo "PATH=/root/.cargo/bin:$PATH" >> ~/.profile
59+
# only update cargo if it is not installed
60+
if [ ! -d /root/.rustup ]
61+
then
62+
echo "Installing rustup"
63+
curl https://sh.rustup.rs -sSf | sh -s -- -y
64+
. ~/.profile
65+
rustup default stable
66+
else
67+
echo "rustup already installed"
68+
. ~/.profile
69+
fi
70+
RUST_BACKTRACE=1
71+
rustc -vV
72+
cargo -V
73+
gcc -v
74+
g++ -v
75+
cmake --version
76+
curl -L https://nodejs.org/dist/v12.3.1/node-v12.3.1-linux-x64.tar.xz -o node.tar.xz
77+
tar -xJf node.tar.xz
78+
cp -r node-v12.3.1-linux-x64/* /usr/
79+
- run:
80+
name: Install Parity
81+
command: |
82+
. ~/.profile
83+
cd ..
84+
# If the parity-ethereum directory does not exist, clone it
85+
pwd
86+
ls
87+
if [ ! -d parity-ethereum ]
88+
then
89+
echo "Parity not installed, cloning..."
90+
git clone https://github.com/Daohub-io/parity-ethereum.git
91+
fi
92+
cd parity-ethereum
93+
git fetch --all
94+
git checkout master
95+
# cargo build -j 1
96+
# cargo build --verbose --release --features final
97+
# strip target/debug/parity
98+
# file target/debug/parity
99+
if parity --version; then
100+
echo "Parity node installed"
101+
else
102+
cargo install --bin parity -j 1 --path . --bin parity parity-ethereum
103+
fi
32104
- save_cache:
105+
key: deps7-{{ .Branch }}-cargo #-{{ checksum "kernel-ewasm/Cargo.lock" }}
33106
paths:
34-
- evm1/node_modules
35-
key: v1-dependencies-{{ checksum "evm1/package.json" }}
36-
37-
# run tests!
38-
- run:
39-
name: Running Local EVM Instance
40-
command: ./evm1/node_modules/.bin/ganache-cli
107+
- "~/.cargo"
108+
- "~/.rustup"
109+
- /tmp/parity-ethereum
110+
- checkout:
111+
path: cap9
112+
- run:
113+
name: Get Submodules
114+
command: |
115+
cd cap9
116+
git submodule update --init
117+
- run:
118+
name: Start local Ethereum network
119+
command: |
120+
. ~/.profile
121+
cd cap9
122+
cd kernel-ewasm
123+
# we need to run parity once to set up the accounts and keys
124+
# this only needs to be active for a few seconds (hence timeout)
125+
# timeout 5 parity --config dev || true
126+
# We then run parity properly, now unlocking the previously setup
127+
# account
128+
parity --config dev --chain ./wasm-dev-chain.json --jsonrpc-apis=all --ws-apis=all --reseal-min-period 0 --gasprice 0
41129
background: true
42-
- run: cd evm1 && ./node_modules/.bin/truffle test
43-
44-
130+
- run:
131+
name: Wait for Parity startup
132+
command: sleep 10
133+
- run:
134+
name: Build Rust Component
135+
command: |
136+
. ~/.profile
137+
cd cap9
138+
cd kernel-ewasm && ./build.sh
139+
- run:
140+
name: Test Rust Component
141+
command: |
142+
. ~/.profile
143+
cd cap9
144+
cd kernel-ewasm && npm install
145+
npm run test

cap9-build/.gitignore

+1
Original file line numberDiff line numberDiff line change
@@ -0,0 +1 @@
1+
target

cap9-build/Cargo.toml

+10
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,10 @@
1+
[package]
2+
name = "cap9-build"
3+
version = "0.1.0"
4+
authors = ["Jake O'Shannessy <[email protected]>"]
5+
edition = "2018"
6+
7+
[dependencies]
8+
parity-wasm = "0.31"
9+
pwasm-utils = "0.7.0"
10+
clap = "~2.32.0"

cap9-build/src/main.rs

+192
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,192 @@
1+
extern crate parity_wasm;
2+
extern crate pwasm_utils;
3+
4+
use parity_wasm::elements::{Module,MemoryType};
5+
use clap::{Arg, App, SubCommand, ArgMatches};
6+
use parity_wasm::elements::{Instructions, Instruction};
7+
8+
fn main() {
9+
let matches = App::new("cap9-build")
10+
.version("0.2.0")
11+
.author("Cap9 <[email protected]>")
12+
.about("A command-line interface for linking Cap9 procedures.")
13+
.subcommand(SubCommand::with_name("build-proc")
14+
.about("Convert a regular contract into a cap9 procedure.")
15+
.arg(Arg::with_name("INPUT-FILE")
16+
.required(true)
17+
.help("input file"))
18+
.arg(Arg::with_name("OUTPUT-FILE")
19+
.required(true)
20+
.help("output file")))
21+
.subcommand(SubCommand::with_name("set-mem")
22+
.about("Set the number of memory pages in a procedure.")
23+
.arg(Arg::with_name("INPUT-FILE")
24+
.required(true)
25+
.help("input file"))
26+
.arg(Arg::with_name("OUTPUT-FILE")
27+
.required(true)
28+
.help("output file"))
29+
.arg(Arg::with_name("pages")
30+
.short("p")
31+
.long("pages")
32+
.value_name("PAGES")
33+
.required(true)
34+
.help("Number of pages to set the memory to")))
35+
.get_matches();
36+
37+
match matches.subcommand() {
38+
("build-proc", Some(opts)) => {
39+
let input_path = opts.value_of("INPUT-FILE").expect("input file is required");
40+
let output_path = opts.value_of("OUTPUT-FILE").expect("output path is required");
41+
42+
let module = parity_wasm::deserialize_file(input_path).expect("parsing of input failed");
43+
let new_module = contract_build(module);
44+
parity_wasm::serialize_to_file(output_path, new_module).expect("serialising to output failed");
45+
},
46+
("set-mem", Some(opts)) => {
47+
let input_path = opts.value_of("INPUT-FILE").expect("input file is required");
48+
let output_path = opts.value_of("OUTPUT-FILE").expect("output path is required");
49+
let mem_pages = opts.value_of("pages").expect("number of memory pages is required");
50+
51+
let module = parity_wasm::deserialize_file(input_path).expect("parsing of input failed");
52+
let new_module = set_mem(module, mem_pages.parse().expect("expected number for number of pages"));
53+
parity_wasm::serialize_to_file(output_path, new_module).expect("serialising to output failed");
54+
},
55+
_ => panic!("unknown subcommand")
56+
}
57+
}
58+
59+
60+
/// Perform the operations necessary for cap9 procedures.
61+
fn contract_build(module: Module) -> Module {
62+
63+
// TODO: we need to make sure these values never change between now and when
64+
// we use them. In the current set up they will not, but it is fragile,
65+
// there are changes that could be introduced which would change this.
66+
let syscall_instructions_res = get_syscall_instructions(&module);
67+
68+
// TODO: what is the index of this newly added function?
69+
let mut new_module_builder = parity_wasm::builder::from_module(module);
70+
// Add the syscall function, if applicable.
71+
let mut new_module = if let Ok(syscall_instructions) = syscall_instructions_res {
72+
new_module_builder
73+
.function()
74+
.signature()
75+
.with_param(parity_wasm::elements::ValueType::I32)
76+
.with_param(parity_wasm::elements::ValueType::I32)
77+
.with_param(parity_wasm::elements::ValueType::I32)
78+
.with_param(parity_wasm::elements::ValueType::I32)
79+
.with_return_type(Some(parity_wasm::elements::ValueType::I32))
80+
.build()
81+
.body()
82+
.with_instructions(syscall_instructions)
83+
.build()
84+
.build()
85+
.build()
86+
} else {
87+
new_module_builder.build()
88+
};
89+
90+
// TODO: robustly determine the function index of the function we just
91+
// added. I think at this point it's simply the last funciton added, thereby
92+
// functions_space - 1, but this is not guaranteed anywhere.
93+
let added_syscall_index = new_module.functions_space() - 1;
94+
95+
// If we find cap9_syscall_low as an import, we need to replace all
96+
// references to it with a reference to this newly added function, and
97+
// remove the import. Once we replace the internal references and run optimize, it will be removed anyway.
98+
let cap9_syscall_low_index = find_import(&new_module, "env", "cap9_syscall_low");
99+
match cap9_syscall_low_index {
100+
None => (),
101+
Some(syscall_index) => {
102+
// Search though the code of each function, if we encounter a
103+
// Call(syscall_index), replace it with Call(added_syscall_index).
104+
// TODO: investigate the use of CallIndirect
105+
for f in new_module.code_section_mut().unwrap().bodies_mut().iter_mut() {
106+
for i in 0..f.code().elements().len() {
107+
let instruction = &f.code().elements()[i];
108+
if instruction == &Instruction::Call(syscall_index) {
109+
f.code_mut().elements_mut()[i] = Instruction::Call(added_syscall_index as u32);
110+
}
111+
}
112+
}
113+
}
114+
}
115+
116+
// Next we want to delete dummy_syscall if it exists. First we find it among
117+
// the exports (if it doesn't exist we don't need to do anything). We take
118+
// the reference of the export (i.e. the function it exports) and delete
119+
// both that function and the export. One way to do this would be to delete
120+
// the export and run the parity's optimizer again.
121+
// 1. Get the index of the export
122+
if let Some(dummy_syscall_export_index) = find_export(&new_module, "dummy_syscall") {
123+
// println!("dummy_syscall_export_index: {}", dummy_syscall_export_index);
124+
// 2. Delete the export
125+
new_module.export_section_mut().unwrap().entries_mut().remove(dummy_syscall_export_index as usize);
126+
}
127+
// 3. At this stage the dummy_syscall function still exists internally. We
128+
// can't use the same remove procedure without screwing up the internal
129+
// references, so we will just run the parity optmizer again for now to
130+
// let it deal with that.
131+
pwasm_utils::optimize(&mut new_module, vec!["call","deploy"]).unwrap();
132+
new_module
133+
}
134+
135+
fn set_mem(mut module: Module, num_pages: u32) -> Module {
136+
// We want to find the single memory section, and change it from its current
137+
// value to the one we've requested.
138+
let mut mem_entry: &mut Vec<MemoryType> = module.memory_section_mut().unwrap().entries_mut();
139+
mem_entry[0] = parity_wasm::elements::MemoryType::new(num_pages,None);
140+
module
141+
}
142+
143+
// Find the function index of an import
144+
fn find_import(module: &Module, mod_name: &str, field_name: &str) -> Option<u32> {
145+
let imports = module.import_section().unwrap().entries();
146+
for (i,import) in imports.iter().enumerate() {
147+
if import.module() == mod_name && import.field() == field_name {
148+
return Some(i as u32);
149+
}
150+
}
151+
return None;
152+
}
153+
154+
// Find the function index of an export
155+
fn find_export(module: &Module, field_name: &str) -> Option<u32> {
156+
let exports = module.export_section().unwrap().entries();
157+
for (i,export) in exports.iter().enumerate() {
158+
if export.field() == field_name {
159+
return Some(i as u32);
160+
}
161+
}
162+
return None;
163+
}
164+
165+
enum SysCallError {
166+
NoDCall,
167+
NoGasLeft,
168+
NoSender,
169+
}
170+
171+
fn get_syscall_instructions(module: &Module) -> Result<Instructions,SysCallError> {
172+
// If any of these three environments are not pulled in from the
173+
// environment, we cannot have syscalls.
174+
let dcall_index = find_import(module, "env", "dcall").ok_or(SysCallError::NoDCall)?;
175+
let gasleft_index = find_import(module, "env", "gasleft").ok_or(SysCallError::NoGasLeft)?;
176+
let sender_index = find_import(module, "env", "sender").ok_or(SysCallError::NoSender)?;
177+
let syscall_instructions = parity_wasm::elements::Instructions::new(vec![
178+
// Call gas
179+
Instruction::Call(gasleft_index),
180+
// Call sender
181+
Instruction::Call(sender_index),
182+
Instruction::GetLocal(0),
183+
Instruction::GetLocal(1),
184+
Instruction::GetLocal(2),
185+
Instruction::GetLocal(3),
186+
// Do the delegate call
187+
Instruction::Call(dcall_index),
188+
// End function
189+
Instruction::End,
190+
]);
191+
Ok(syscall_instructions)
192+
}

kernel-ewasm/.gitignore

+4-1
Original file line numberDiff line numberDiff line change
@@ -3,4 +3,7 @@ node_modules
33

44
#CARGO
55
target
6-
**/*.rs.bk
6+
**/*.rs.bk
7+
8+
#Built Contracts
9+
build/

0 commit comments

Comments
 (0)