Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

DAT-2194; Substreams SQL; Uniswap v2; Add DBT models #154

Open
wants to merge 6 commits into
base: master
Choose a base branch
from
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions .gitignore
Original file line number Diff line number Diff line change
Expand Up @@ -9,6 +9,8 @@ Reference/
**/target/
**/build/
**/data/
**/logs/dbt.log
**/logs/
**/*.spkg
**/buf.gen.yaml
replay.log
Expand Down
131 changes: 112 additions & 19 deletions Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

1 change: 1 addition & 0 deletions Cargo.toml
Original file line number Diff line number Diff line change
Expand Up @@ -15,6 +15,7 @@ members = [
"eth-supply",
"synthetix",
"aave-v2",
"sql/uniswap_v2",
]
exclude = ["messari-cli"]

Expand Down
4 changes: 4 additions & 0 deletions sql/dbt/.gitignore
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

target/
dbt_packages/
logs/
15 changes: 15 additions & 0 deletions sql/dbt/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
Welcome to your new dbt project!

### Using the starter project

Try running the following commands:
- dbt run
- dbt test


### Resources:
- Learn more about dbt [in the docs](https://docs.getdbt.com/docs/introduction)
- Check out [Discourse](https://discourse.getdbt.com/) for commonly asked questions and answers
- Join the [chat](https://community.getdbt.com/) on Slack for live discussions and support
- Find [dbt events](https://events.getdbt.com) near you
- Check out [the blog](https://blog.getdbt.com/) for the latest news on dbt's development and best practices
Empty file added sql/dbt/analyses/.gitkeep
Empty file.
37 changes: 37 additions & 0 deletions sql/dbt/dbt_project.yml
Original file line number Diff line number Diff line change
@@ -0,0 +1,37 @@

# Name your project! Project names should contain only lowercase characters
# and underscores. A good package name should reflect your organization's
# name or the intended use of these models
name: 'messari_substreams'
version: '1.0.0'
config-version: 2

# This setting configures which "profile" dbt uses for this project.
profile: 'messari_substreams'

# These configurations specify where dbt should look for different types of files.
# The `model-paths` config, for example, states that models in this project can be
# found in the "models/" directory. You probably won't need to change these!
model-paths: ["models"]
analysis-paths: ["analyses"]
test-paths: ["tests"]
seed-paths: ["seeds"]
macro-paths: ["macros"]
snapshot-paths: ["snapshots"]

clean-targets: # directories to be removed by `dbt clean`
- "target"
- "dbt_packages"


# Configuring models
# Full documentation: https://docs.getdbt.com/docs/configuring-models

# In this example config, we tell dbt to build all models in the example/
# directory as views. These settings can be overridden in the individual model
# files using the `{{ config(...) }}` macro.
models:
messari_substreams:
# Config indicated by + and applies to all files under models/example/
example:
+materialized: view
Empty file added sql/dbt/macros/.gitkeep
Empty file.
Original file line number Diff line number Diff line change
@@ -0,0 +1,48 @@
{{
config(
materialized = 'view',
alias = 'COMPILED_METRICS_DELTAS_LIQUIDITY_POOL_INPUT_TOKEN_BALANCES',
unique_key='id',
)
}}

{% set deltas_input_token_balances = ref('analytics__protocol_uniswap_v2_ethereum_mainnet__entity_modifications_deltas_input_token_balances') %}

WITH
incremental_deltas AS (SELECT * FROM {{ deltas_input_token_balances }})

, final AS (
SELECT
id.id
, id.pool__id
, id.token__id
, DATE_TRUNC('hour', id.block_timestamp) AS hour

-- Delta Calculations
, id.delta
, COALESCE(id.delta / POWER(10, token.decimals) * token.price_usd, 0) AS delta_usd

-- Cumulative Sum Delta Calculations
, SUM(id.delta) OVER (PARTITION BY id.pool__id, id.token__id ORDER BY id.block_number, id.transaction_index, id.log_index) AS cumulative_sum

-- Cumulative Sum Delta Calculations (Current Timestamp Price)
, COALESCE(cumulative_sum / POWER(10, token.decimals) * token.price_usd, 0) AS cumulative_sum_usd_current_timestamp_price

, id.transaction_hash
, id.block_number
, id.block_timestamp
, id.transaction_index
, id.log_index
FROM incremental_deltas id
LEFT JOIN {{ token_prices_usd_hour }} token
ON id.token__id = token.contract_address
AND DATE_TRUNC('hour', id.block_timestamp) = token.time
ORDER BY
pool__id
, token__id
, block_number
, transaction_index
, log_index
)

SELECT * FROM final
Loading
Loading