Skip to content

Claires suggestions #2

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
clrcrl wants to merge 4 commits into
base: master
Choose a base branch
from
Open

Claires suggestions #2

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
4 commits
Select commit Hold shift + click to select a range
b973071
Add a gitignore
clrcrl Feb 14, 2020
0b3ddc6
Specify a project (database) for sources
clrcrl Feb 14, 2020
5535e15
Fix errant SQL
clrcrl Feb 14, 2020
10a7dd9
Use simpler incremental logic
clrcrl Feb 14, 2020
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .gitignore
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,4 @@

reddit_aita/target/
reddit_aita/dbt_modules/
reddit_aita/logs/
10 changes: 2 additions & 8 deletions reddit_aita/models/aita_comments.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,12 +6,6 @@ FROM {{ source('reddit_comments', '20*') }}
WHERE subreddit = 'AmItheAsshole'
AND _table_suffix > '19_'

{%- if is_incremental() -%}
{%- if execute -%}
Copy link
Owner

@fhoffa fhoffa Feb 14, 2020

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

So I'm using this weird complex logic because one of these queries is much better for BigQuery:

  • SELECT * FROM * WHERE _table_suffix < '2019_01'
  • SELECT * FROM * WHERE _table_suffix < (SELECT date FROM ...)

One is a constant and will prune how much data is read. The other one is variable, and BigQuery doesn't optimize it as well.

I can get that as a constant by using run_query(). I have another option, but it doesn't work now (dbt-labs/dbt-core#2136)

{%- set last_stamp_sql -%}SELECT FORMAT_TIMESTAMP('%y_%mX', MAX(ts)) maxsuffix FROM {{this}}{%- endset -%}
{%- set last_stamp_result = run_query(last_stamp_sql) -%}
AND _table_suffix > "{{last_stamp_result.rows[0].get('maxsuffix')[:-1]}}"
{# Somewhere '19_01' gets transformed to '1901', unless I add an 'X' to FORMAT_TIMESTAMP() and [:-1] later. #}
{% if is_incremental() -%}
AND _table_suffix > (SELECT FORMAT_TIMESTAMP('%y_%m', MAX(ts)) from {{ this }})
{%- endif -%}
{%- endif -%}

10 changes: 2 additions & 8 deletions reddit_aita/models/aita_posts.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,12 +10,6 @@ FROM {{ source('reddit_posts', '20*') }}
WHERE subreddit = 'AmItheAsshole'
AND _table_suffix > '19_'

{%- if is_incremental() -%}
{%- if execute -%}
{%- set last_stamp_sql -%}SELECT FORMAT_TIMESTAMP('%y_%mX', MAX(ts)) maxsuffix FROM {{this}}{%- endset -%}
{%- set last_stamp_result = run_query(last_stamp_sql) -%}
AND _table_suffix > "{{last_stamp_result.rows[0].get('maxsuffix')[:-1]}}"
{# Somewhere '19_01' gets transformed to '1901', unless I add an 'X' to FORMAT_TIMESTAMP() and [:-1] later. #}
{% if is_incremental() -%}
AND _table_suffix > (SELECT FORMAT_TIMESTAMP('%y_%m', MAX(ts)) from {{ this }})
{%- endif -%}
{%- endif -%}

6 changes: 3 additions & 3 deletions reddit_aita/models/aita_posts_gendered.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,7 +7,7 @@ WITH data AS (
, ARRAY_LENGTH(REGEXP_EXTRACT_ALL(CONCAT(selftext, title), r'(?i)\bthey\b')) theys
, ARRAY_LENGTH(REGEXP_EXTRACT_ALL(CONCAT(selftext, title), r'(?i)\bgirlfriend\b')) gfs
, ARRAY_LENGTH(REGEXP_EXTRACT_ALL(CONCAT(selftext, title), r'(?i)\bboyfriend\b')) bfs
FROM {{ref('aita_posts')}}
FROM {{ref('aita_posts')}}
WHERE link_flair_text IS NOT NULL
), gendered_data AS (
SELECT *
Expand All @@ -23,12 +23,12 @@ WITH data AS (
)


SELECT CASE link_flair_text
SELECT CASE link_flair_text
WHEN 'not the a-hole' THEN 'no asshole'
WHEN 'no a-holes here' THEN 'no asshole'
WHEN 'everyone sucks' THEN 'asshole'
WHEN 'asshole' THEN 'asshole'
END judgement
, *
FROM gendered_data
AND link_flair_text IS NOT NULL
WHERE link_flair_text IS NOT NULL
2 changes: 2 additions & 0 deletions reddit_aita/models/schema.yml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -3,9 +3,11 @@ version: 2

sources:
- name: reddit_comments
database: fh-bigquery
tables:
- name: '20*'
- name: reddit_posts
database: fh-bigquery
tables:
- name: '20*'

Expand Down