mirror of
https://github.com/olehomelchenko/econ250_2025.git
synced 2025-12-21 21:22:26 +00:00
init
This commit is contained in:
5
.gitignore
vendored
Normal file
5
.gitignore
vendored
Normal file
@@ -0,0 +1,5 @@
|
|||||||
|
|
||||||
|
target/
|
||||||
|
dbt_packages/
|
||||||
|
logs/
|
||||||
|
env/
|
||||||
7
.vscode/extensions.json
vendored
Normal file
7
.vscode/extensions.json
vendored
Normal file
@@ -0,0 +1,7 @@
|
|||||||
|
{
|
||||||
|
"recommendations": [
|
||||||
|
"innoverio.vscode-dbt-power-user",
|
||||||
|
"mhutchie.git-graph",
|
||||||
|
"ms-python.python"
|
||||||
|
]
|
||||||
|
}
|
||||||
113
README.md
Normal file
113
README.md
Normal file
@@ -0,0 +1,113 @@
|
|||||||
|
Welcome to your new dbt project!
|
||||||
|
|
||||||
|
# Installation
|
||||||
|
|
||||||
|
The following tutorial assumes you're already familiar with git and command line usage.
|
||||||
|
|
||||||
|
## Getting the code to your local machine
|
||||||
|
1. Fork this github repository into your local account
|
||||||
|
|
||||||
|
2. Copy it to your local machine: `git clone https://github.com/your_account_name/econ250_2025.git`
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## gcloud authentication
|
||||||
|
|
||||||
|
To run queries from your command line, you'll first need to install `gcloud` utility.
|
||||||
|
|
||||||
|
Follow the instructions here: https://cloud.google.com/sdk/docs/install. After installation you should have `gcloud` command available for running in the terminal.
|
||||||
|
|
||||||
|
Now, try to authenticate with your **kse email** using the following command:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
gcloud auth application-default login \
|
||||||
|
--scopes=https://www.googleapis.com/auth/bigquery,\
|
||||||
|
https://www.googleapis.com/auth/drive.readonly,\
|
||||||
|
https://www.googleapis.com/auth/iam.test
|
||||||
|
```
|
||||||
|
|
||||||
|
Now, when you run the following commands something similar should be response:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
$ gcloud auth list
|
||||||
|
|
||||||
|
Credentialed Accounts
|
||||||
|
ACTIVE ACCOUNT
|
||||||
|
* o_omelchenko@kse.org.ua
|
||||||
|
|
||||||
|
```
|
||||||
|
To set the active project, run the following:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
gcloud config set project econ250-2025
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
## venv and libraries
|
||||||
|
Prerequisites: having Python installed on your machine.
|
||||||
|
Following instructions are for Linux or WSL; if you'd like to run Windows - please refer to the documentation below.
|
||||||
|
|
||||||
|
```bash
|
||||||
|
|
||||||
|
# change directory to the one you just copied from github
|
||||||
|
cd econ250_2025
|
||||||
|
|
||||||
|
# create and activate venv
|
||||||
|
python3 -m venv env
|
||||||
|
source env/bin/activate
|
||||||
|
|
||||||
|
pip install -r requirements.txt
|
||||||
|
|
||||||
|
```
|
||||||
|
|
||||||
|
If everything is installed correctly, you should run the following commands successfully:
|
||||||
|
|
||||||
|
|
||||||
|
```
|
||||||
|
$ dbt --version
|
||||||
|
|
||||||
|
Core:
|
||||||
|
- installed: 1.9.3
|
||||||
|
- latest: 1.9.3 - Up to date!
|
||||||
|
|
||||||
|
Plugins:
|
||||||
|
- bigquery: 1.9.1 - Up to date!
|
||||||
|
```
|
||||||
|
|
||||||
|
|
||||||
|
For more detailed reference, refer to the official documentation here:
|
||||||
|
- https://docs.getdbt.com/docs/core/pip-install
|
||||||
|
- https://docs.getdbt.com/docs/core/connect-data-platform/bigquery-setup#local-oauth-gcloud-setup
|
||||||
|
|
||||||
|
## Adjusting the configuration
|
||||||
|
|
||||||
|
You'll need to specify your own dataset to save your models to. To do so, navigate to the `profiles.yml` in the root directory of the project, and replace `o_omelchenko` with your bigquery dataset name with which you have been working previously.
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
|
||||||
|
## Final check
|
||||||
|
|
||||||
|
Try running the following command:
|
||||||
|
- dbt run
|
||||||
|
|
||||||
|
If everything is set up well, you will see similar output:
|
||||||
|
|
||||||
|
```log
|
||||||
|
❯ dbt run
|
||||||
|
01:18:56 Running with dbt=1.9.3
|
||||||
|
01:18:57 Registered adapter: bigquery=1.9.1
|
||||||
|
01:18:57 Found 2 models, 4 data tests, 491 macros
|
||||||
|
01:18:57
|
||||||
|
01:18:57 Concurrency: 2 threads (target='dev')
|
||||||
|
01:18:57
|
||||||
|
01:19:00 1 of 2 START sql table model o_omelchenko.my_first_dbt_model ................... [RUN]
|
||||||
|
01:19:04 1 of 2 OK created sql table model o_omelchenko.my_first_dbt_model .............. [CREATE TABLE (2.0 rows, 0 processed) in 4.44s]
|
||||||
|
01:19:04 2 of 2 START sql view model o_omelchenko.my_second_dbt_model ................... [RUN]
|
||||||
|
01:19:06 2 of 2 OK created sql view model o_omelchenko.my_second_dbt_model .............. [CREATE VIEW (0 processed) in 2.13s]
|
||||||
|
01:19:06
|
||||||
|
01:19:06 Finished running 1 table model, 1 view model in 0 hours 0 minutes and 9.64 seconds (9.64s).
|
||||||
|
```
|
||||||
|
|
||||||
|
If you have any troubles with installation, please contact the course instructor (Oleh Omelchenko) in slack for assist.
|
||||||
|
|
||||||
0
analyses/.gitkeep
Normal file
0
analyses/.gitkeep
Normal file
36
dbt_project.yml
Normal file
36
dbt_project.yml
Normal file
@@ -0,0 +1,36 @@
|
|||||||
|
|
||||||
|
# Name your project! Project names should contain only lowercase characters
|
||||||
|
# and underscores. A good package name should reflect your organization's
|
||||||
|
# name or the intended use of these models
|
||||||
|
name: 'econ250_2025'
|
||||||
|
version: '1.0.0'
|
||||||
|
|
||||||
|
# This setting configures which "profile" dbt uses for this project.
|
||||||
|
profile: 'econ250_2025'
|
||||||
|
|
||||||
|
# These configurations specify where dbt should look for different types of files.
|
||||||
|
# The `model-paths` config, for example, states that models in this project can be
|
||||||
|
# found in the "models/" directory. You probably won't need to change these!
|
||||||
|
model-paths: ["models"]
|
||||||
|
analysis-paths: ["analyses"]
|
||||||
|
test-paths: ["tests"]
|
||||||
|
seed-paths: ["seeds"]
|
||||||
|
macro-paths: ["macros"]
|
||||||
|
snapshot-paths: ["snapshots"]
|
||||||
|
|
||||||
|
clean-targets: # directories to be removed by `dbt clean`
|
||||||
|
- "target"
|
||||||
|
- "dbt_packages"
|
||||||
|
|
||||||
|
|
||||||
|
# Configuring models
|
||||||
|
# Full documentation: https://docs.getdbt.com/docs/configuring-models
|
||||||
|
|
||||||
|
# In this example config, we tell dbt to build all models in the example/
|
||||||
|
# directory as views. These settings can be overridden in the individual model
|
||||||
|
# files using the `{{ config(...) }}` macro.
|
||||||
|
models:
|
||||||
|
econ250_2025:
|
||||||
|
# Config indicated by + and applies to all files under models/example/
|
||||||
|
example:
|
||||||
|
+materialized: view
|
||||||
0
macros/.gitkeep
Normal file
0
macros/.gitkeep
Normal file
27
models/example/my_first_dbt_model.sql
Normal file
27
models/example/my_first_dbt_model.sql
Normal file
@@ -0,0 +1,27 @@
|
|||||||
|
|
||||||
|
/*
|
||||||
|
Welcome to your first dbt model!
|
||||||
|
Did you know that you can also configure models directly within SQL files?
|
||||||
|
This will override configurations stated in dbt_project.yml
|
||||||
|
|
||||||
|
Try changing "table" to "view" below
|
||||||
|
*/
|
||||||
|
|
||||||
|
{{ config(materialized='table') }}
|
||||||
|
|
||||||
|
with source_data as (
|
||||||
|
|
||||||
|
select 1 as id
|
||||||
|
union all
|
||||||
|
select null as id
|
||||||
|
|
||||||
|
)
|
||||||
|
|
||||||
|
select *
|
||||||
|
from source_data
|
||||||
|
|
||||||
|
/*
|
||||||
|
Uncomment the line below to remove records with null `id` values
|
||||||
|
*/
|
||||||
|
|
||||||
|
-- where id is not null
|
||||||
6
models/example/my_second_dbt_model.sql
Normal file
6
models/example/my_second_dbt_model.sql
Normal file
@@ -0,0 +1,6 @@
|
|||||||
|
|
||||||
|
-- Use the `ref` function to select from other models
|
||||||
|
|
||||||
|
select *
|
||||||
|
from {{ ref('my_first_dbt_model') }}
|
||||||
|
where id = 1
|
||||||
21
models/example/schema.yml
Normal file
21
models/example/schema.yml
Normal file
@@ -0,0 +1,21 @@
|
|||||||
|
|
||||||
|
version: 2
|
||||||
|
|
||||||
|
models:
|
||||||
|
- name: my_first_dbt_model
|
||||||
|
description: "A starter dbt model"
|
||||||
|
columns:
|
||||||
|
- name: id
|
||||||
|
description: "The primary key for this table"
|
||||||
|
data_tests:
|
||||||
|
- unique
|
||||||
|
- not_null
|
||||||
|
|
||||||
|
- name: my_second_dbt_model
|
||||||
|
description: "A starter dbt model"
|
||||||
|
columns:
|
||||||
|
- name: id
|
||||||
|
description: "The primary key for this table"
|
||||||
|
data_tests:
|
||||||
|
- unique
|
||||||
|
- not_null
|
||||||
22
profiles.yml
Normal file
22
profiles.yml
Normal file
@@ -0,0 +1,22 @@
|
|||||||
|
econ250_2025:
|
||||||
|
outputs:
|
||||||
|
dev:
|
||||||
|
dataset: o_omelchenko # TODO: Change this to your personal dataset
|
||||||
|
job_execution_timeout_seconds: 300
|
||||||
|
job_retries: 1
|
||||||
|
location: US
|
||||||
|
method: oauth
|
||||||
|
priority: interactive
|
||||||
|
project: econ250-2025
|
||||||
|
threads: 2
|
||||||
|
type: bigquery
|
||||||
|
target: dev
|
||||||
|
my-bigquery-db:
|
||||||
|
outputs:
|
||||||
|
dev:
|
||||||
|
dataset: o_omelchenko # TODO: Change this to your personal dataset
|
||||||
|
method: oauth
|
||||||
|
project: econ250-2025
|
||||||
|
threads: 4
|
||||||
|
type: bigquery
|
||||||
|
target: dev
|
||||||
3
requirements.txt
Normal file
3
requirements.txt
Normal file
@@ -0,0 +1,3 @@
|
|||||||
|
dbt-bigquery==1.9.1
|
||||||
|
dbt-core==1.9.3
|
||||||
|
google-cloud-core==2.4.3
|
||||||
0
seeds/.gitkeep
Normal file
0
seeds/.gitkeep
Normal file
0
snapshots/.gitkeep
Normal file
0
snapshots/.gitkeep
Normal file
0
tests/.gitkeep
Normal file
0
tests/.gitkeep
Normal file
Reference in New Issue
Block a user