Data Engineer Pipeline Productionizing High Level Task Document 

Task	Description
1. Understand use case and expected input output-output	Get the initial understanding of the project. Expected Input and output
2. Hadover checklist	Get Handover: Checklist from Data Scientist and Review it
3. Create SR for Data Sources Access	Raise SR for Required Data Sources. It’s better to have a service account setup instead of getting individual account access
4. Run the scripts on local	Run all scripts on local and monitor the execution time
5. SRs for SSO set, If it’s a Chatbot	Required for Cloud Memo Approval per ITD
6. Architecture diagram	High level Diagram Showcasing the flow and Resources used
7. Data Model	Relationship between data elements
8. Table creation	Snowflake table Creation for storing output
9. Process Flow	ER Diagram, Flow Diagram etc
10. Code changes to dump output to snowflake	Write function to dump output into snowflake table
11. Create Roles and policy create	Terraform Scripts for roles and policy creation.
12. Repo creation for infrastructure — communicate –> Roles and Policy	Terraform scripts for communication of roles and policy
13. Raise SR for Gitaction setup and repo whitelisting	Gitaction setup
14. Build docker and run the code on docker	Docker image creation and Testing code on docker
15. Pipeline Scheduler Testing end to end	Test pipeline end to end

---

Roles and AWS Infrastructure Deployment

Overview

The diagram describes a workflow for deploying AWS infrastructure using roles and permissions via two separate pipelines. These pipelines handle different responsibilities:

• Pipeline 1: Deploys AWS roles and policies.
• Pipeline 2: Deploys specific infrastructure using those roles and policies.

---

Key Components

1. Repositories & Pipelines

• pbc-gitactions-role-management
• Purpose: Manages AWS Roles and Permissions.
• Triggers: On Push to Test and Main Branch, or manual workflow dispatch.
• It uses one git repository to manage all projects’ role and policy. For each project, it requires to create a related feature branch.
• The benefit of it is that it does not require to whitelist the repository every time when lunch a new project.
• eg. pbc-ghg-recommender
• Purpose: Deploys specific AWS infrastructure (e.g., data quality pipelines).
• Triggers: On Push to Test and Main Branch, or manual workflow dispatch.

Both use GitHub Actions for their CI/CD pipelines.

2. Terraform

• Both pipelines use Terraform for infrastructure as code (IaC) deployments.
• Terraform interacts with AWS to provision resources.

---

Workflow Steps

Step 1: Roles and Policies Deployment

Repository: pbc-gitactions-role-management

1. Code Commit or Workflow Dispatch
   • A developer pushes changes to the repository, or manually triggers a workflow.
2. GitHub Actions Pipeline
   • The pipeline runs the configured workflow.
3. Terraform Execution
   • The workflow uses Terraform scripts to define and create AWS roles and policies.
   • These roles and policies specify what actions various users or services can perform in AWS.
4. AWS Cloud
   • Terraform, authenticated via pre-created permissions, creates/updates:
     • Roles
     • Permissions
     • Policies

Key Point:

• This pipeline is responsible for security management: setting up who can do what in AWS.

---

Step 2: Infrastructure Deployment

Repository: pbc-ghg-recommender

1. Code Commit or Workflow Dispatch
   • A developer pushes infrastructure code (e.g., for a data quality pipeline), or manually triggers a workflow.
2. GitHub Actions Pipeline
   • The pipeline runs the configured workflow.
3. Assume Role for AWS Authentication
   • This pipeline does not have direct permissions to create resources.
   • Instead, it assumes a role created by the first pipeline for authentication.
   • This mechanism is called role assumption in AWS, where an entity temporarily gets the permissions of the assumed role.
4. Terraform Execution
   • The pipeline uses Terraform scripts to define and create specific AWS infrastructure (e.g., Lambda functions, S3 buckets, etc.).
5. AWS Cloud
   • Terraform, authenticated via the assumed role, provisions the required infrastructure.

Key Point:

• This pipeline is responsible for resource deployment: setting up the actual infrastructure using specific permissions.

Gitaction Yaml file for reference:

name: Role-Management

on:
  push:
    tags:
      - "v[0-9]+.[0-9]+.[0-9]+*"
    branches:
      - dev
      - main
    # trigger this pipeline only when there is change in terraform directory
    paths:
      - 'terraform/**'
# For the push event to trigger the workflow, you need:
# A push to EITHER a matching tag (like v1.0.0) OR one of the specified branches 
# (dev/main)
# AND changes must be in the terraform/ directory
  workflow_dispatch:
    inputs:
      DESTROY_FLAG:
        type: boolean
        required: true
        description: "Check this box if you want a destroy infrastructre"

# Sets required permissions for GitHub OIDC token authentication
permissions:
  id-token: write
  contents: read

jobs:
  Infrastructure_Deployment:
    runs-on: ubuntu-latest
    env:
      environment: ${{ github.ref == 'refs/heads/main' && 'prod' || 'dev' }}
      account: ${{ github.ref == 'refs/heads/main' && '992382849580' || '301691044089' }}
    steps:
      - name: Check Out Repo
        uses: actions/checkout@v3

      - name: Auth to AWS
        uses: PaccarInc-Digital/actions-aws-auth@v1.1.1
        with:
          role: "arn:aws:iam::${{ env.account }}:role/pbc-gitactions-role-management-role"
          role-hop-target: "arn:aws:iam::022592466027:role/aws-oidc-legacy-role-2"
      
      - name: Call AWS Identity
        shell: bash
        run: aws sts get-caller-identity
      
      - name: Set Environment variables
        id: branch_check
        run: |
          echo "Running on branch ${{ github.ref }}"
          if [ "${{ github.ref }}" = "refs/heads/main" ]; then
            echo "environment=prod" >> "$GITHUB_ENV"
          elif [ "${{ github.ref }}" = "refs/heads/dev" ]; then
          echo "environment=dev" >> "$GITHUB_ENV"
          fi   

      - name: Remove existing Terraform state and lock file
        run: |
          rm -rf .terraform .terraform.lock.hcl
          rm -rf .terraform
          rm -rf .terraform terraform.tfstate
          rm -rf .terraform terraform.tfstate.backup
          rm -rf .terraform plan.tfplan

      - name: Setup Terraform
        uses: hashicorp/setup-terraform@v2
        with:
          terraform_version: 1.3.7
          terraform_wrapper: false

      - name: Terraform Init
        working-directory: ./terraform
        run: terraform init -backend-config="backends/${{ env.environment }}.s3.tfbackend"
      
      - name: Terraform Plan
        id: plan
        working-directory: ./terraform
        run: terraform plan -var-file="variables/${{ env.environment }}.tfvars" -out=plan.tfplan
        continue-on-error: true

      - name: Terraform Plan Status
        if: steps.plan.outcome == 'failure'
        run: exit 1
      
      - name: Terraform Deploy
        working-directory: ./terraform
        run: |
          if ${{ github.event.inputs.DESTROY_FLAG == 'true' }}; then
            terraform destroy -auto-approve
          else
            terraform apply plan.tfplan
          fi

Explain a little bit about the aws part:

- name: Auth to AWS
        uses: PaccarInc-Digital/actions-aws-auth@v1.1.1
        with:
          role: "arn:aws:iam::${{ env.account }}:role/pbc-gitactions-role-management-role"
          role-hop-target: "arn:aws:iam::022592466027:role/aws-oidc-legacy-role-2"

How It Works

1. Initial OIDC Authentication:
   • Uses GitHub’s OIDC (OpenID Connect) integration with AWS
   • The permissions: id-token: write at the top of the workflow enables this
   • No long-term AWS credentials need to be stored as secrets
2. Dynamic Account Selection:
   • ${{ env.account }} resolves to:
     • 992382849580 when running on main branch (prod)
     • 301691044089 when running on other branches (dev)
3. First Role Assumption:
   • Assumes the role pbc-gitactions-role-management-role in the selected account
4. Role Hopping (Cross-Account Access):
   • After assuming the first role, it then assumes a second role
   • This second role aws-oidc-legacy-role-2 is in a different AWS account (022592466027)
   • This is a common pattern for centralized permissions management
5. Result:
   • After this step completes, all subsequent AWS operations (like Terraform commands) will use the permissions of the final role
   • The workflow verifies successful authentication in the next step using aws sts get-caller-identity

This approach improves security by:

• Using short-lived credentials instead of static access keys
• Implementing the principle of least privilege
• Creating an audit trail of which GitHub workflow accessed AWS resources
• Enabling cross-account access without storing credentials

---

Workflow Interactions

• The role management pipeline (pipeline 1) must run before the infrastructure pipeline (pipeline 2), because the latter depends on roles and permissions created by the former.
• By separating roles/policies from infrastructure, the workflow enforces security best practices, ensuring only authorized actions are performed by the infrastructure pipeline.

---

Security Model

• Pipelines do not share AWS credentials directly.
• The infrastructure pipeline assumes a pre-created role (from the first pipeline) for limited access, adhering to the principle of least privilege.
• This division minimizes risks and simplifies audit and compliance.

---

Summary Table

Pipeline	Purpose	Trigger	AWS Access Method	Terraform Action
pbc-gitactions-role-management	Manage roles & policies	Push/Dispatch	Pre-created permissions	Create roles & policies
pbc-warrant-data-quality	Deploy infrastructure	Push/Dispatch	Assume role (from pipeline 1)	Create specific infrastructure

---

Illustrative Flow

1. Roles & Policies Pipeline:
   • Runs → Creates/updates roles and policies in AWS via Terraform.
2. Infrastructure Pipeline:
   • Runs after roles are in place → Assumes a role → Provisions infrastructure via Terraform.

---

Benefits of This Workflow

• Separation of concerns: Security and resource provisioning are managed independently.
• Scalability: Roles can be reused across multiple infrastructure deployments.
• Security: Only necessary permissions are granted to pipelines.
• Auditability: Clear tracking of changes to roles/policies vs. infrastructure.