EF
///
Philippines Finest Barong 3/20/2024

Data Pipeline Architecture

Deep dive into the BigQuery and Cloud Run implementation that powers the inventory intelligence system.

The Data Flow

Data is ingested from the Point of Sale (POS) system and the e-commerce platform into a unified data lake in BigQuery.

ETL Process

  1. Extraction: Nightly dumps from Shopify and Square.
  2. Transformation: Python scripts running on Cloud Run normalize the data.
  3. Loading: Cleaned data is pushed to BigQuery for analysis.
def normalize_sku(sku_string):
    """
    Standardizes SKU formats between legacy and new systems.
    """
    return sku_string.strip().upper()

Insights Generated

  • Dead Stock Analysis: Identifying items that haven’t moved in 6 months.
  • Reorder Points: Automated alerts when stock dips below calculated thresholds.