Objective
Build a time-series forecasting model that can predict daily network egress costs for a cloud environment based on historical usage patterns, enabling accurate budget planning and cost optimization.
Business Value
- Financial Planning: Accurate forecasting for quarterly and annual cloud budget allocation
- Cost Optimization: Identify patterns to implement data transfer optimization strategies
- Anomaly Detection: Detect unusual cost spikes that may indicate misconfigurations or security issues
- Budget Control: Prevent unexpected overspend on cloud egress charges
- Capacity Planning: Predict when egress patterns require infrastructure changes
Core Libraries
- prophet: Time-series forecasting with automatic seasonality detection and trend analysis
- pandas: Time-series data manipulation and date handling
- numpy: Numerical computations and statistical operations
- scikit-learn: Model evaluation metrics
- matplotlib: Time-series visualization and forecast plotting
Dataset
- Source: Synthetically Generated (realistic daily egress cost data)
- Size: 730 days (2 years) of historical cost data
- Features: Date (ds) and daily egress cost (y)
- Patterns: Weekly seasonality, growth trends, random spikes, weekend reductions
- Type: Time-series regression with temporal dependencies
Step-by-Step Guide
1. Environment Setup
# Create virtual environment
python -m venv egress_cost_env
source egress_cost_env/bin/activate # On Windows: egress_cost_env\Scripts\activate
# Install required packages
pip install pandas numpy scikit-learn matplotlib prophet
2. Data Generation and Preparation
# Generate realistic time-series egress cost data
import pandas as pd
import numpy as np
# Create 2 years of daily data with realistic patterns
days = 730
cost_per_gb = 0.05
start_date = pd.to_datetime('2022-01-01')
dates = pd.date_range(start_date, periods=days, freq='D')
# Combine multiple patterns: trend + seasonality + noise + spikes
trend = np.linspace(500, 1500, days) # Growth from 500GB to 1500GB/day
weekly_seasonality = np.sin(dates.dayofweek (2 np.pi / 7)) * 100
weekend_reduction = np.where(dates.dayofweek >= 5, 0.2, 1.0) # 80% reduction on weekends
noise = np.random.normal(0, 50, days)
spikes = np.random.choice([0, 1], size=days, p=[0.97, 0.03]) * np.random.uniform(500, 1000, days)
# Calculate final egress cost
egress_gb = (trend + weekly_seasonality * weekend_reduction + noise + spikes)
egress_gb = np.maximum(100, egress_gb) # Minimum 100GB/day
cost_usd = egress_gb * cost_per_gb
# Create Prophet-compatible DataFrame
df = pd.DataFrame({'ds': dates, 'y': cost_usd})
3. Data Exploration and Visualization
import matplotlib.pyplot as plt
# Visualize time-series patterns
plt.figure(figsize=(15, 10))
# Plot full time series
plt.subplot(2, 2, 1)
plt.plot(df['ds'], df['y'], alpha=0.8)
plt.title('Historical Daily Egress Cost')
plt.ylabel('Cost (USD)')
# Monthly aggregation to show trend
df_monthly = df.set_index('ds').resample('M')['y'].mean()
plt.subplot(2, 2, 2)
plt.plot(df_monthly.index, df_monthly.values, marker='o')
plt.title('Monthly Average Egress Cost')
# Weekly pattern analysis
df['weekday'] = df['ds'].dt.day_name()
weekday_avg = df.groupby('weekday')['y'].mean()
plt.subplot(2, 2, 3)
plt.bar(weekday_avg.index, weekday_avg.values)
plt.title('Average Cost by Day of Week')
plt.xticks(rotation=45)
plt.tight_layout()
plt.show()
4. Data Splitting for Time-Series
# Split data chronologically (train on past, test on future)
split_point = len(df) - 90 # Hold out last 90 days for testing
df_train = df.iloc[:split_point]
df_test = df.iloc[split_point:]
print(f"Training period: {df_train['ds'].min()} to {df_train['ds'].max()}")
print(f"Test period: {df_test['ds'].min()} to {df_test['ds'].max()}")
5. Model Training with Prophet
from prophet import Prophet
# Configure Prophet for cost forecasting
model = Prophet(
yearly_seasonality=True,
weekly_seasonality=True,
daily_seasonality=False,
changepoint_prior_scale=0.05, # Controls trend flexibility
seasonality_prior_scale=10.0, # Controls seasonality flexibility
interval_width=0.95 # 95% confidence intervals
)
# Train the model
model.fit(df_train)
print("Prophet model trained successfully")
6. Forecasting and Prediction
# Create future dataframe and generate forecast
future = model.make_future_dataframe(periods=90) # 90 days ahead
forecast = model.predict(future)
# Extract key forecast components
forecast_cols = ['ds', 'yhat', 'yhat_lower', 'yhat_upper', 'trend', 'weekly']
print("Forecast generated with uncertainty intervals")
print(forecast[forecast_cols].tail())
7. Model Evaluation
from sklearn.metrics import mean_absolute_error
# Evaluate on test set
y_true = df_test['y'].values
y_pred = forecast.iloc[split_point:]['yhat'].values
# Calculate performance metrics
mae = mean_absolute_error(y_true, y_pred)
mape = np.mean(np.abs((y_true - y_pred) / y_true)) * 100
rmse = np.sqrt(np.mean((y_true - y_pred) ** 2))
print(f"Mean Absolute Error: ${mae:.2f}")
print(f"Mean Absolute Percentage Error: {mape:.2f}%")
print(f"Root Mean Square Error: ${rmse:.2f}")
# Check prediction interval coverage
forecast_test = forecast.iloc[split_point:]
within_interval = np.sum((y_true >= forecast_test['yhat_lower'].values) &
(y_true <= forecast_test['yhat_upper'].values))
coverage = (within_interval / len(y_true)) * 100
print(f"Prediction interval coverage: {coverage:.1f}%")
8. Extended Future Predictions
# Generate 6-month ahead forecast
extended_future = model.make_future_dataframe(periods=270) # 90 + 180 days
extended_forecast = model.predict(extended_future)
# Monthly cost summaries for planning
future_6_months = extended_forecast.iloc[len(df):]
future_6_months['month'] = future_6_months['ds'].dt.to_period('M')
monthly_forecast = future_6_months.groupby('month')['yhat'].agg(['sum', 'mean'])
print("Monthly cost forecasts for next 6 months:")
print(monthly_forecast.round(2))
Success Criteria
- Low MAPE (<15%): Accurate percentage-based forecasting for budget planning
- High Interval Coverage (>90%): Reliable uncertainty bounds for risk assessment
- Seasonal Pattern Detection: Model correctly identifies weekly and yearly patterns
- Trend Capture: Successfully models growth trends in egress usage
Next Steps & Extensions
1. Real-time Integration: Connect with cloud billing APIs for live cost monitoring
2. Multi-region Forecasting: Extend to predict costs across different cloud regions
3. Service-level Breakdown: Forecast costs by individual services or applications
4. Automated Alerting: Set up alerts when actual costs exceed prediction intervals
5. Cost Optimization: Identify peak usage periods for data transfer optimization
6. Budget Integration: Connect forecasts with financial planning and approval workflows
Files Structure
031_Cloud_Network_Egress_Cost_Prediction/
├── readme.md
├── cloud_network_egress_cost_prediction.ipynb
├── requirements.txt
└── data/
└── (Generated time-series cost data)
Running the Project
1. Install required dependencies from requirements.txt
2. Execute the Jupyter notebook step by step
3. Analyze forecast components to understand cost drivers
4. Use extended forecasts for 6-month budget planning
5. Implement real-time monitoring based on prediction intervals
This project demonstrates how time-series forecasting can transform cloud cost management by providing accurate egress cost predictions, enabling proactive budget planning and cost optimization strategies for cloud environments.