Forecasting in Power BI

I have written and talked about Time Series Forecasting previously, especially on how to create forecasts in Power BI. You can refer to my previous blog posts below:

Time series Forecasting in Python & R, Part 1 (EDA): Link This blog covers exploratory data analysis techniques for time series data before creating the forecast.

Time series Forecasting in Python & R, Part 2 (Forecasting): Link Here I go into theoretical and practical aspects of creatings forecasts using many different classical forecasting methods

Time series Forecasting in Power BI: Link This blog shows how to create forecasts in Power BI, its strengths, limitations, and my recommendations.

I recently presented on this topic at Global AI Bootcamp, Singapore. Here is the recording of that presentation. It summarizes my findings from the blog above.

Machine Learning in Power BI

Power BI and Azure ML have native integration with each other, which means not only you can consume the model deploy in Power BI but also use the resources/tools in Azure ML to manage the MLOps process. There is a misconception that you need Power BI Premium to use Azure ML. In this session I show that's not the case. Below are the topics covered:

Limitation of using Python or R in Power BI
- Formula firewall
- Privacy setting of the datasource must be Public
- 30 min query timeout
- Have to use personal gateway
- Cannot use enhanced metadata
- Dependency management
- Not scaleable
- No MLOps
Steps to access Azure ML models in Power BI
- Give Azure ML access to Power BI tenant in IAM
- Use scoring script with input/output schema

Topics Covered:

Invoking Azure ML models in Power Query in Desktop and using dataflow in service
Thoughts on batch-scoring
Other considerations:
- Azure ML models can only be called for real-time inference.
- Use ACI for small models and AKS for large models that requore GPU, scaleable compute, low latency, high availability
- Real-time inferencing is expensive! Use batch scoring if real-time is not needed
- Incremental refreshes in Power BI will work with models invoked from Azure ML but watch out for (query) performance
- With Azure Arc, you can use your own Kubernets service! Deploy the model in Azure ML but use your own K8 (AWS, GCP etc.)
- Premium Per User gives you access to Auto ML. If you have access to PPU, it may be worth first experimenting with AutoML before using Azure ML model to save cost. PPU Automl doesn't offer the same level of MLOps capabilities though.

References:

Power BI AutoML Deep Dive

In this session I wanted to show, not only how to create the machine learning models, but more importantly how to create models that answer business questions and are explainable and robust. Please watch and let me know your thoughts. Topic covered:

Machine Learning Basics
License Requirements
How to create models that are:
- Business focused (cost/loss functions/profit curves)
- Generalizable (underfitting/overfitting/error terrain analysis)
- Interpretable (SHAP, feature importance)
- Manageable (MLOps process and workflow)
Recommendations, Limitations

youtube https://youtu.be/_CaFacQiW_E

Video:

The Problem:

In Power BI, when you create a scatter plot, you have an option to change the size of the marker based on a column or a measure. This helps the user to do multivariate analysis by encoding the marker size. Although you can change the size of the marker, only the developer can set it and a big limitation is that the size scales uniformly. See below for example. The left scatter plot is the default and the one on the right is after controlling the sensitivity.

The Solution

Solution is rather simple. We just create another measure and exponentiate it to lengthen the scale. For example, if you have a measure [Sales] used to denote marker size, you use [Sales]^100. To give the developer and user better control, we can create parameter to adjust the value. Watch the video on how to do this :

Adjusting Time Based on DST

It's often a requirement in Power BI reports to use dates or display refresh times. The best practice is to calculate the refresh time using M but if you are using Direct Query or streaming dataset, it's not an option. You have to create a DAX measure. I recently had such requirement to show refresh time based on Pacific Time zone. Below measure calculates the current time in UTC timezone and returns time in Pacific time zone adjusted for daylight savings time. I used official rules from here, which state:

DST begins on the second Sunday of March
DST ends on the first Sunday of November

Page Refreshed Datetime =
// https://www.nist.gov/pml/time-and-frequency-division/popular-links/daylight-saving-time-dst
// Calculates time in Pacific timezone

VAR _utcnow =
    UTCNOW ()
VAR _year =
    YEAR ( _utcnow )
VAR _utcdate =
    DATE ( YEAR ( _utcnow ), MONTH ( _utcnow ), DAY ( _utcnow ) ) // DST begins on the second Sunday of March    
VAR _dstbegins =
    DATE ( _year, 3, 1 )
        + MOD ( 8 - WEEKDAY ( DATE ( _year, 3, 1 ) ), 7 ) + 7 // DST ends on the first Sunday of November            
VAR _dstends =
    DATE ( _year, 11, 1 )
        + MOD ( 8 - WEEKDAY ( DATE ( _year, 11, 1 ) ), 7 )
VAR _IsDST = _utcdate >= _dstbegins
    && _utcdate < _dstends
RETURN
    SWITCH (
        TRUE,
        _IsDST, _utcnow - TIME ( 7, 00, 00 ), //For Pacific Timezone
        _utcnow - TIME ( 8, 00, 00 )
    )

Stripping Off AS Connection

I recently completed a project to migrate 45 Power BI reports from analysis services to Power BI Premium datasets. If you have a Power BI file (*.pbix) that's connected to AS instance, and if you don't have access to AS, opening the Power BI file give you an error message below.

The solution to this is to use a PowerShell script developed and shared by Steve Campbell. This script works perfectly. But in my case, I had 45 reports so instead of manually applying the script to 45 reports, I just modified it a bit to strip off the connection, save the pbix and also copy the original pbix files to another folder for backup. Please note that Steve gets 100% credit for this script, I merely modified it to suit my purposes. Please refer to blog for his instructions to use the original script.

Steps

Save all the pbix files to a folder. Copy the folder path.

Replace the folder path for $dir variable as shown below. Note that the folder path must end with \

Run the PowerShell
That's it. If you now go back to the above folder, you will see another folder created called original_pbix. This folder has the original pbix files with AS connections. Files in the base folder should now be devoid of AS connection.

Open the pbix and point to the Power BI dataset. As long as all the column, table, measure names etc are the same as the AS, you should be able to successfully map the report to the Power BI dataset.

Script


##### Script to strip-off AS Connection and save the PBIX files 
##### Original script: https://powerbi.tips/2020/05/hot-swap-power-bi-report-connections/
##### Input from Bryan Campbell, Hitachi Solutions America
##### Modified the original by Sandeep Pawar to apply en-masse to many files
##### Ver 1.0 | Last Update: 10/26/2022

<#
Instructions: 

1. Save the PBIX with Analysis Services connections to a folder
2. Change the $dir variable below to above folder path

#> 


$dir = "C:\Users\<path>\AS PBI Files\" #Be sure to have a \ at the end

"Checking if $dir  exists"
if (Test-Path -Path $dir) {

    #Copy files to a new folder for backup
    New-Item -Path $dir'original_pbix' -ItemType Directory
    Copy-Item -Path $dir'*.pbix' -Destination $dir'original_pbix'


    $pbi_files = Get-ChildItem -Path $dir -Filter *.pbix


    # Loop over each file, strip off and save the file as a zip file

    foreach ($file in $pbi_files) {

    $pathn =  $dir+$file.Name 

        Write-Output "Extracting " $file
        #Error check function
        function IsFileLocked([string]$filePath){
            Rename-Item $filePath $filePath -ErrorVariable errs -ErrorAction SilentlyContinue
            return ($errs.Count -ne 0)
        }


        #Choose file
        try {$pathn}
        catch { "Incompatible File" }


        #Check for errors
        If([string]::IsNullOrEmpty($pathn )){            
            exit } 

        elseif ( IsFileLocked($pathn) ){
            exit } 

        #Run Script
        else{    

            #Unzip pbix
            [Reflection.Assembly]::LoadWithPartialName('System.IO.Compression')
            $zipfile = $pathn.Substring(0,$pathn.Length-4) + "zip"
            Rename-Item -Path $pathn -NewName  $zipfile

            #Delete files
            $files   = 'Connections', 'SecurityBindings'
            $stream = New-Object IO.FileStream($zipfile, [IO.FileMode]::Open)
            $mode   = [IO.Compression.ZipArchiveMode]::Update
            $zip    = New-Object IO.Compression.ZipArchive($stream, $mode)
            ($zip.Entries | ? { $files -contains $_.Name }) | % { $_.Delete() }

            #Close zip
            $zip.Dispose()
            $stream.Close()
            $stream.Dispose()


        }

    }


    # Change the *.zip to *.pbix

    $stripped_models = Get-ChildItem -Path $dir -Filter *.zip
    $stripped_models | Rename-Item -newname { $_.name -replace '.zip','.pbix' }




    } 
else {
    ">>>>>>> ERROR Path:  $dir doesn't exist   <<<<<<<<<<<"
}

Thanks to my colleague Bryan Campbell for his help.

Streaming Dataset

It's easy to create a streaming dataset in Power BI for real-time applications. You do not need a premium capacity. To keep the blog short, I am going to assume you already know what is a streaming dataset and how it differs from import/DQ. If you are not familiar, this documentation will help.

Imagine you are creating a POC and would like to develop the real-time dashboard, report or just want to show to your stakeholders what the solution would loook like, you will need data to stream to Power BI service. You can create Azure Event Hub, Streaming Analytics. But in this blog, I will show you how you can stream data from an existing file. I will also include a link below in the resources for a few sample datasets you can use depending on the use case.

You will need a Python IDE with pandas installed.

Create A Streaming dataset

First create a streaming dataset in Power BI service using the API method.

I will be using this dataset for my example. Note the columns and column types from your data and create the dataset in Power BI service accordingly.

Note the API endpoint:

That's it. We are now ready to stream the data to Power BI service.

Python Script

I will import the data using pandas.

import requests
from datetime import datetime
from dataclasses import dataclass
import numpy as np
from typing import List, Dict
from time import sleep
import pandas as pd

@dataclass


class StreamToPowerBI:
    
    """

    Class to stream data to Power BI Streaming dataset
    --------------------------------------------------
    Example:

     - endpoint :  API ednpoint, e.g. "https://api.powerbi.com/beta/<ws_id>/datasets/<dataset_id>/rows?key=<___key___>"
     - data     :  Data in dictionary e.g. {"id":1, "Value":2.33, "Color":"Red"} 
                   Number of columns, column names and column type must match with Power BI.
     - delay    :  Delay in seconds in sending the data to Power BI. e.g. if delay is 3, data will be sent every 3 seconds.
                   default value is 2 seconds. To overwrite, use the wait() method
                   
                   
     example:      To stream data to the specified endpoint with 5 s delay
                   StreamToPowerBI(endpoint,data).wait(5).post_data()              
 
    """
    endpoint: str
    data: Dict
    delay: int=2   
        
    def wait(self, other=None):
        ''' Delay in seconds to stream the data '''
        if other==None:
            # default value is 2 seconds
            other = 2 
        sleep(other)    
        return self
    
    def post_data(self):
        ''' Use post method send the data to Power BI'''
        if isinstance(self.data, dict):
            endpoint  = self.endpoint
            payload = [self.data]

            x = requests.post(endpoint, json = payload)
            if x.status_code==200:
                x=x
            else:
                print("Failed ! Check endpoint, data")
        else:
            print("Data passed is not a dictionary. It should be in the format {}".format('{}'))
        return x

Data

Import the data using pandas or any other library (spark, polars etc.). Be sure to make the numeric columns type float otherwise json serialization will fail. I am also removing null values.

df = (pd.read_csv("https://raw.githubusercontent.com/reisanar/datasets/master/eBayAuctions.csv", 
                  index_col=None, 
                  dtype={'Category':str,
                         'currency':str,
                         'sellerRating':float,
                         'Duration':float,
                         'ClosePrice':float,
                         'OpenPrice':float,
                         'Competitive?':float},
                  usecols=['Category','currency','sellerRating','Duration','ClosePrice','OpenPrice','Competitive?'])
     ).dropna()

Test

Before streaming data from the whole file, let's first test using one data point. Note that in the orginal data, there is no timestamp, so I am adding the timestamp , i.e DateTime column, to the dictionary. In your case, you may not want to do that. comment it out if you want to stream the data as is.

d = dict(df.iloc[0])
d['DateTime']=datetime.now().isoformat()
d

{'Category': 'Music/Movie/Game',
 'currency': 'US',
 'sellerRating': 3249.0,
 'Duration': 5.0,
 'ClosePrice': 0.01,
 'OpenPrice': 0.01,
 'Competitive?': 0.0,
 'DateTime': '2023-01-13T13:44:05.740998'}

endpoint = "https://api.powerbi.com/beta/xxxxxxx/datasets/xxxxxxxxxx"
StreamToPowerBI(endpoint,d).wait(1).post_data()

<Response [200]>

Note: Note that the API endpoint Power BI provides includes the key so never share the endpoint with anyone. If you do, they will be able to stream the data without any authentication !

If the reponse returned is <Response [200]>, it was successful. We can now stream more data.

Streaming Rows

To stream the rows, we will just iterate over the dataframe and send one row at a time with the specified delay.

To stream the entire file

for record in range(0,len(df)):
    data = dict(df.iloc[record])
    
    #including datetime is optional. If you do want it, make sure it's included in the streaming dataset defintion in Power BI service
    data['DateTime']=datetime.now().isoformat()
    
    StreamToPowerBI(endpoint,data).wait(1).post_data()

To randomly stream x% of the data

Remember the purpose here is POC so don't stream all the rows (read the documentation for limtations, especially the limitations on number of datapoints, no of requests etc.). You can use below to randomly sample x % of the rows from the dataframe.

x = 0.1

for record in range(0,len(df.sample(frac=x))):
    data = dict(df.iloc[record])
    
    #including datetime is optional. If you do want it, make sure it's included in the streaming dataset defintion in Power BI service
    data['DateTime']=datetime.now().isoformat()
    
    StreamToPowerBI(endpoint,data).wait(1).post_data()

Resources

Documentation : https://learn.microsoft.com/en-us/power-bi/connect-data/service-real-time-streaming
Example datasets: https://github.com/reisanar/datasets
Azure Open Datasets : https://github.com/Azure/OpenDatasetsNotebooks/

The Issue

I recently completed a project where we migrated Analysis Services models to Power BI Premium datasets. Since Power BI datasets are nothing but AS tabular models, it's easy to copy/paste (manually or using C# scripts) tabular model objects such as relationships, measures, calculated columns, hierarchies, RLS roles etc. Everything went smoothly except when we set up deployment pipelines for CI/CD, it failed with an error that gave no clue why it was failing. We created a support ticket with Microsoft but they were stumped as well. They suggested using Large Dataset Format to resolve the error but that did not work. But after several tests, I figured out the issue.

The Solution

The solution was simple: Change the Model Permission from Administrator to Read. The original model had Administrator role defined for one of the roles. When the object properties were copied, I copied everything along with the associated metadata. For a Power BI dataset, the security roles and access are defined in the service and not in the Power BI dataset. If you keep the Model Permission as Administrator, the Power BI dataset still works and refreshes fine but the Deployment Pipeline will fail. Don't ask me how long it took to figure this out!

Motivation

I have presented on many topics related to Machine Learning, Data Science, Azure ML, AI Visuals in Power BI but I have always wanted to present how a Power BI developer can use data science in their workflow. The common misconception is that you have to know a ton of Statistics to use data science principles. While it will always help, that's not a requirement to get started. The difference between a BI developer and a data scientist is essentially how they analyze the data to get differrent insights. In this session I show what that approach looks like from a Power BI developer's perspective, common tasks/use cases in data science and how to get started. As a Power BI developer you will be able to enrich your reports with additional insights that you won't otherwise won't without data science.

Thank you to Teo, Damu and others for inviting me to present.

Sandeep Pawar

Forecasting in Power BI

Machine Learning in Power BI

Topics Covered:

Power BI AutoML Deep Dive

Controlling Marker Size Sensitivity in Scatter Plot in Power BI

Video:

Controlling Marker Size Sensitivity in Power BI Scatter Plot

The Problem:

The Solution

DAX Measure to Dynamically Offset Datetime Based on Daylight Savings Time

Adjusting Time Based on DST

Stripping Off Analysis Services Connection From Multiple Power BI Files

Stripping Off AS Connection

Steps

Script

Streaming Data From A File To Power BI Streaming Dataset Using Python

Streaming Dataset

Create A Streaming dataset

Python Script

Data

Test

Streaming Rows

To stream the entire file

To randomly stream x% of the data

Resources

Fixing the Power BI Deployment Pipeline Error For Migrated Tabular Model With RLS

The Issue

The Solution

Data Science For Power BI Developers

Motivation