This blog has been moved to: http://bi-and-analytics.blogspot.nl/

BIML Series, Part 1, BIML Development approach

This summer I will post quite a bit about BIML. In this post I will share how one can develop BIML packages without purchasing additional software. I installed Microsoft Visual C# 2008 express edition, because it hardly occupies any disk space and is a free download. Within Visual C# is run this code:

using System;
using System.Data;
using System.Data.SqlClient;
using System.Collections.Generic;
using System.ComponentModel;
using System.Text;
using System.Diagnostics;
using System.Windows.Forms;


 public class ExtractTable
    {
        public string PackageName { get; set; }
        public string SequenceName { get; set; }
        public string DataFlowName { get; set; }
        public string SchemaName { get; set; }
        public string SourceSystemName { get; set; }
        public string SelectQuery { get; set; }
        public string SQLFlavour { get; set; }
        public string ExtractTableName { get; set; }
        public string SourceTableName { get; set; }
        public string ArchiveProcedure { get; set; }


        private static string metadataConnectionString = "Server=localhost;Database=meta;User Id=lalal; Password=dada;";


    private static DataTable getData(string query)
    {
        DataTable dataTable = new DataTable();

        using (SqlDataAdapter dataAdapter = new SqlDataAdapter(query, metadataConnectionString))
        {
            dataAdapter.Fill(dataTable);
        }

        return dataTable;
    }

    private static DataTable getData(string query, string parameter, string parameterValue)
    {
        DataTable dataTable = new DataTable();

        using (SqlConnection conn = new SqlConnection(metadataConnectionString))
        {
            using (SqlCommand cmd = new SqlCommand(query, conn))
            {
                cmd.Parameters.AddWithValue(parameter, parameterValue);
                SqlDataAdapter adapter = new SqlDataAdapter(cmd);

                adapter.Fill(dataTable);
            }
        }

        return dataTable;
    }

 private static ExtractTable fillExtractTable
    (
        string PackageName,
  string SequenceName,
        string DataFlowName,
        string SchemaName,
        string SourceSystemName,
        string SelectQuery,
        string SQLFlavour,
        string ExtractTableName,
  string SourceTableName,
  string ArchiveProcedure
    )
    {
        ExtractTable extractTable = new ExtractTable();

        extractTable.PackageName = PackageName;
  extractTable.SequenceName = SequenceName;
        extractTable.DataFlowName = DataFlowName;
        extractTable.SchemaName = SchemaName;
        extractTable.SourceSystemName = SourceSystemName;
        extractTable.SelectQuery = SelectQuery;
        extractTable.SQLFlavour = SQLFlavour;
        extractTable.ExtractTableName = ExtractTableName;
  extractTable.SourceTableName = SourceTableName;
  extractTable.ArchiveProcedure = ArchiveProcedure;
  
        return extractTable;
    }
    [STAThread]
    static void Main()
    {
      //  Package objects
 var extractTables = new List();

 //  Variables used to retrieve and write data
 string query;
 DataTable tblExtractTables;

 // Add ExtractTables
    query = "SELECT top 1 "
    + "   'Extract_Entrino_' + bet.SourceTableName AS PackageName,"
    + "   bet.SequenceName,"
    + "   bet.DataFlowName,"
    + "   bet.SchemaName,"
    + "   bet.SourceSystemName,"
    + "   bet.SelectQuery,"
    + "   bet.SQLFlavour,"
    + "   bet.ExtractTableName,"
    + "   bet.SourceTableName,"
    + "   bet.ArchiveProcedure"
    + " FROM META.BimlExtractArchivePackages AS bet       ";
 

 tblExtractTables = getData(query);

 foreach (DataRow row in tblExtractTables.Rows)
 {
  extractTables.Add(fillExtractTable(row[0].ToString(), row[1].ToString(), row[2].ToString(), row[3].ToString(), row[4].ToString(), row[5].ToString(), row[6].ToString(), row[7].ToString(), row[8].ToString(), row[9].ToString()));
 }


    
    foreach (ExtractTable extractTable in extractTables)
    {
        string text = System.IO.File.ReadAllText(@"C:\TFS\Temp\biml.txt");
        text = text.Replace("<#=extractTable.PackageName#>", extractTable.PackageName);
        text = text.Replace("<#= extractTable.SelectQuery#>", extractTable.SelectQuery);
        Clipboard.SetText(text);

   
    }   
    }

}

The most important thing to note about this code is the fact a predefined piece of biml code is loaded from a text file and the variables are replaced by the values coming from the meta data repository in SQL. The resulting non dynamic biml code can then be pasted in VS2008 or later with BIDS helper to generate the package. Any error messages will be a lot easier to debug. Will be continued..

First experience with Neo4J

During the DWH and BI summit I got inspired to look into Graph Databases. A former colleague recommended looking into Neo4J. It turned out to be quite easy to load data in Neo4J en more easily explore related data. In our case we wanted to analyse our Meta Data Driven DWH Framework. Which had grown quite a bit and become more and more difficult to maintain. We gave the 'Force directed Graph' visualization in Power BI a try, but this was to limited for our analysis goals.

The video below shows the results.

More in Neo4J in the future!

Remote controlled car with Raspberry Pi (piborg) and HC-SR04 ultrasonic sensor (to detect walls and stop car)

I this post I will show how to build a remote controlled car with your Raspberry Pi that will stop whenever it encounters a wall at a predefined distance. The result will look like this:

The car looks like this in detail:

Components

Smart buggy 16 euro.

Two additional wheels, that I glued to the plastic board.
PIBORG - PICOBORG V2 - MOTOR CONTROLLER, 13 euro.

Triborg, 9 euro.

The ultra sonic distance sensor, HCSR04, 2,50 euro.

Wiring was done as described on the internet. The car is remote controlled via telnet using Putty.

Code

The code below is executed. This was tricky because the key listener needs to be non blocking. Once I found one that is non blocking. Its just a matter of using keys, I, J, K, L for steering the car. The distance sensor checks for distances less than 30 CM. The motor is turned off when this happens.

import sys
import select
import tty
import termios
import contextlib
import time
import RPi.GPIO as GPIO
GPIO.setmode(GPIO.BCM)


#used by the distance meter
TRIG = 23
ECHO = 24

GPIO.setup(TRIG,GPIO.OUT)
GPIO.setup(ECHO,GPIO.IN)

#used by the piborg
# Set which GPIO pins the drive outputs are connected to
DRIVE_1 = 4
DRIVE_2 = 18
DRIVE_3 = 8
DRIVE_4 = 7

# Set all of the drive pins as output pins
GPIO.setup(DRIVE_1, GPIO.OUT)
GPIO.setup(DRIVE_2, GPIO.OUT)
GPIO.setup(DRIVE_3, GPIO.OUT)
GPIO.setup(DRIVE_4, GPIO.OUT)

# Map current on/off state to command state
dInvert = {}
dInvert[True] = GPIO.LOW
dInvert[False] = GPIO.HIGH

# Map the on/off state to nicer names for display
dName = {}
dName[True] = 'ON '
dName[False] = 'OFF'

# Function to set all drives off
def MotorOff():
    GPIO.output(DRIVE_1, GPIO.LOW)
    GPIO.output(DRIVE_2, GPIO.LOW)
    GPIO.output(DRIVE_3, GPIO.LOW)
    GPIO.output(DRIVE_4, GPIO.LOW)

def MotorOn():
    GPIO.output(DRIVE_1, GPIO.HIGH)
    GPIO.output(DRIVE_2, GPIO.HIGH)
    GPIO.output(DRIVE_3, GPIO.HIGH)
    GPIO.output(DRIVE_4, GPIO.HIGH)


def afstand():
        time.sleep(0.2)
        GPIO.output(TRIG, False)
        GPIO.output(TRIG, True)
        time.sleep(0.00001)
        GPIO.output(TRIG, False)
        while GPIO.input(ECHO)==0:
            pulse_start = time.time()

        while GPIO.input(ECHO)==1:
            pulse_end = time.time()

        pulse_duration = pulse_end - pulse_start
        distance = pulse_duration * 17150
        distance = round(distance, 2)

        return distance
  #distance

def isData():
    return select.select([sys.stdin], [], [], 0) == ([sys.stdin], [], [])


old_settings = termios.tcgetattr(sys.stdin)
try:
    tty.setcbreak(sys.stdin.fileno())


    while 1:
        if (afstand() < 50):
            MotorOff()


        if isData():
            c = sys.stdin.read(1)
            print str((c))
            if c == '\x1b':  # x1b is ESC
                break
            elif c == 'i': #Vooruit
                MotorOn()
            elif c == 'k': #Stop
                MotorOff()
            elif c == 'j': #Links
                print 'links'
                if ((GPIO.input(DRIVE_1)) & (GPIO.input(DRIVE_4))):
                    print 'aan het rijden'
                    GPIO.output(DRIVE_1, GPIO.LOW)
                    GPIO.output(DRIVE_2, GPIO.LOW)

                    time.sleep(0.1)
                    GPIO.output(DRIVE_1, GPIO.HIGH)
                    GPIO.output(DRIVE_2, GPIO.HIGH)
                else:
                    GPIO.output(DRIVE_3, dInvert[GPIO.input(DRIVE_3)])
                    GPIO.output(DRIVE_4, dInvert[GPIO.input(DRIVE_4)])
            elif c == 'l': #Rechts
                print 'rechts'
                if ((GPIO.input(DRIVE_1)) & (GPIO.input(DRIVE_4))):
                    print 'aan het rijden'
                    GPIO.output(DRIVE_4, GPIO.LOW)
                    GPIO.output(DRIVE_3, GPIO.LOW)

                    time.sleep(0.1)
                    GPIO.output(DRIVE_4, GPIO.HIGH)
                    GPIO.output(DRIVE_3, GPIO.HIGH)
                else:
                    GPIO.output(DRIVE_1, dInvert[GPIO.input(DRIVE_1)])
                    GPIO.output(DRIVE_2, dInvert[GPIO.input(DRIVE_2)])
finally:
    print("stop")
    termios.tcsetattr(sys.stdin, termios.TCSADRAIN, old_settings)

Market Basket Analysis (Association Rule Learning) with Power BI (DAX) and R

Introduction

In this post I will show how to run an R script from Power BI which will execute an Association rule learning script to perform market basket analysis.

In this example we will not look at products sold, but products sharing shelf space.

The dataset

Our basic dataset looks like this.

 Our products:

The distribution / presence of products on the shelf of a customer:

The Power BI building blocks

The data model

As for the DAX part we will start with this post of Marco Russo and Alberto Ferrari.

So the data model in Power BI looks like this:

The R visualization

We will look at the DAX part later on. First we add an R component with a script that will return the AR rules it found.

The table contains the basic output that is to be expected from AR. We will try to build these measures in DAX later on.

The R script

As for the R script it looks like this:

   
save(dataset, file="C:/TFS/dataset.rda")

library(arules, lib.loc="C:/TFS/Rlib/a/" , logical.return = FALSE,
        warn.conflicts = F, quietly = T,verbose = F)
library(plotrix, lib.loc="C:/TFS/Rlib/p/" , logical.return = FALSE,
        warn.conflicts = F, quietly = T,verbose = F)

dataset = cbind(dataset, 1)
colnames(dataset) = c("ProductID", "CustomerID", "Waarde")
reports = xtabs(Waarde~CustomerID+ProductID, data=dataset)
reports[is.na(reports)] <- 0
rules <- apriori(as.matrix(as.data.frame.matrix(reports)),parameter = list(supp = 0.03, conf = 0.5, target = "rules"))
t = inspect(head(sort(rules, by ="support"),15))

par(mar = c(0,0,0,0))
plot(c(0, 0), c(0, 0))
if (is.null(t)) {
  t = data.frame("no rules found")
  text(x = 0.5, y = 0.5, paste("No Rules found"), 
       cex = 1.6, col = "black")
} else {
  addtable2plot(-1, -1, t, bty = "n", display.rownames = F, hlines = F,
                vlines = F)
}

Unfortunately Power BI initializes a new R sessions each time the R visualization is run / cross filtered.  Therefore I tried to use a much base R as possible. As for the libraries that need to be loaded. I put these in a separate folder on my local drive and specified the folder name in the library command.

Building it in DAX

Support

The output of the arules R script can be built in DAX whenever it concerns single item combinations, so X -> Y. So not A, B -> Y.  The 'support' measure is basically the '[Orders with Both Products %]' described by Russo and Ferrari. Just to show how its implemented on our dataset.

   
Customers with Both Products % = 
IF (
    NOT ( [SameProductSelection] );
    DIVIDE ( [Customers with Both Products]; [Unique Customers All] )
)

The building blocks of this formula:

Same product selection, since this is useless. SameProductSelection = IF ( HASONEVALUE ( Products[ID] ) && HASONEVALUE ( 'Filter Products'[ID] ); IF ( VALUES ( Products[ID] ) = VALUES ( 'Filter Products'[ID] ); TRUE ) )
Customers with both products: Customers with Both Products = CALCULATE ( DISTINCTCOUNT ( Distribution[Customer ID] ); CALCULATETABLE ( SUMMARIZE ( Distribution; Distribution[Customer ID] ); ALL ( Products ); USERELATIONSHIP ( Distribution[Product ID]; 'Filter Products'[ID] ) ) )	Number of customers in total: Unique Customers All = CALCULATE ( DISTINCTCOUNT ( Distribution[Customer ID] ); ALL ( Products ) )

Confidence

   
Confidence = [Customers with Both Products] / [Unique Customers LHS]

Unique Customers LHS:

   
Unique Customers LHS = DISTINCTCOUNT(Distribution[Customer ID])

Lift

Lift = [Confidence] / [Proportion Product RHS]

Proportion product RHS:

Proportion Product RHS = Distribution[Unique Customers RHS] / [Unique Customers All]

Unique customer RHS:

 
Unique Customers RHS = 
CALCULATE (
    DISTINCTCOUNT ( Distribution[Customer ID] );
    CALCULATETABLE (
        SUMMARIZE ( Distribution; Distribution[Customer ID] );
        ALL ( Products );
        USERELATIONSHIP ( Distribution[Product ID]; 'Filter Products'[ID] )
    ); ALL(Products)
)

You can download the Power BI file here.

In this video you see the Power BI file in use:

What if Analysis with Microsoft Analysis Services

In this post we will perform a what if analysis on a Bill of Material (BOM) using Analysis Services 2008. Consider the following BOM:

We would like to calculate the price of the Bike considering the quantity and price of the components. Lets say we use the following prices; 

- The Rim 3 euro

- Bolt, Nut and Cap 2 euro each

- Spokes, 1 euro. 

- Color, 2 euro. 

- Frame, 2 euro, and 2 pieces (made mistake entering data.. we can correct using what if analysis).

Lets go bottom up; 

Level 3; Tire Valve, would be Quantity 24 (1 x 2 x 2 x 2 x 3) x 2 euro = 48 euro. 

Level 2; 

• Rim; Quantity 4 (1 x 2 x 2) x 3 euro = 12 euro.
• Tire; Quantity 2 x 24 euro = 48 euro. 
• Spokes; Quantity 144 (1 x 2 x 72) x 1 euro = 144 euro.

Level 1; Frame Quantity 2 x 2 euro  = 4 euro + Wheel 204 euro (12 + 48 + 144) = 208 euro. 

I entered this data, incl. the mistake of the quantity of frames in the following spreadsheet:

In the video below I show how to build a cube that calculates the finished good price. It will also be possible to perform what if analysis. 

In the video below I correct the mistake of 2 frames, which should be one frame of course. 

The following MDX code is used in the solution. You can download the solution file here. 

    
        /*
        The CALCULATE command controls the aggregation of leaf cells in the cube.
        If the CALCULATE command is deleted or modified, the data within the cube is affected.
        You should edit this command only if you manually specify how the cube is aggregated.
        */
        CALCULATE;

        SCOPE([Measures].[Quantity]);  
         THIS=[Item].[Parent].CURRENTMEMBER.DATAMEMBER;  
        END SCOPE;  

        SCOPE([Measures].[Price]);  
         THIS=[Item].[Parent].CURRENTMEMBER.DATAMEMBER;  
        END SCOPE;  


        CREATE MEMBER CURRENTCUBE.[Measures].[Quantity2]
         AS 
        ( [Measures].[Quantity]);   


        CREATE MEMBER CURRENTCUBE.[Measures].[lvl]
         AS 
        [Parent].currentmember.level.name , 
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';   


        CREATE MEMBER CURRENTCUBE.[Measures].[Q1]
         AS 
        (ancestor([Parent].currentmember.parent.datamember,1), [Quantity2] ) , 
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';   

        CREATE MEMBER CURRENTCUBE.[Measures].[Q2]
         AS 
        (ancestor([Parent].currentmember.parent.datamember,2), [Quantity2] ) , 
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';   

        CREATE MEMBER CURRENTCUBE.[Measures].[Q3]
         AS 
        (ancestor([Parent].currentmember.parent.datamember,3), [Quantity2] ) , 
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';   

        CREATE MEMBER CURRENTCUBE.[Measures].[Q4]
         AS 
        (ancestor([Parent].currentmember.parent.datamember,4), [Quantity2] ) , 
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';   

        CREATE MEMBER CURRENTCUBE.[Measures].[Q5]
         AS 
        (ancestor([Parent].currentmember.parent.datamember,5), [Quantity2] ) , 
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';   

        CREATE MEMBER CURRENTCUBE.[Measures].[Q6]
         AS 
        (ancestor([Parent].currentmember.parent.datamember,6), [Quantity2] ) , 
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';   


        CREATE MEMBER CURRENTCUBE.[Measures].[RQ]
         AS 
        case    when lvl = '02' then [Quantity2]     
                when lvl = '03' then [Quantity2] * Q1    
                when lvl = '04' then [Quantity2] * Q1 * Q2    
                when lvl = '05' then [Quantity2] * Q1 * Q2 * Q3     
                when lvl = '06' then [Quantity2] * Q1 * Q2 * Q3 * Q4     
                when lvl = '07' then [Quantity2] * Q1 * Q2 * Q3 * Q4 * Q5 
            
        else 0
        end,
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';   

        CREATE MEMBER CURRENTCUBE.[Measures].[Item_Usage_Price]
         AS null, 
        VISIBLE = 1 ,  ASSOCIATED_MEASURE_GROUP = 'BOM';  

        Scope ([Measures].[Item_Usage_Price]);   
            Scope (Descendants([Parent].CurrentMember,,LEAVES));   
                this =  iif([Parent].CurrentMember.parent.Properties('Key') <> [Parent].CurrentMember.Properties('Key'), [Measures].[Price] * [RQ],0);   
            end scope;   
            scope([Parent].members); 
                this = sum(
                            Filter(
                            Descendants([Parent].CurrentMember,,LEAVES)
                            , [Parent].CurrentMember.parent.Properties('Key') <> [Parent].CurrentMember.Properties('Key')
                                   )
                            ,
                           ( [Measures].[Price] * [Measures].[RQ] )  
                        );   
            end scope;  
        end scope; 

Internet of Things, my first "thing"

After watching this very nice episode of Tegenlicht about smart cities I got inspired to start sharing my experiences with my Raspberry Pi. I started playing with the Pi about 2 years ago and tried a most of the common sensors (temp, light, sound, movement, etc). The coming period I will start sharing my experiences. The is the Tegenlicht episode on smart cities, higly recommended!



By the end of this post we are able to view the temperature on a webpage which can be viewed with your computer or any mobile device of course.

The required hardware components:

- Raspberry Pi 2 or 3.. (35 euro)

- Wifi adapter

- SD card

- DHT11 Temparature sensor (2,45 euro)

- mini breadboard (1,55 euro)

- 10 Kohm resistor (0,10 euro) 

Regarding the hardware purchase; I like to buy the Pi at SOS solutions because this guy makes sure the peripherals are high quality. You can get the Pi for a slightly lower price elsewere but he makes up with good service 7 days a week!

I buy all the componts at vanallesenmeer.nl. This shop offers the products at the lowest price possible and delivery has always been on time.  

Software used:

- Putty for telnet session

- FileZilla to transfer the files with FTP

- Notepad++ to edit / write the code. 

Putting it together:

This shows how I wired the DHT11:

It looks like this in real life:

The software:
We install the ADA fruit DHT library for convenience as instructed.

  
sudo apt-get update
sudo apt-get install build-essential python-dev
sudo python setup.py install

Now we can create a file with the following contents, just like the example that has been provided:

#!/usr/bin/python

import Adafruit_DHT
sensor = Adafruit_DHT.DHT11
pin = 18
humidity, temperature = Adafruit_DHT.read_retry(sensor, pin)

if humidity is not None and temperature is not None:
 print 'Temp={0:0.1f}*C  Humidity={1:0.1f}%'.format(temperature, humidity)
else:
 print 'Failed to get reading. Try again!'

When you place this file in the /var/www/ folder next to a php file with the following contents you will be able to disply the temperature on a webpage.

<html>
<?php
echo exec('sudo python ./simpletest.py')
?>
</html>

You need to make the www-data group owner of the file:

sudo chown www-data temp.php

When you open the page in a browser the result looks like this. Mind its hot in the closet where the Pi resides..