Forecasting for Demand Planning: Dynamic Regression

A lot of business forecasting is done by the classical time series methods that are solely based on past history. Methods like exponential smoothing and Box Jenkins are usually applied as time series methods and they are good at pulling trends and seasonal patterns and projecting them forward. However, the future of business (sales, demand etc.) can’t rest solely on its past pattern, there are always external factors “influencing” the behavior. In addition to using the historical data, we can look at relationships between what we are trying to forecast, e.g. sales and explanatory variables like prices, measures of the economy (GDP growth and inflation), promotional schedules, market opportunity etc.

Unlike Regression, in Time series models the sequence of measurements of a process over time introduces auto-correlations between measured values. The serial nature of the measurements must be addressed by careful examination of the lag structure of the model.

Regression technique is slightly different from time-series as it allows for the introduction of explanatory variables. So it’s a model that can not only generate forecasts but also provides insights into relationships between variables. This model also allows for a What if? scenario, say, what if we lower the price/ what if we keep it constant? Here, by varying the scenarios for the explanatory variables we can see what kind of impact it has on the forecasts.

Dynamic regression can also exploit what we know as leading indicators. One of the key things about using variables here is the question, when does the impact occur? It is here that we can think of what is called concurrent relationships. For that let’s take a simple example of price and demand. Thus if we change the price this month, when does the change in demand occur. If its concurrent, then its immediate. On the other hand somethings are not so immediate. Thus in the latter case ,what we need to consider are leading indicators that would take into account the time lag for the change to reflect.

It’s worth noting that dynamic regression is not automated and does require some amount of expertise and experimentation. Besides, even if the model built is good, there can be difficulties in forecasting the exogenous predictors (explanatory variables) which in turn will result in poor forecasts of the dependent variable. As an instance, consider building a forecasting model for the demand for an existing product. It may be price sensitive, so the relationship between price and sales (volume) with changes in price has an impact on the sales. The downside is that if it is concurrent relationship, then we have to incorporate in this scenario the price in the forecast period, i.e. future price. Put in another way, we have to forecast the prices before we can forecast the sales, which isn’t easy, and if we do not do a good job with that, then we are going to get poor forecast of sales down the line. Finally dynamic regression requires a lot of data, so the more amount of data we have, we can better understand the relationships among all the variables.

The Classic Regression Model

Sales= b x Advertising + Constant + Error

The classic regression model is the basis of dynamic regression model. For example when it is a variable we are trying to forecast, in our case its sales also may be called as the dependent variable. In the above example we are relating sales, with advertising, the independent variable or the explanatory variable. The relationship between the variables sales and advertising is quantified by using what is called the regression co-efficient. Here we have symbolized it by “b”. What it means here in simple terms is that if we spend for example a dollar on advertising, we are going to get b amount of sales. The rest of the equation has a constant term, which is just a number, some people call it an Intercept but the idea is what if our independent variable were zero, in our example advertising were zero, what would we sell. So in a month when we are not going to have any advertising expense, we are still going to sell something and the constant term lets us know what that is. Finally we have what is called the error .In text books it’s mostly represented as:

Yt = bXt + C + Et
  • Yt: Dependent variable
  • Xt: Independent(explanatory) variable
  • b: Regression coefficient
  • C: Constant term
  • Et: Error term
  • t: times series

This is the basis or the framework on which the Dynamic regression model can be extended and is basically here where it all begins.

The Variables – Internal and External

The selection of variables is the most challenging part the dynamic regression process. Here is where we need to ask ourselves what are our key sales drivers and what data or variables we have to capture them. For forecasting, it’s important to focus on the handful of the important sales drivers, so if you see graphically and visually identify the explanatory variables that have a direct impact on the sales variable we can categorise all the variables broadly into external and internal variables.

Internal Variables: These are the ones that we can control, for example, price, promotion, number of sales representatives etc. Anything that we are doing to drive sales are the type of internal variables that we would need to keep in mind when we are collecting data.

External Variables: In contrast to internal variables, there are external variables. These are the ones that have the potential to impact sales but we do not have any control over them. These may include variables like weather, economy, competition, demographics trends.

Likewise for both we need to provide forecasts for them .This is pretty straight forward for internal variables as we can plan for prices and promotions etc. For the external variables on the other hand, it is a lot more challenging and if we have a leading indicator, we would be in much better shape. If its concurrent, then it becomes a lot more challenging again, as a lot of external variables (aforementioned), will be difficult to forecast. It is here that the, “What if? “analysis can come handy. For example to forecast external variables we can create a range of possibilities like what if the economy is booming, we then can create a very optimistic sales outlook as against a period where the economy is slowing down, we would then have a very pessimistic sales outlook, and between these two outcomes would our real range for possibilities sales would lie.

Another thing that we would need to determine is the number of variables we would be using in the model ,as the more variables we are going to put in the model, the more unstable the model shall become and subsequently lesser accurate would the outcome of the model would be. Thus for a reasonable forecast we would need to be using the most important and filtered variables.

To summarise it a bit, we actually start of by building a exponentially smoothing model for the data and that can be used as a base line so that we can use dynamic regression to improve upon the results. It is often useful to creating time series model for the dependent variables prior to building the dynamic regression model. Again this is quick and easy and gives us a benchmark to outperform.

What is the work involved in regression?

The most important chore is to collect all the necessary independent variables that we think might be important and if they are not leading indicators to also forecast the future scenarios for them. After we have that, we need to build the model. It is here that a lot of softwares are available in the market that automate the dynamic regression. The dynamic models use the past history as well as the explanatory variables to generate forecasts. This can take different forms but the common forms includes lags of the dependent variables, Cochrane-Orcutt terms. Any forecasting textbooks that include dynamic regression models will also describe these models. What we are doing here is combining the past history much in the same way exponential smoothing in the Box Jenkins do with using explanatory variables.

Conclusion

Building regression models is generally an iterative process. We start with an initial model and experiment with adding or removing variables, lags and dynamic terms. Intuition, hypothesis tests and other diagnostics can guide the process. Historically, automatic algorithms have not performed well for dynamic regression modelling, but significant strides are now being made. Validate the model by making sure that it makes economic sense (e.g. Coefficients must be with the proper sign).The coefficients should be significant (therefore the t-statistics probability should ideally be .99 or above).Errors should not be auto-correlated (Ljung-box probability less than .95 error auto-correlation function un-patterned).In the end dynamic regression should be considered when there are important explanatory variables. It lends insights into relationships between variables and allows for “what-if” scenarios.

Alternative approach such as Vector autoregression (VAR) or vector error correction model (VECM) is relevant for forecasting a system of time series with all of the variables within one model. One-step-ahead forecasts from a VAR are straightforward, while multiple-step-ahead forecasts can be constructed iteratively.

Intervention Analysis using Dummy Variables

Sales promotions are often used by businesses to highlight their products and services, and increase their demand. These promotions cost money and these costs must be justified by structured analysis. Additionally, proposed sales promotions affect future demand (sometimes adversely because of cannibalization effect), so resources must be allocated accordingly to satisfy the promoted demand. In many cases, promotions can be powerful for business revenue, but disruptive for production planning and inventory management.

Business often want to understand whether a promotion activity was successful/ profitable, and whether it generated more customer base for longer term or just capped on short-term buying. This is where promotional analysis comes in picture. By encoding promotional activity as an intervention eventt) over limited duration, and looking at historical data to see how the event caused deviation, we can isolate the effect of the promotion. Intervention events are essentially dummy variables regressors or indicator variables (explicitly determined by the time, duration, and type) that are introduced through a transfer function filter (specified by the response) that results in an intervention effect.

Types of Intervention Events

  1. Point Intervention: Used for events that occur at a single, specific event of time t, like a pulse. An example is buy-two-get-one free offer on certain items during clearance sale. The value of exogenous variable ξt is 1 during the event, and 0 before and after it.
  2. Step Intervention: For events that start at a certain time t and stay on, in a permanent fashion. ξt initializes as 0 and steps-up to 1 once the event starts.
  3. Ramp Intervention:
    A ramp intervention is a dummy regressor whose values before and during the time of the intervention are zero and whose subsequent values increase linearly thereafter.
    • ξt = 0, if t < time
    • ξt = (t – time), otherwise

Interventions Response/ Transfer Function

We can define a transfer function to explain the effects of introducing exogenous variable (ξt in this case). In general, a typical transfer function model has the form:

υ(B) = ω0 ω(B) Bb/ δ(B) 
ω(B) =1 - ω1B - ω2B2 - … - ωsBs 
δ(B) =1 - δ1B - δ2B2 - … - δrBr 

where υ(B) is the transfer function filter, ω0 is the scaling factor, ω(B) is the sth order numerator polynomial, δ(B) is the rth order denominator polynomial, and b accounts for lagged effects. If r = 0, υ(B) is of finite order; otherwise, it is of infinite order. The overall influence of the intervention event is subsequently referred to as the intervention effect (υ(B)ξt) which describes the promotion’s influence over time.

Links for Further Reading:

Click to access PromotionalAnalysis.pdf

Top 5 AI Chatbot platforms comparison

Chat is the new email! With the emerge of mobile devices, chatbots have garnered amazing traction in the past few years and nowadays, it looks like chatbots have morphed from a “good-to-have” to a “must-have” feature on your website or mobile app. Chatbots have been able to help companies reduce the waiting time for customers and even scale down the size of the customer service teams in some cases. Since you are here, we are assuming that you already have the answers regarding the problems the chatbot is going to solve for you. So, let us see what are some of the most popular chatbot platforms available out there!

Please note that this is not a ranking. Chatbot platforms have been listed considering their ease of use, pricing, features offered and online reviews.

1. Dialogflow

Dialogflow has championed natural language processing and has been a pioneer in the domain. Currently the company is owned by the search engine behemoth Google. Dialogflow is capable of doing many things; a chatbot being one of them. The main idea is to train dialogflow with some samples of the questions you will like it to handle and let it know the corresponding answers. It is intelligent enough to figure out the variations in language and can understand what information is being requested for.

It supports external integrations through API requests and can be integrated with twitter, facebook, skype, slack and many others. Some web development knowledge is necessary to integrate it with external APIs.

Dialogflow is available for free (Standard Edition) with some basic limitations and it is a great way to explore the tool and get familiar with the features. The application includes Text or Google Assistant, Audio input (speech-to-text recognition) and output.
They also have paid plans – Essentials and Plus – under Enterprise Edition, which you can go for once you are production ready. Some features such as Sentiment analysis and Toll-free call are available only with Enterprise edition.

2. Flow AI

Flow.ai has attracted a lot attention in the recent times. Leveraging their platform which is offered as an SaaS, one can quickly build elegant chatbots. The tool includes features like contextual guidance, intent recognition, multiple languages, speech recognition etc. It supports external integration through API calls.

They offer a free plan using which the platform can be explored. The standard plan starts from $19/month.

flow.ai pricing by sessions

Popular platform integrations supported include WhatsApp, Messenger, Telegram, Twitter, Instagram, Amazon Alexa, Slack, Salesforce and Hubspot.

3. Flow XO

Flow XO has been a popular tool among the community. Using the FlowXO platform, one can build simple to quite complex bots. A sequence of questions and corresponding responses can be designed into a flow – hence the name.

It can interact with the outside world through API calls. It also supports good number of external integration plugins for communicating with different tools like ZenDesk, Slack etc. Complex flows can be designed leveraging the Flow XO platform.

Flow XO promotes themselves as a tool which can be used by technical and non-technical people alike. However, people without programming knowledge will be a bit out of their depth trying to design complex bots. The complexity also has a tendency to escalate quickly as the bot increases in size; managing the flow becomes increasingly challenging.

Flow XO offers a free plan through which one can explore the tool. They have a good online community where you can ask questions. Paid plans start from $19 per month and you can add features with topups.

4. Botsify

Botsify is another tool which boasts of a chatbot platform that can be leveraged by non-technical people for creating chatbots. Like many other tools available on the market, Botsify is offered as an SaaS/MSaaS. They support facebook, slack and website integrations.

It comes with pre-built templates which could be leveraged for building the bot. Even though these templates could not be used “as-is”, they provide some guidance and you may even be able to reuse most of the parts and add/modify some components to quickly get your chatbot up and running. Botsify also has a feature which can preserve responses as a form which can later be exported as CSV. This makes it easy to collect and export data through the tool.

Currently they offer two plans – self-serviced and fully managed. The self-services plan start at $50 per month and the fully managed plan starts at $300 per month. There is a 14-day free trial available for Botsify.

5. QnA Maker API

QnA maker API is a REST based API which can be used to send to queries to its knowledge base and the receive a quick reply. The tool is AI powered and it needs to be trained with sample interactions. It has the capability to understand natural language.  The bot can easily be integrated with your website. Some programming knowledge is required in order to use the platform. It is a great tool for quickly building a chat-bot out of something as simple as your website’s FAQ page. The basic plan of QnA maker comes for free. A paid subscription plan starts from $10 per month.

6. Recast API

Recast API boasts to offer a conversational tool capable of understanding natural language. It is currently owned by the ERP giant SAP. It offers a rich UI through which flows can be built. Recast API claims to be a tool for developers by developers. Programming knowledge is required to leverage this tool. It supports external integrations with Twitter, Slack, Skype etc. Recast supports multiple communication channels. It comes packed with features and it can even accept payment which makes it a good candidate for e-commerce websites.

Recast API comes with a free plan. However, in order to get access to the analytics services, a paid subscription is required.

Conclusion

Apart from the ones listed above, some of the other chatbots worth a mention are Chatfuel, Manychat, Pandorabots, TARS, Wit.ai. Finding out what tool suites you best will take some effort on your part to evaluate some of the platforms mentioned here. Once you have evaluated these tools, you will be in a position to select the one which caters to your needs in the best possible way. This guide nevertheless will give you a head start in your exploration for the chatbot platform that suites your requirement!

My Decision: Saying ‘Yes’ to Python

Ok, I’m not exactly a newbie to coding. I took a course in C as a part of my undergrad degree, and have decent experience working with VB & R. So not that bad, right?

The problem is, it’s hard to say yes to a new language. Even harder, if that language is widely popular and used by “experts”, because then you’re starting at the bottle of the barrel. And leaving the comfort zone of your mother tongue (a language you are proficient at) to switch to a new one takes mental conditioning. You have to unlearn to learn, and you have to converse in the new language obsessively. Why? Because if you don’t commit, you won’t be able to get the nuance right. And that means you will not be fluent.

So, starting today (13 Nov 2019) I have to get my hands dirty, start with the basic elements and take the baby steps. I have to switch my mother tongue to snakespeak.