---
title: "Distributions Provided by the **mniw** Package"
author: "Martin Lysy"
date: "`r Sys.Date()`"
output:
  rmarkdown::html_vignette:
    toc: true
    keep_md: false
vignette: >
  %\VignetteIndexEntry{Distributions provided by mniw}
  %\VignetteEngine{knitr::rmarkdown}
  %\VignetteEncoding{UTF-8}
---

<!-- <script type="text/x-mathjax-config"> -->
<!-- MathJax.Hub.Config({ -->
<!--     TeX: { -->
<!-- 	Macros: { -->
<!-- 	    bm: ["{\\boldsymbol{#1}}",1], -->
<!-- 		tx: ["{\\textrm{#1}}",1], -->
<!-- 	    tfrac: ["{\\textstyle{\\frac{#1}{#2}}}",2], -->
<!-- 		rv: ["{#2_{#1},\\ldots,#2_{#3}}",3,"1"], -->
<!-- 		iid: ["{\\overset{\\;\\tx{iid}\\;}{\\sim}}"], -->
<!-- 		ind: ["{\\overset{\\:\\tx{ind}\\:}{\\sim}}"], -->
<!-- 	    XX: ["{\\bm{X}}"], -->
<!-- 	    YY: ["{\\bm{Y}}"], -->
<!-- 	    LL: ["{\\bm{L}}"], -->
<!-- 	    PPs: ["{\\bm{\\Psi}}"], -->
<!-- 	    ZZ: ["{\\bm{Z}}"], -->
<!-- 	    N: ["{\\mathcal{N}}"], -->
<!-- 	    LLa: ["{\\bm{\\Lambda}}"], -->
<!-- 	    SSi: ["{\\bm{\\Sigma}}"], -->
<!-- 	    GGa: ["{\\bm{\\Gamma}}"], -->
<!-- 	    UU: ["{\\bm{U}}"], -->
<!-- 	    VV: ["{\\bm{V}}"], -->
<!-- 		aa: ["{\\bm{a}}"], -->
<!-- 		bb: ["{\\bm{b}}"], -->
<!-- 	    xx: ["{\\bm{x}}"], -->
<!-- 		yy: ["{\\bm{y}}"], -->
<!-- 		bbe: ["{\\bm{\\beta}}"], -->
<!-- 	    mmu: ["{\\bm{\\mu}}"], -->
<!-- 		kka: ["{\\bm{\\kappa}}"], -->
<!-- 		lla: ["{\\bm{\\lambda}}"], -->
<!-- 		TTh: ["{\\bm{\\Theta}}"], -->
<!-- 	    GG: ["{\\bm{G}}"], -->
<!-- 	    OOm: ["{\\bm{\\Omega}}"], -->
<!-- 		bz: ["{\\bm{0}}"], -->
<!-- 		wish: ["{\\tx{Wish}}"], -->
<!-- 		iwish: ["{\\tx{InvWish}}"], -->
<!-- 		MN: ["{\\tx{MatNorm}}"], -->
<!-- 		MT: ["{\\tx{MatT}}"], -->
<!-- 		mniw: ["{\\tx{MNIW}}"], -->
<!-- 		re: ["{\\tx{RxNorm}}"] -->
<!-- 	} -->
<!--     } -->
<!-- }); -->
<!-- </script> -->

\newcommand{\bm}[1]{\boldsymbol{#1}}
\newcommand{\tx}[1]{\textrm{#1}}
\newcommand{\tfrac}[2]{\textstyle{\frac{#1}{#2}}}
\newcommand{\rv}[3][1]{#2_{#1},\ldots,#2_{#3}}
\newcommand{\iid}{\overset{\;\tx{iid}\;}{\sim}}
\newcommand{\ind}{\overset{\:\tx{ind}\:}{\sim}}
\newcommand{\XX}{{\bm{X}}}
\newcommand{\YY}{{\bm{Y}}}
\newcommand{\LL}{{\bm{L}}}
\newcommand{\PPs}{{\bm{\Psi}}}
\newcommand{\ZZ}{{\bm{Z}}}
\newcommand{\N}{\mathcal{N}}
\newcommand{\LLa}{{\bm{\Lambda}}}
\newcommand{\SSi}{{\bm{\Sigma}}}
\newcommand{\GGa}{{\bm{\Gamma}}}
\newcommand{\UU}{{\bm{U}}}
\newcommand{\VV}{{\bm{V}}}
\newcommand{\aa}{{\bm{a}}}
\newcommand{\bb}{{\bm{b}}}
\newcommand{\xx}{{\bm{x}}}
\newcommand{\yy}{{\bm{y}}}
\newcommand{\bbe}{{\bm{\beta}}}
\newcommand{\mmu}{{\bm{\mu}}}
\newcommand{\kka}{{\bm{\kappa}}}
\newcommand{\lla}{{\bm{\lambda}}}
\newcommand{\TTh}{{\bm{\Theta}}}
\newcommand{\GG}{{\bm{G}}}
\newcommand{\OOm}{{\bm{\Omega}}}
\newcommand{\bz}{{\bm{0}}}
\newcommand{\wish}{\tx{Wish}}
\newcommand{\iwish}{\tx{InvWish}}
\newcommand{\MN}{\tx{MatNorm}}
\newcommand{\MT}{\tx{MatT}}
\newcommand{\mniw}{\tx{MNIW}}
\newcommand{\re}{\tx{RxNorm}}


## Wishart Distribution

The Wishart distribution on a random positive-definite matrix $\XX_{q\times q}$ is is denoted $\XX \sim \wish(\PPs, \nu)$, and defined as $\XX = (\LL \ZZ)(\LL \ZZ)'$, where:

- $\PPs_{q\times q} = \LL\LL'$ is the positive-definite matrix scale parameter,
- $\nu > q$ is the shape parameter, 
- $\ZZ_{q\times q}$ is a random lower-triangular matrix with elements

    \[
    Z_{ij} \begin{cases} 
           \iid \N(0,1) & i < j \\ 
           \ind \chi^2_{(\nu-i+1)} & i = j \\
           = 0 & i > j.
           \end{cases}
    \]

The log-density of the Wishart distribution is
\[
\log p(\XX \mid \PPs, \nu) = -\tfrac{1}{2} \left[\mathrm{tr}(\PPs^{-1} \XX) + (q+1-\nu)\log |\XX| + \nu \log |\PPs|  + \nu q \log(2) + 2 \log \Gamma_q(\tfrac \nu 2)\right],
\]
where  $\Gamma_n(x)$ is the multivariate Gamma function defined as
\[
\Gamma_n(x) = \pi^{n(n-1)/4} \prod_{j=1}^n \Gamma\big(x + \tfrac 1 2 (1-j)\big).
\]

### Inverse-Wishart Distribution

The Inverse-Wishart distribution $\XX \sim \iwish(\PPs, \nu)$ is defined as $\XX^{-1} \sim \wish(\PPs^{-1}, \nu)$.  Its log-density is given by
\[
\log p(\XX \mid \PPs, \nu) = -\tfrac 1 2 \left[\mathrm{tr}(\PPs \XX^{-1}) + (\nu+q+1) \log |\XX| - \nu \log |\PPs| + \nu q \log(2) + 2 \log \Gamma_q(\tfrac \nu 2)\right].
\]

### Properties

If $\XX_{q\times q} \sim \wish(\PPs,\nu)$, the for a nonzero vector $\aa \in \mathbb R^q$ we have
\[
\frac{\aa'\XX\aa}{\aa'\PPs\aa} \sim \chi^2_{(\nu)}.
\]

## Matrix-Normal Distribution

The Matrix-Normal distribution on a random matrix $\XX_{p \times q}$ is denoted $\XX \sim \MN(\LLa, \SSi_R, \SSi_C)$, and defined as $\XX = \LL\ZZ \UU + \LLa$, where:

- $\LLa_{p \times q}$ is the mean matrix parameter,
- ${\SSi_R}_{p \times p} = \LL\LL'$ is the row-variance matrix parameter,
- ${\SSi_C}_{q \times q} = \UU'\UU$ is the column-variance matrix parameter,
- $\ZZ_{q\times q}$ is a random matrix with $Z_{ij} \iid \N(0,1)$.

The log-density of the Matrix-Normal distribution is
\[
\log p(\XX \mid \LLa, \SSi_R, \SSi_C) = -\tfrac 1 2 \left[\mathrm{tr}\big(\SSi_C^{-1}(\XX-\LLa)'\SSi_R^{-1}(\XX-\LLa)\big) + \nu q \log(2\pi) + \nu \log |\SSi_C| + q \log |\SSi_R|\right].
\]

### Properties

If $\XX_{p \times q} \sim \MN(\LLa, \SSi_R, \SSi_C)$, then for nonzero vectors $\aa \in \mathbb R^p$ and $\bb \in \mathbb R^q$ we have
\[
\aa' \XX \bb \sim \N(\aa' \LLa \bb, \aa'\SSi_R\aa \cdot \bb'\SSi_C\bb).
\]

## Matrix-Normal Inverse-Wishart Distribution

The Matrix-Normal Inverse-Wishart Distribution on a random matrix $\XX_{p \times q}$ and random positive-definite matrix $\VV_{q\times q}$ is denoted $(\XX,\VV) \sim \mniw(\LLa, \SSi, \PPs, \nu)$, and defined as
\[
\begin{aligned}
\XX \mid \VV & \sim \MN(\LLa, \SSi, \VV) \\
\VV & \sim \iwish(\PPs, \nu).
\end{aligned}
\]

### Properties

The MNIX distribution is conjugate prior for the multivariable response regression model
\[
\YY_{n \times q} \sim \MN(\XX_{n\times p} \bbe_{p \times q}, \VV, \SSi).
\]
That is, if $(\bbe, \SSi) \sim \mniw(\LLa, \OOm^{-1}, \PPs, \nu)$, then
\[
\bbe, \SSi \mid \YY \sim \mniw(\hat \LLa, \hat \OOm^{-1}, \hat \PPs, \hat \nu),
\]
where
\[
\begin{aligned}
\hat \OOm & = \XX'\VV^{-1}\XX + \OOm 
&
\hat \PPs & = \PPs + \YY'\VV^{-1}\YY + \LLa'\OOm\LLa - \hat \LLa'\hat \OOm \hat \LLa
\\
\hat \LLa & = \hat \OOm^{-1}(\XX'\VV^{-1}\YY + \OOm\LLa)
&
\hat \nu & = \nu + n.
\end{aligned}
\]

## Matrix-t Distribution

The Matrix-$t$ distribution on a random matrix $\XX_{p \times q}$ is denoted $\XX \sim \MT(\LLa, \SSi_R, \SSi_C, \nu)$, and defined as the marginal distribution of $\XX$ for $(\XX, \VV) \sim \mniw(\LLa, \SSi_R, \SSi_C, \nu)$.  Its log-density is given by
\[
\begin{aligned}
\log p(\XX \mid \LLa, \SSi_R, \SSi_C, \nu) 
& = -\tfrac 1 2 \Big[(\nu+p+q-1)\log | I + \SSi_R^{-1}(\XX-\LLa)\SSi_C^{-1}(\XX-\LLa)'| \\
& \phantom{= -\tfrac 1 2 \Big[} + q \log |\SSi_R| + p \log |\SSi_C| \\
& \phantom{= -\tfrac 1 2 \Big[} + pq \log(\pi) - \log \Gamma_q(\tfrac{\nu+p+q-1}{2}) + \log \Gamma_q(\tfrac{\nu+q-1}{2})\Big].
\end{aligned}
\]

### Properties

If $\XX_{p\times q} \sim \MT(\LLa, \SSi_R, \SSi_C, \nu)$, then for nonzero vectors $\aa \in \mathbb R^p$ and $\bb \in \mathbb R^q$ we have
\[
\frac{\aa'\XX\bb - \mu}{\sigma} \sim t_{(\nu -q + 1)},
\]
where 
\[
\mu = \aa'\LLa\bb, \qquad \sigma^2 = \frac{\aa'\SSi_R\aa \cdot \bb'\SSi_C\bb}{\nu - q + 1}.
\]

## Random-Effects Normal Distribution

Consider the multivariate normal distribution on $q$-dimensional vectors $\xx$ and $\mmu$ given by
\[
\begin{aligned}
\xx \mid \mmu & \sim \N(\mmu, \VV) \\
\mmu & \sim \N(\lla, \SSi).
\end{aligned}
\]
The random-effects normal distribution is defined as the posterior distribution $\mmu \sim p(\mmu \mid \xx)$, which is given by
\[
\mmu \mid \xx \sim \N\big(\GG(\xx-\lla) + \lla, \GG\VV\big), \qquad \GG = \SSi(\VV + \SSi)^{-1}.
\]
The notation for this distribution is $\mmu \sim \re(\xx, \VV, \lla, \SSi)$.

## Hierarchical Normal-Normal Model

The hierarchical Normal-Normal model is defined as
\[
\begin{aligned}
\yy_i \mid \mmu_i, \bbe, \SSi & \ind \N(\mmu_i, \VV_i) \\
\mmu_i \mid \bbe, \SSi & \iid \N(\xx_i'\bbe, \SSi) \\
(\bbe, \SSi) & \sim \mniw(\LLa, \Omega^{-1}, \PPs, \nu),
\end{aligned}
\]
where:

- ${\yy_i}_{q\times 1}$ is the response vector for subject $i$,
- ${\mmu_i}_{q\times 1}$ is the random effect for subject $i$,
- ${\VV_i}_{q\times q}$ is the error variance for subject $i$,
- ${\xx_i}_{p\times 1}$ is the covariate vector for subject $i$,
- $\bbe_{p \times q}$ is the random-effects coefficient matrix,
- $\SSi_{q \times q}$ is the random-effects error variance.

Let $\YY_{n\times q} = (\rv \yy n)$, $\XX_{n\times p} = (\rv \xx n)$, and $\TTh_{n \times q} = (\rv \mmu n)$.  If interest lies in the posterior distribution $p(\TTh, \bbe, \SSi \mid \YY, \XX)$, then a Gibbs sampler can be used to cycle through the following conditional distributions:
\[
\begin{aligned}
\mmu_i \mid \bbe, \SSi, \YY, \XX & \ind \re(\yy_i, \VV_i, \xx_i'\bbe, \SSi) \\
\bbe, \SSi \mid \TTh, \YY, \XX & \sim \mniw(\hat \LLa, \hat \OOm^{-1}, \hat \PPs, \hat \nu),
\end{aligned}
\]
where $\hat \LLa$, $\hat \OOm$, $\hat \PPs$, and $\hat \nu$ are obtained from the MNIW conjugate posterior formula with $\YY \gets \TTh$.
