Markov Chains: Continuous Time & Poisson Processes
Markov Chains: Continuous Time & Poisson Processes
A continuous-time Markov chain (CTMC) $\{X_t\}_{t\ge 0}$ satisfies the Markov property for all real times. Its dynamics are characterized by a generator matrix (Q-matrix) $Q$ with $Q_{ij}\ge 0$ for $i\ne j$ and $Q_{ii}=-\sum_{j\ne i}Q_{ij}$. The Kolmogorov forward equation $P'(t)=P(t)Q$ yields transition probabilities $P(t)=e^{tQ}$.
The Poisson process is the canonical CTMC: it counts events arriving at rate $\lambda$ with exponentially distributed inter-arrival times. Its defining properties — stationary and independent increments, and $P(N_t=k)=e^{-\lambda t}(\lambda t)^k/k!$ — make it the foundation for queueing theory and risk models.
Generator & Kolmogorov Equations
The generator $Q$ satisfies: $Q_{ij}\ge 0$ $(i\ne j)$, $Q_{ii}=-\sum_{j\ne i}Q_{ij}$. Forward equation: $\frac{d}{dt}P(t)=P(t)Q$. Backward equation: $\frac{d}{dt}P(t)=QP(t)$. Stationary distribution: $\pi Q=0$, $\sum_i\pi_i=1$.
Poisson Process Characterization
A counting process $\{N_t\}$ is Poisson with rate $\lambda$ iff: (i) $N_0=0$; (ii) independent increments; (iii) stationary increments; (iv) $P(N_{t+h}-N_t=1)=\lambda h+o(h)$ and $P(N_{t+h}-N_t\ge 2)=o(h)$. Inter-arrival times $T_k\stackrel{iid}{\sim}\text{Exp}(\lambda)$ and $N_t\sim\text{Poisson}(\lambda t)$.
Example 1 — M/M/1 Queue
Arrivals $\sim\text{Poisson}(\lambda)$, service $\sim\text{Exp}(\mu)$. Generator: $Q_{n,n+1}=\lambda$, $Q_{n,n-1}=\mu$ $(n\ge 1)$. Stationary: $\pi_n=(1-\rho)\rho^n$ where $\rho=\lambda/\mu<1$. Mean queue length $E[N]=\rho/(1-\rho)$.
Example 2 — Superposition & Thinning
If $N^{(1)},\ldots,N^{(k)}$ are independent Poisson processes with rates $\lambda_1,\ldots,\lambda_k$, their superposition is Poisson($\sum\lambda_i$). Conversely, independently thinning a Poisson($\lambda$) process with probability $p$ yields Poisson($p\lambda$).
Practice
- Verify $E[N_t]=\lambda t$ and $\text{Var}(N_t)=\lambda t$ for a Poisson process.
- For M/M/1, derive $\pi_n$ by solving the balance equations.
- Compute $P(N_1>2)$ when $\lambda=1$.
Show Answer Key
1. $N_t\sim\text{Poisson}(\lambda t)$, so $E[N_t]=\lambda t$ and $\text{Var}(N_t)=\lambda t$ by the Poisson distribution properties.
2. Balance equations $\lambda\pi_n=\mu\pi_{n+1}$ give $\pi_n=(1-\rho)\rho^n$ where $\rho=\lambda/\mu<1$.
3. $P(N_1>2)=1-P(N_1\le2)=1-e^{-1}(1+1+\tfrac{1}{2})=1-\tfrac{5}{2}e^{-1}\approx0.0803$.