PASSION The greatest waste is that of human talent. I sincerely
believe in the power of the human spirit and spend my free time developing
people through mentoring projects that I envision and design and am engaged
in other service-related work. EDUCATION & AWARDS
|
|
|
Fig. 1 Dual Regime behavior: |
____________________________________________________________
Modeling this sort of behavior is an important area since
system identification in the presence of non-stationary noise is a relatively
unexplored area. We achieve this via superimposing a finite-state Markov
chain on top of a linear disturbance model.
The resulting model is termed a Markov Jump Linear System and is more
expressive in capturing reality. Consequently, the identification of this
concatenated model is of interest and we show that using stationary models
for process identification in the event of the aforementioned dynamics, as
might be done by practising control engineers faced
with real-world signals, results in unsatisfactory performance.
1.1 Application area (1): Hybrid cybernetic model-based
simulation of continuous production of
lignocellulosic ethanol: Rejecting abruptly changing feed
conditions
Fermenting
various sugars derived from lignocellulosic biomass
promises to be attractive for producing ethanol, an important alternative
fuel. Diversity of lignocellulosic biomass sources
and pre-processing variations mean entering sugars are expected to experience
large, though infrequent, changes. Recent developments in hybrid cybernetic
modeling allow efficient in-silico studies. This enables studying sequential
linearization-based model predictive control for ensuring high productivity
and conversion, for a chemostat seeded with yeast
capable of co-fermentation. An appropriate controlled variable (conversion)
and control formulation are ascertained. Also, the proposed hidden-Markov
disturbance model, capable of describing the aforesaid changes, results in
closed-loop performance superior to the typical integrated white-noise
assumption.
1.2 Proposed
Application area (2): Applying HMM-based frameworks for continuous blood
glucose monitoring
For diabetic patients, blood glucose levels rise rapidly (in the
absence of an insulin bolus) upon the consumption of a meal. As such, a
mechanism for continuous blood-glucose monitoring, part of an automatic Meal
Detection
Algorithm
(MDA), forms an integral aspect of a comprehensive healthcare solution for
diabetic patients. Continuous Glucose Monitoring (CGM) has the added benefit
of informing the patient of either hypo-or-hyper glycemia.
Linear
state-space models coupled with Kalman filtering
constitute a popular means for achieving blood glucose monitoring. These
state-space models are typically based on double-or-triple integrators.
Neither of the double-and-triple integrator models are expected to be
faithful descriptions of the pre-and post-prandial blood glucose levels, which behave like
stationary stochastic signals fluctuating around a mean level before
consuming the meal leading to a non-stationary, ramp-like increase right
after.
One obvious short-coming of a Kalman filter in the
present context is the absence of a dynamic observer gain that adapts to the
different (pre and post meal) regimes. A well-tuned Kalman
filter, with its attendant observer gain typically needs to strike a fine
balance between robustness against measurement noise before meal consumption
(thereby minimizing false positives) and sensitivity towards actual, post-prandial blood glucose spikes.
In
light of such regime-like behavior, the potential effectiveness of an
HMM-based modeling frameworking for the purpose of
GCM and MDA is explored.
2. Model-free
Control of Dynamical Systems
Loosely speaking, reinforcement learning can be interpreted as direct
adaptive control. That is, the parameters of a policy (or controller) are
learnt directly, bypassing the need for the model of the plant/ system of
concern. Q-learning is a specific
method within the family of reinforcement learning techniques, and has its
roots in dynamic programming. The latter provides an avenue for optimal
decision making in the presence of uncertainty. Below, we look at specific
case studies where the Q-learning/
reinforcement learning approach may change the way practioners
approach process control.
2.1 Linear dynamical systems
Reinforcement learning where decision-making agents learn optimal
policies through environmental interactions is an attractive paradigm for
model-free, adaptive controller design. However, results for systems with
continuous state and action variables are rare. We present convergence
results for optimal linear-quadratic control of discrete-time linear
stochastic systems. This work can be viewed as a generalization of a previous
work on deterministic linear systems. Key differences between the algorithms
for deterministic and stochastic systems are highlighted through examples.
The usefulness of the algorithm is demonstrated through a nonlinear chemostat bioreactor case study.
____________________________________________________________
|
|
a) Chemostat output (substrate
concentration) vs. time in the absence of control |
____________________________________________________________
2.2 Markov Jump Linear Systems
MJLS's represent a useful non-linear framework
for the description of systems which switching probabilistically. The
switching rules are governed by a Hidden Markov chain. Applications include
disturbance modeling. We are looking into model-free methods for the control
of MJLS's. This is beneficial, since the
identification of MJLS, normally achieved by maximum likelihood methods,
involve local minima.
2.3 Nonlinear
systems
Black-box modeling for non-linear systems suffers from the lack of a
systematic approach. This has an adverse impact on the extent to which
non-linear model-based control techniques are adopted. Here, we look towards Q-learning as a potential solution.
Journal PAPERS
W. C. Wong
and J. H. Lee, "A Reinforcement-Learning Scheme for Direct Adaptive
Optimal Control of Linear Stochastic Systems”. Optimal Control, Applications and Methods (in review).
W. C.
Wong and J. H. Lee, "Realistic Disturbance Modeling using Hidden Markov
Models: Applications in Model-based Control". Journal of Process Control (in review)
Wong, W. C. and J. H. Lee. “Hybrid Cybernetic Model-based
Simulation of Continuous Production of Lignocellulosic
Ethanol: Rejecting Abruptly Changing Feed Conditions”. Control Engineering Practice (in
review)
CONFERENCE PAPERS
W. C.
Wong and J. H. Lee, "Disturbance Modeling for Process Control via Hidden
Markov Models," in 8th International Symposium on Dynamics and
Control of Process Systems.
Cancun, Mexico, 2007.
W. C.
Wong and J. H. Lee, "A Hidden Markov Disturbance Model for Offset-Free
Linear Model Predictive Control," in 17th International Federation of
Automatic Control World Congress.
Seoul, South Korea, 2008.
Wong, W. C. and J. H. Lee (2008). “A Reinforcement
Learning-Based Scheme for Adaptive Optimal Control of Linear Stochastic
Systems.” American Control
Conference,
Wong, W. C. and J. H. Lee (2008).
“Control of a Fermentor in the Presence of
Abruptly-Changing Feed Conditions” Advanced Control of Industrial
Processes - International Conference.
PRESENTATIONS
W. C. Wong and J. H. Lee,
"Construction of Disturbance Models using Hidden Markov
Identification," in American Institute of Chemical Engineers Annual
Meeting. Cincinnati, Ohio, 2005.
W. C. Wong and J. H. Lee, "Disturbance
Modeling Via Hidden Markov Techniques - An Extension," in American
Institute of Chemical Engineers Annual Meeting,
Last Modified ,LIDCUS© 2009