Dynamic System Control using a Static Generative Model

By

Chinese version: 静态生成式模型的动态系统控制算法 - 知乎 (zhihu.com)

Recap from the last blog post, we formulated the control problem as a generative model in the following fashion:

\[[v,w,x] = g(z), \quad z \sim \mathbb{D},\]

in which \(v\) is the collection of control objectives to be minimized, \(w\) contains the output control signals, and \(x\) represents the input signals that are thought to offer information to the control problem. 

This model is static for the lack of explicit formulation of the continuous evolution of the system states \([v,w,x]\). It is possible to change the model to a dynamic one, but in this blog post we show an algorithm that can offer dynamic control using a static model.

\[ \begin{align} & \text{Initialize } \epsilon_0 \text{ and } [v_0, x_0] = r (u_0, w_0) \\ & \text{for } t = 1 \text{ to } \infty \text{ do} \\ & \quad \text{Initialize } [v', w', x'] = [v_{t-1}, w_{t-1}, x_{t-1}] \\ & \quad \text{While } v_{t-1} - v' >  \epsilon_{t-1} \text{ and } v' > v_{\text{min}} \text{ do} \\ & \quad \quad z' = \underset{z}{\text{argmin}} \| g(z) - [v' - \epsilon_{t-1}, w', x'] \| \\ & \quad \quad [v', w', x'] = g(z') \\ & \quad w_t = w' \\ & \quad [v_t, x_t] = r (u_t, w_t) \\ & \quad \text{if } \|[v_t, x_t] - [v', x'] \| < \delta_{\text{min}} \text{ then} \\ & \quad \quad \epsilon_t = \gamma_{\text{upscale}} \cdot \epsilon_{t-1} \\ & \quad \text{else if } \|[v_t, x_t]- [v', x'] \| > \delta_{\text{max}} \text{ then} \\ & \quad \quad \epsilon_t = \gamma_{\text{downscale}} \cdot \epsilon_{t-1} \end{align} \]

In the algorithm above, \([v, x] = r(u, w)\) represents the objectives and the input signals obtained from the system after applying the control signals \(w\), with internal unobserved states \(u\). Intuitively, the more we can observe the signals from \(u\) i.e., moving it from \(u\) to \(x\), the better we might be able to disentangle the uncertainty of the controlled system. This is an open area for theoretical research, but the algorithm does not need \(u\) as long as it can run the controlled system to obtain \([v, x]\).

The algorithm attempts to reduce the objectives \(v_{t-1}\) by an amount \(\epsilon_{t-1}\), and then verify whether it matches with what the system returns. If it does not, in the next step it will try to use a smaller \(\epsilon_t\), the amount of objective reduction in the next step. This mechanism is controlled by a set of parameters satisfying \(\delta_{\text{min}} \leq \delta_{\text{max}}\) and \(0 < \gamma_{\text{downscale}} < 1 < \gamma_{\text{upscale}} \).

Since the loop on \([v',w',x']\) is a fixed process with inputs \([v_{t-1}, w_{t-1}, x_{t-1}]\) and \(\epsilon_{t-1}\), the uncertainties in the outputs are disambiguited. As a result, we can amortize the process as \(w_t = h(v_{t-1}, w_{t-1}, x_{t-1}, \epsilon_{t-1})\). Brandon Amos presented a good mathematical tutorial on amortization.

Finally, all the samples \([v_t, w_t, x_t] \) obtained from the controlled system \(r\) can be brought back as training data to improve the model \(g\), realizing what is perhaps the simplest form of System 2 to System 1 knowledge learning. A philosophical discussion on both amortization and System-2-to-1 learning exists in Yann LeCun's position paper.

Comments

Post a Comment

Popular posts from this blog

Serving Llama-2 7B using llama.cpp with NVIDIA CUDA on Ubuntu 22.04

By
This blog post is a step-by-step guide for running Llama-2 7B model using llama.cpp, with NVIDIA CUDA and Ubuntu 22.04.  llama.cpp is an C/C++ library for the inference of Llama/Llama-2 models. It has grown insanely popular along with the booming of large language model applications. Throughout this guide, we assume the user home directory (usually /home/username) is the working directory. Install NVIDIA CUDA To start, let's install NVIDIA CUDA on Ubuntu 22.04. The guide presented here is the same as the CUDA Toolkit download page provided by NVIDIA, but I deviate a little bit by installing CUDA 11.8 instead of the latest version. At the time of writing, PyTorch 2.0 stable is released for CUDA 11.8 and I find it convenient to keep my deployed CUDA version in sync with that. $ wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2204/x86_64/cuda-keyring_1.1-1_all.deb $ sudo dpkg -i cuda-keyring_1.1-1_all.deb $ sudo apt update $ sudo apt install cuda-11-8 After
Read more

A Perplexity Benchmark of llama.cpp

By
Without further ado, here are the results (explanations and discussions later): Table 1: Perplexity on wikitext-2 test set.   Model \ Quantization q4_0 q4_1 q5_0 q5_1 q8_0 fp16 llama-7b 6.157 6.0915 5.9846 5.948 5.9063 5.68 llama-13b 5.385 5.3608 5.285 5.2702 5.2547 5.09 llama-30b 4.2707 - - - - 4.1 alpaca-30b 4.4521 - - - - - llama-2-7b 5.9675 6.0398 5.8328 5.8435 5.7897 - llama-2-7b-chat 7.7641 7.7853 7.5055 7.5392 7.5014 - llama-2-13b 5.2172 5.2115 5.1343 5.1289 5.1005 - llama-2-13b-chat 6.62
Read more

A Peek into Generative Control

By
Chinese version:  一窥生成式控制论 - 知乎 (zhihu.com) As a peek into some of my most recent work in applying generative modeling into various industries, I'm proudly presenting an idea that illustrates how powerful generative models can be for industrial controling. My team and I are working to rapidly expand this idea into many different areas, and we still haven't seen its limit yet. A General Formulation of Generative Models \[y = g(z), \quad z \sim \mathbb{D}\] where \(y\) is a sample from data, \(g\) is a generator neural network written as a function, and \(z\) follows a pre-defined distribution \(\mathbb{D}\). It is easy to identify that generative adversarial networks (GANs) naturally result in models in the above fashion. For autoregressive language models (transformers or otherwise), \(z\) is the concatenation of all the random sampling variables used during the decoding process. For stable diffusion, \(z\) is the concatenation of all the noise added in the diffusion process. A
Read more