Attention for Inference Compilation

William Harvey, Andreas Munk, Atılım Baydin, Alexander Bergholm, Frank Wood

2022

Abstract

We present a neural network architecture for automatic amortized inference in universal probabilistic programs which improves on the performance of current architectures. Our approach extends inference compilation (IC), a technique which uses deep neural networks to approximate a posterior distribution over latent variables in a probabilistic program. A challenge with existing IC network architectures is that they can fail to capture long-range dependencies between latent variables. To address this, we introduce an attention mechanism that attends to the most salient variables previously sampled in the execution of a probabilistic program. We demonstrate that the addition of attention allows the proposal distributions to better match the true posterior, enhancing inference about latent variables in simulators.

Paper Citation

in Harvard Style

Harvey W., Munk A., Baydin A., Bergholm A. and Wood F. (2022). Attention for Inference Compilation. In Proceedings of the 12th International Conference on Simulation and Modeling Methodologies, Technologies and Applications - Volume 1: SIMULTECH, ISBN 978-989-758-578-4, pages 80-91. DOI: 10.5220/0011277700003274

in Bibtex Style

@conference{simultech22,
author={William Harvey and Andreas Munk and Atılım Baydin and Alexander Bergholm and Frank Wood},
title={Attention for Inference Compilation},
booktitle={Proceedings of the 12th International Conference on Simulation and Modeling Methodologies, Technologies and Applications - Volume 1: SIMULTECH,},
year={2022},
pages={80-91},
publisher={SciTePress},
organization={INSTICC},
doi={10.5220/0011277700003274},
isbn={978-989-758-578-4},
}

in EndNote Style

TY - CONF

JO - Proceedings of the 12th International Conference on Simulation and Modeling Methodologies, Technologies and Applications - Volume 1: SIMULTECH,
TI - Attention for Inference Compilation
SN - 978-989-758-578-4
AU - Harvey W.
AU - Munk A.
AU - Baydin A.
AU - Bergholm A.
AU - Wood F.
PY - 2022
SP - 80
EP - 91
DO - 10.5220/0011277700003274