Probabilistic Programming Languages (PPLs) are a powerful tool in machine learning, allowing highly expressive generative models to be expressed succinctly. They couple complex inference algorithms, implemented by the language, with an expressive modelling language that allows a user to implement any computable function as the generative model.

Such languages are usually implemented on top of existing high level programming languages and do not make use of hardware accelerators. PPLs that do make use of accelerators exist, but restrict the expressivity of the language in order to do so.

In this extended abstract, we present a language and toolchain that generates highly efficient code for both CPUs and GPUs. The language is functional in style, and the toolchain is built on top of LLVM. Our implementation uses delimited continuations on CPU to perform inference, and custom CUDA codes on GPU.

We obtain significant speed ups across a suite of PPL workloads, compared to other state of the art approaches on CPU. Furthermore, our compiler can also generate efficient code that runs on CUDA GPUs.