Inferring the presence and abundance of rare species from scarce data

Abundance data are regularly used in ecology, particularly for species monitoring for conservation purposes. However, these count data often display several specific characteristics such as numerous missing data, high variance, and a high proportion of zeros, particularly when monitoring rare species.
We present a new model that aims to both impute missing data and estimate the effect of environmental covariates on species presence and abundance. It is based on the Poisson log-normal model, which offers more flexibility in the variance of counts than a Poisson model. A latent variable is added to account for the overrepresentation of zeros in the data. The imputation of missing data is made possible by assuming that the latent variance matrix has low rank and the inclusion of a series of covariates. 
We demonstrate that the model is the identifiable in the presence of missing data. Since maximum likelihood inference is intractable, we use a variational expectation maximization algorithm to infer the model parameters. We provide an estimate of the asymptotic variance of the estimators, from which we derive prediction intervals for the imputations, an estimate of the temporal trend, and a procedure for detecting a potential change in this trend.
We evaluate our imputations and associated prediction intervals using complete and artificially degraded monitoring data set. We conclude with an illustration on the monitoring of waterbirds in North Africa, for which we provide complete estimates of abundance and probability.