We study how offline data can help online learning in stochastic linear bandits.
We focus on partially covered offline data, where only $k$ of the $d$ feature directions are poorly covered.
Existing analyses of algorithms based on phased elimination and optimism give regret bounds of order $\widetilde O(\sqrt{dkT})$ in this regime.
We show that this dependence arises from loose control of the reward prediction error, and obtain tighter bounds using concentration of Lipschitz functions of Gaussian vectors.
Based on this analysis, we develop a phased elimination algorithm with regret $\widetilde O(k\sqrt T)$.
For the unit Euclidean ball, we prove a matching regret lower bound under partial coverage, as well as matching upper and lower bounds under uniform offline coverage.
We also study pure exploration with offline data, providing algorithms and matching upper and lower bounds on simple regret for fixed-design algorithms and also for adaptive algorithms under partial and uniform coverage of offline data.