Moveable looks like a super full-featured library for this kind of thing https://daybrush.com/moveable/

Check out https://konvajs.org/docs/sandbox/Canvas_Scrolling.html

Maybe this: https://interactjs.io/