I got some clear skies and tried out the tracker again. I’ve added a finder scope to help align the tracker better and I think it helped. I tried out the Pleiades since they are relatively easy to find, and the Moon was too bright in the other part of the sky visible from my backyard.
In this gif you can see how the house moves from shot to shot, but the stars don’t move much in the frame. They do move a little bit, but it’s much better than before.
I need to improve focusing, and I am still having some wobble where the mount meets the tripod, but it’s a good start.