I end up fixing the problem, I did a few things so I have no idea what was the actual reason for it working now.
- I removed the discs from both back wheels
- Sprayed and wiped both the shoe and inner disc with brake cleaner
- Scrubbed both shoe and inner disc with emery cloth. The disc was slightly glazed, but still had heaps of wear material left.
- The springs that hold the shoe back against the hub assembly weren't attached, the shoe was just free-floating which probably didn't help. I clipped the spring back on to the shoe in all 3 spots.
- I adjusted the shoes so there was no drag on one click and equal drag both wheels on one click, then no movement on 3 clicks.
- Put everything back together.
- Put the car in neutral on a hill and tried the handbrake. Car still rolled backwards easily with 5 clicks. I went back up the hill and let the car roll backwards with the handbrake applied. After each roll the grip got more and more. After about 10 times the car was holding on the hill with only 3 clicks. Car now holds on steep inclines.