This took me some time. The problem was occurring only on https, not on http. So ruby code, gems, versions were fine.
Turns out I needed this line in my apache config:
RequestHeader set X_FORWARDED_PROTO 'https' env=HTTPS
For nginx it is something similar:
proxy_set_header X-Forwarded-Proto https;