User Agents and Unintended DDoS
After my surge in visitors during the last couple of days, I noticed a curious anomaly in my web server logs. A very particular user agent appeared over and over again, repeatedly accessing the same URL up to a dozen times per second. Each time, the entire HTML contents of the accessed page gets transferred but no other assets.
The user agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_11_1) AppleWebKit/601.2.4 (KHTML, like Gecko) Version/9.0.1 Safari/601.2.4 facebookexternalhit/1.1 Facebot Twitterbot/1.0
This is at times followed up by this user agent:
Mozilla/5.0 (Macintosh; Intel Mac OS X 10_15_4) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/13.1 Safari/605.1.15
My research yielded speculation about this being the iOS Messages.app user agent string when generating previews. My tests show that this is not correct. It may have been at some time, I couldn’t say. Currently, the app uses
com.apple.WebKit.Networking/8614.2.9.0.10 CFNetwork/1399 Darwin/22.1.0
as the user agent string. Also, it tends not to make more than one access per requested asset, like favicons.
The most likely cause is a recent GitHub issue I discovered.
https://github.com/mozilla-mobile/firefox-ios/issues/12617
Apparently Firefox on iOS sends a torrent of unnecessary requests on each interaction:
The behavior is increasingly worse if you have multiple tabs open. Refreshing a single page also generates at least a couple of these extra requests to the next tab’s URL for whatever reason. If all tabs are loaded, switching between them also seems to generate requests. Closing & reopening the app normally before it is terminated sends requests to seemingly 2 random tabs. Terminating the app & reopening with multiple tabs open seemingly sends numerous requests to each of the open tabs. The behavior differs some depending on which tab was last open. It seems like being parked on the 1st tab makes the most requests in any case, which maybe is a clue as to what’s going on.
If Firefox is indeed the source of this DDoS-like behaviour, that’d be some seriously shoddy engineering. It’s therefore possible that Firefox Mobile brought down my site. At the very least, the browser uses misleading user agents.
To this moment, I get dozens of identical requests from a wide range of IPs every second. For the time being, I have instructed Nginx to return HTTP code 429 for this user agent string.