0

I have found some cheap SEO pages, whose only purpose is to promote some other sites by feeder links. On Chrome and Safari and Firefox, they show the SEO web page, but on Kiwi Web Browser on my Android phone, the page is redirected to "Congratulations! You may be our next lucky winner!". How does that happen? Why does it only redirect on one web browser?

I am interested in filtering out these sites, and it would be helpful if I can expose the redirection artificially, since the SEO pages don't have any identifiable features.

Update: I noticed the redirection happened only when clicking on a spam link I emailed to myself, and only on the gmail app on my phone. So I have added 'referer': 'com.google.android.gm' to the requests() header. I don't know if that solves the issue, but it can't hurt.

mekineer
  • 11
  • 5
  • Those cheap SEO page might be shady and boost their revenue by sending some browsers to these shady raffle pages.. – Tero Kilkanen Oct 08 '20 at 07:00
  • can you give some more information on what you mean by "filtering out these sites" – Tom Oct 08 '20 at 16:09
  • I'm looking through search results that use few resources, or do not use advertising. I'm going for 90's retro web. You can see the whitelist in param.json. https://github.com/mekineer/single_domain_search In the future I will be using blacklists as well. But towards the end of the search results, there is a lot of garbage, which leads to false positives. If only I knew the secret to the Kiwi Web Browser. – mekineer Oct 08 '20 at 19:15
  • Have you tried browsing to a site that tells you the user-agent string for the Kiwi web browser and sending that? e.g. https://www.whatismybrowser.com/detect/what-is-my-user-agent ... also do you have an example of a site that has different behaviour for each client? – Tom Oct 08 '20 at 19:35
  • Better, I used kiwi://version, and it shows me the user agent, plus a ton of command line options I will be trying out in requests(). I'm going to be taking a break though. – mekineer Oct 09 '20 at 02:42
  • Sorry I didn't get back sooner. What finally worked, was using seleniumwire and driver.header_overrides = {'Referer': 'com.google.android.gm'} Apparently, it was clicking the link from my gmail app, rather than using Kiwi. I reproduced it on Chrome. – mekineer Oct 17 '20 at 22:16

2 Answers2

0

The web server probably does some sort of User-Agent header inspection and redirects based on that.

Using apache, something like the following could be used to conditionally redirect specific clients to different locations:

RewriteEngine On
RewriteCond %{HTTP_HOST} example.com$ [NC]
RewriteCond %{HTTP_USER_AGENT} Firefox [OR]
RewriteCond %{HTTP_USER_AGENT} Mozilla
RewriteRule ^(.*)$ http://other.com/$1 [L,R=301]

Or alternatively, the application that is serving the page could return different content based on user-agent, without ever redirecting. e.g. with PHP:

https://www.php.net/manual/en/function.get-browser.php

$u_agent = $_SERVER['HTTP_USER_AGENT'];

if (preg_match('/Firefox/i', $u_agent)) {
    $output = 'some list of SEO urls';
}
elseif (preg_match('/Android/i', $u_agent)) {
    $output = '"Congratulations! You may be our next lucky winner!"';
}

To investigate whether some URL is returning different results depending on the user-agent is, you can spoof the user-agent string you send using curl or some http client library that support setting request headers.

For example to pretend to be "firefox"...

curl -A "Mozilla/5.0 (Windows NT 6.1; Win64; x64; rv:59.0) Gecko/20100101 Firefox/59.0" \
    https://example.com/some_web_page.html
Tom
  • 10,886
  • 5
  • 39
  • 62
  • It definitely redirects. I end up at a different domain. – mekineer Oct 08 '20 at 01:52
  • I'm currently using: chrome_options = webdriver.ChromeOptions() chrome_options.add_argument('--user-agent=Mozilla/5.0 (Linux; Android 8.1.0; SM-J701F) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/77.0.3865.92 Mobile Safari/537.36') – mekineer Oct 08 '20 at 01:54
  • I just checked, and Kiwi on Android claims to be Linux. Will try that and report back. – mekineer Oct 08 '20 at 01:58
  • Looking into https://requests.readthedocs.io/en/latest/user/advanced/ – mekineer Oct 08 '20 at 07:57
  • No luck looking into https://requests.readthedocs.io/en/latest/user/advanced/. The sites didn't have any consistency in the headers, and verify=False may have helped, but not much. – mekineer Oct 08 '20 at 15:31
  • Is using requests() with a different user-agent equivalent to using curl? – mekineer Oct 08 '20 at 18:59
  • yes, I would think so... – Tom Oct 08 '20 at 19:34
0

What finally worked, was using seleniumwire and driver.header_overrides = {'Referer': 'com.google.android.gm'} Apparently, it was clicking the link from my gmail app, rather than using Kiwi. I reproduced the redirection on Chrome using the same referer.

mekineer
  • 11
  • 5