1
I am using bash to search for a term in a website and I get the following message (instead of the website):
Please verify you are a human
Access to this page has been denied because we believe you are using automation tools to browse the website.
This may happen as a result of the following:
Javascript is disabled or blocked by an extension (ad blockers for example)
Your browser does not support cookies
Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading.
Reference ID: #1cdaf6b0-b84f-11e9-bb3c-6337d1b9880d
For example:
curl -s https://www.couchsurfing.com/users/1000525404 | grep Gracias
What can I do to avoid this? I want to use command line as I can automate some stuff?
Or is there any other opensource project that lets us browse such website from the command line?
The message states that Please make sure that Javascript and cookies are enabled on your browser and that you are not blocking them from loading.
Is it possible to do so from the commandline?
Nikhil
Posted 2019-08-06T13:42:48.567
Reputation: 217
@Hashim I get the follwoing:
$ wget https://www.couchsurfing.com/users/1000525404 > a.html --2019-08-06 15:49:51-- https://www.couchsurfing.com/users/1000525404 Resolving www.couchsurfing.com (www.couchsurfing.com)... 100.24.158.220, 35.169.227.38, 3.209.172.214 Connecting to www.couchsurfing.com (www.couchsurfing.com)|100.24.158.220|:443... connected. HTTP request sent, awaiting response... 403 Forbidden 2019-08-06 15:49:52 ERROR 403: Forbidden.– Nikhil – 2019-08-06T13:50:17.900Does
wget -U firefox https://www.couchsurfing.com/users/1000525404work? – Arkadiusz Drabczyk – 2019-08-06T13:55:00.227@ArkadiuszDrabczyk
Please verify you are a humandoesn't appear anymore, and the website seems to be able to downloaded. Is it possible if I could give me login credentials to this URL? Also is it safe to give login credentials(given that no one else is looking at my screen)? I have logged into the website, from firefox browser, but this login credentials doesn't seem to work with wget. – Nikhil – 2019-08-06T14:01:03.910That would be harder to achieve with wget. You could save cookies and then re-use them with wget. Or maybe they provide some API that you can use? – Arkadiusz Drabczyk – 2019-08-06T14:02:38.283
1
There is one but it's unmaintained: https://github.com/nderkach/couchsurfing-python
– Arkadiusz Drabczyk – 2019-08-06T14:05:13.983