Logging Firefox request URLs to stdout or a file?

Question

I need to get a log of all files retrieved from the Internet when I load a page in Firefox; I cannot really use the Firefox Web Developer (Tools) / Network Monitor because some pages which use Flash, tend to crash Firefox for me when I have the Network Monitor open (even if they seemingly load properly without Network monitor). So I need to obtain a list of URLs that will "survive" such crashes.

I am aware that I can get a log of network traffic from Firefox by using environment variables, for instance on Linux (this works on Firefox 43, but I couldn't get it to work with MOZ_LOG_MODULES/MOZ_LOG_FILE):

NSPR_LOG_MODULES=timestamp,nsHttp:3,sync NSPR_LOG_FILE=/tmp/ff.log /path/to/firefox/firefox

However, this prints very verbose information with full HTTP requests/responses, like:

2017-01-07 19:18:49.799922 UTC - -1371059392[b726e9c0]: Http2Session::ALPNCallback version=303
2017-01-07 19:18:49.981528 UTC - -1220458752[b726e0c0]: http request [
2017-01-07 19:18:49.981551 UTC - -1220458752[b726e0c0]:   POST / HTTP/1.1
2017-01-07 19:18:49.981560 UTC - -1220458752[b726e0c0]:   Host: ocsp.int-x3.letsencrypt.org
2017-01-07 19:18:49.981567 UTC - -1220458752[b726e0c0]:   User-Agent: Mozilla/5.0 (X11; Linux i686; rv:4
3.0) Gecko/20100101 Firefox/43.0
2017-01-07 19:18:49.981574 UTC - -1220458752[b726e0c0]:   Accept: text/html,application/xhtml+xml,applic
ation/xml;q=0.9,*/*;q=0.8
2017-01-07 19:18:49.981580 UTC - -1220458752[b726e0c0]:   Accept-Language: en-US,en;q=0.5
2017-01-07 19:18:49.981589 UTC - -1220458752[b726e0c0]:   Accept-Encoding: gzip, deflate
2017-01-07 19:18:49.981595 UTC - -1220458752[b726e0c0]:   DNT: 1
2017-01-07 19:18:49.981604 UTC - -1220458752[b726e0c0]:   Content-Length: 85
2017-01-07 19:18:49.981611 UTC - -1220458752[b726e0c0]:   Content-Type: application/ocsp-request
2017-01-07 19:18:49.981617 UTC - -1220458752[b726e0c0]:   Connection: keep-alive
2017-01-07 19:18:49.981639 UTC - -1220458752[b726e0c0]: ]
2017-01-07 19:18:50.071135 UTC - -1371059392[b726e9c0]: http response [
2017-01-07 19:18:50.071174 UTC - -1371059392[b726e9c0]:   HTTP/1.1 200 OK
2017-01-07 19:18:50.071195 UTC - -1371059392[b726e9c0]:   Server: nginx
2017-01-07 19:18:50.071203 UTC - -1371059392[b726e9c0]:   Content-Type: application/ocsp-response
2017-01-07 19:18:50.071210 UTC - -1371059392[b726e9c0]:   Content-Length: 527
2017-01-07 19:18:50.071218 UTC - -1371059392[b726e9c0]:   Etag: "3FFFC13100849446EA3D8FA68B0077AF4792776825416386C5CF76104592E0CE"
2017-01-07 19:18:50.071225 UTC - -1371059392[b726e9c0]:   Last-Modified: Wed, 04 Jan 2017 00:00:00 UTC
2017-01-07 19:18:50.071233 UTC - -1371059392[b726e9c0]:   Cache-Control: public, no-transform, must-revalidate, max-age=24762
2017-01-07 19:18:50.071240 UTC - -1371059392[b726e9c0]:   Expires: Sun, 08 Jan 2017 02:11:32 GMT
2017-01-07 19:18:50.071247 UTC - -1371059392[b726e9c0]:   Date: Sat, 07 Jan 2017 19:18:50 GMT
2017-01-07 19:18:50.071254 UTC - -1371059392[b726e9c0]:   Connection: keep-alive
2017-01-07 19:18:50.071259 UTC - -1371059392[b726e9c0]: ]

... and I really only need the full URLs, something like the lists shown by Firefox' Network Monitor:

So is there a way to persuade Firefox to dump only the accessed URLs in realtime to stdout or to a file, such that if Firefox crashes, I still have a log of accessed URLs?

Answer 1

I need to obtain a list of URLs that will "survive" such crashes.

The HTTP Request Logger Firefox add-on would appear to meet your exact needs:

Logs HTTP requests (referrer, HTTP method and URL) to a text file for security and auditing reasons.

Source HTTP Request Logger

Sometimes it might be useful to log each and every URL that is accessed through your Mozilla Firefox web browser. It might be for security, auditing or other reasons that you want to log accessed URLs. I developed the HTTP Request Logger add-on in order to track and discover a malicious web page that triggered the CVE-2010-4452 vulnerability in Oracle Java through a specially crafted applet tag and a respective Java applet class. You can download and install the add-on from here: HTTP Request Logger. You can also view the project on GitHub: prekageo/http-request-logger.

The add-on implements a JavaScript XPCOM component. This component is registered in chrome.manifest. The component registers an observer for http-on-modify-request topics. The observer’s callback function is called when a new HTTP request is ready to be sent. The function appends into a simple text file located on the user’s desktop a line consisting of the referrer, HTTP method and URL of the HTTP request.

Source HTTP Request Logger add-on for Mozilla Firefox

I don't know whether the output will survive a Firefox crash.

The output is written to http-request-log.txt on the desktop (full name C:\Users\username\Desktop\http-request-log.txt

Example output (from reloading https://superuser.com/questions):

(none) GET https://superuser.com/questions
(none) POST http://clients1.google.com/ocsp
https://superuser.com/questions GET https://ajax.googleapis.com/ajax/libs/jquery/1.12.4/jquery.min.js
https://superuser.com/questions GET https://cdn.sstatic.net/Js/stub.en.js?v=5d747a3197db
https://superuser.com/questions GET https://cdn.sstatic.net/Sites/superuser/all.css?v=39a3353d7dea
https://superuser.com/questions GET https://www.gravatar.com/avatar/1c46f449aee8300e9bcf2e6462e0127c?s=32&d=identicon&r=PG
https://chat.stackexchange.com/rooms/114/ask-a-super-user-moderator POST https://chat.stackexchange.com/events
https://superuser.com/questions GET https://i.stack.imgur.com/Ohyu2.png?s=48&g=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/adab46b83f2276ff0a2da05c3aea1a1e?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/49427fda382a6ebf890c6401eb1dc117?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/8c7bfdd599703d0f59b8adb35c33384d?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/ce3dd556aaf605869d870670f4addad2?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/b79191086ccc83d2efbf0346a9498552?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/2b4de46accb1b608fc45963de8c171d8?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/b61ea906fab08361132d0f98ffb89906?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://i.stack.imgur.com/pw4Ug.gif?s=32&g=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/93ec7cff90c748bc4f4da0c540e68bfc?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/5df3eeb6bd2ca404e3b21a08721ad45d?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/c6bac24248317129b390d60b7c088e40?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/1bcd453599a20c9eb444902d81c827c0?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/2e6a415fd9d93e089c5ac7138b306c8d?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://i.stack.imgur.com/VyvYN.png?s=32&g=1
https://superuser.com/questions GET https://lh5.googleusercontent.com/-snn4nMpy9Pw/AAAAAAAAAAI/AAAAAAAAATY/f_fJpek-wHo/photo.jpg?sz=32
https://superuser.com/questions GET https://i.stack.imgur.com/xObp1.png?s=32&g=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/994136ff5c5a426f77ded906bc1323b0?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://i.stack.imgur.com/98CKO.png?s=32&g=1
https://superuser.com/questions GET https://graph.facebook.com/867261659995682/picture?type=large
https://superuser.com/questions GET https://i.stack.imgur.com/oGmjT.png?s=32&g=1
https://superuser.com/questions GET https://i.stack.imgur.com/gaAXg.jpg?s=32&g=1
https://superuser.com/questions GET https://lh6.googleusercontent.com/-EwJ5_a_MCkc/AAAAAAAAAAI/AAAAAAAAAD8/Hra648Whrto/photo.jpg?sz=32
https://superuser.com/questions GET https://www.gravatar.com/avatar/bb3b38b5e3dac0b2df433d5c78cf9fc5?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/763da2b58630f974d468d88c26ac2e4e?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/c273f9992292a61959df77bca08665a3?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/41e856251e65f04dd94d72eed93e48e6?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/bc9f58c57918b259c487410155bd9e23?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/7db005472dc422ca5cbb143a5fd36125?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/b08fc2b0be91884d666ef12eac8d0d8f?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/6d1f8e5a125a589884b3784caa11736b?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://i.stack.imgur.com/ORos2.png?s=32&g=1
https://superuser.com/questions GET https://lh3.googleusercontent.com/-2DRzHlFuo4A/AAAAAAAAAAI/AAAAAAAAAJ4/UoePoxsDseg/photo.jpg?sz=32
https://superuser.com/questions GET https://i.stack.imgur.com/cYO05.jpg?s=32&g=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/bb7cd698db4bb8e9fa866d5154a05355?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/f9c738932874403fd70f2c617951f9e2?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://i.stack.imgur.com/Dqrx8.png?s=32&g=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/fc34739487a5e7e115acdbadf95c8fb3?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://lh4.googleusercontent.com/-Bah9vRlCwX8/AAAAAAAAAAI/AAAAAAAAAF4/sHjIDOHJT-c/photo.jpg?sz=32
https://superuser.com/questions GET https://www.gravatar.com/avatar/01cc31c9b124d090aeb3e06f94bb8ed8?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/417ffb01cabc10df5dc2ffad281e6a65?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/52eda3f9c34ef88ca164606301366366?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/a7da99412e7177afd0951d7511bdfec8?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/b3ac28f02a59435e921d244c0d82fbbf?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/e880c06c3667fe7c282f0dcb22e48525?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/f7a735cfc6d5fdf3ef44bdbb94fe3beb?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/7ebac3a51e6820a973e14c5b8b28e452?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://www.gravatar.com/avatar/1859855084a2f798e294422cf6ba2d31?s=32&d=identicon&r=PG&f=1
https://superuser.com/questions GET https://www.gravatar.com/avatar/ee06ba399c5eaf3f5a337a96207c2768?s=32&d=identicon&r=PG
https://superuser.com/questions GET https://graph.facebook.com/100000616410966/picture?type=large
https://superuser.com/questions GET https://engine.adzerk.net/ados.js
https://superuser.com/questions GET https://www.google-analytics.com/analytics.js
https://superuser.com/questions GET https://secure.quantserve.com/quant.js
https://superuser.com/questions GET https://sb.scorecardresearch.com/beacon.js
https://cdn.sstatic.net/Sites/superuser/all.css?v=39a3353d7dea GET https://cdn.sstatic.net/img/share-sprite-new.svg?v=78be252218f3
https://cdn.sstatic.net/Sites/superuser/all.css?v=39a3353d7dea GET https://cdn.sstatic.net/Sites/superuser/img/sprites.svg?v=6f5b3bbd35ac
https://cdn.sstatic.net/Sites/superuser/all.css?v=39a3353d7dea GET https://cdn.sstatic.net/img/favicons-sprite16.png?v=41af8148f8e0
(none) GET http://i.stack.imgur.com/05rE1.png
https://superuser.com/questions GET https://cdn.rawgit.com/ofirdagan/cross-domain-local-storage/d779a81a6383475a1bf88595a98b10a8bd5bb4ae/dist/scripts/xdLocalStorage.min.js
https://superuser.com/questions GET https://cdn.sstatic.net/Js/full.en.js?v=53cad7e6923d
https://superuser.com/questions GET https://scontent.xx.fbcdn.net/v/t1.0-1/p200x200/15621850_1188627624525749_7199393790071419683_n.jpg?oh=e4fa46a17dc0d6b646a664641891f18b&oe=58D7DB42
https://superuser.com/questions GET https://scontent.xx.fbcdn.net/v/t1.0-1/p200x200/14141922_1247416555288880_9079557968728006677_n.jpg?oh=05088f1d9fc7ee9e8c50a2795c018cb2&oe=59132CC4
https://superuser.com/questions GET https://maxcdn.bootstrapcdn.com/font-awesome/4.5.0/css/font-awesome.min.css
(none) GET https://i.stack.imgur.com/05rE1.png
https://superuser.com/questions GET https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20json%20where%20url%3D%22http%3A%2F%2Fstackexchange.com%2Fhot-questions-for-mobile%22&format=json
https://superuser.com/questions GET https://api.stackexchange.com/2.2/users/607889;194121;683117;682924;662836;618039;683114;683118;593210;3365;683110;529857;683096;683101;462324;683103;247768;661200;197638;683086;683095;658520;683078;654677;683081;683080;200839;487419;407559;683068;34923;551116;683059;387824;683053;683054;419223;297715;683050;292907;293642;683035;683044;683042;683045;124122;683041;682501;61846;581216?site=superuser&key=lL1S1jr2m*DRwOvXMPp26g((&access_token=eIDDpZG(uvLciqdnktsQmw))
https://superuser.com/questions GET https://api.stackexchange.com/2.2/questions?pagesize=5&order=desc&sort=activity&site=meta.superuser
https://superuser.com/questions GET https://rawgit.com/shu8/SE-Answers_scripts/master/dupeClosedMigratedCSS.css
https://superuser.com/questions GET https://superuser.com/questions/1164534
https://superuser.com/questions GET https://superuser.com/questions/1164524
https://superuser.com/questions GET https://superuser.com/questions/1164522
https://superuser.com/questions GET https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22http%3A%2F%2Fsuperuser.com%2Fquestions%2F1164516%22&diagnostics=true
https://superuser.com/questions GET https://query.yahooapis.com/v1/public/yql?q=select%20*%20from%20html%20where%20url%3D%22http%3A%2F%2Fsuperuser.com%2Fquestions%2F1164496%22&diagnostics=true
https://superuser.com/questions GET https://cdn-prom.sstatic.net/WinterBash/js/qa-wb.js?v=9
(none) POST http://ocsp.digicert.com/
(none) GET https://qa.sockets.stackexchange.com/
https://superuser.com/questions GET https://superuser.com/questions/1164534/what-happens-if-i-sleep-windows-10-while-programs-are-running
https://superuser.com/questions GET https://superuser.com/questions/1164524/select-the-driver-to-be-installed-windows-7-home-premium-x86
https://superuser.com/questions GET https://superuser.com/questions/1164522/how-can-i-build-a-sound-display-to-show-the-alsa-master-volume-for-the-fn-keys
(none) POST http://ocsp.digicert.com/
(none) POST http://ocsp.digicert.com/
https://superuser.com/questions GET https://i.stack.imgur.com/Kmqx8.png
https://superuser.com/questions GET https://cdn-prom.sstatic.net/WinterBash/css/wb-include.css?22
https://superuser.com/questions GET https://cdn-prom.sstatic.net/WinterBash/js/theactualhats.js?v=8
https://superuser.com/questions GET https://metasmoke.erwaysoftware.com/xdom_storage.html
https://superuser.com/questions GET https://cdn-prom.sstatic.net/WinterBash/js/hat-78404333.js?v=8
https://superuser.com/questions GET https://winterbash2016.stackexchange.com/api/current-hats?callback=winterBashCurrentHatsNaN&userids=337631%3B607889%3BNaN&host=superuser.com&_=1483822563839
https://metasmoke.erwaysoftware.com/xdom_storage.html GET https://cdn.rawgit.com/ofirdagan/cross-domain-local-storage/d779a81a6383475a1bf88595a98b10a8bd5bb4ae/dist/scripts/xdLocalStoragePostMessageApi.min.js
https://superuser.com/questions GET https://cdn-prom.sstatic.net/WinterBash/js/hat-78278539.js?v=8
https://superuser.com/questions/1164582/logging-firefox-request-urls-to-stdout-or-a-file GET https://winterbash2016.stackexchange.com/api/current-hats?callback=winterBashCurrentHatsNaN&userids=NaN&host=superuser.com&_=1483822104590
https://superuser.com/questions/1164582/logging-firefox-request-urls-to-stdout-or-a-file POST https://superuser.com/posts/1164582/editor-heartbeat/answer
https://0.docs.google.com/document/d/1AFlWPYR7xnrQRsrVTS2Q_-sLhK7Ew1hLDz5oNKlu_n0/xdbcm.html?internal=false&gcp=%7B%22qlc%22%3Anull%2C%22elh%22%3Afalse%2C%22eoo%22%3Atrue%7D&rt=j&xpc=%7B%22cn%22%3A%22docs%22%2C%22tp%22%3A1%2C%22osh%22%3Atrue%2C%22ppu%22%3A%22https%3A%2F%2Fdocs.google.com%2Frobots.txt%22%2C%22lpu%22%3A%22https%3A%2F%2F0.docs.google.com%2Frobots.txt%22%7D GET https://0.docs.google.com/document/d/1AFlWPYR7xnrQRsrVTS2Q_-sLhK7Ew1hLDz5oNKlu_n0/bind?id=1AFlWPYR7xnrQRsrVTS2Q_-sLhK7Ew1hLDz5oNKlu_n0&sid=1cc64febbb900314&VER=8&token=AC4w5VilbfesZ4ZFdH5AoL0ZwlR4uNlZ2g%3A1483822100782&lsq=1482517927748&u=18029933486027320144&c=1&w=1&gsi=0&smv=2&RID=rpc&SID=EA75E4E583E22ADB&CI=1&AID=48&TYPE=xmlhttp&zx=r45bxqz5hyo4&t=1
https://superuser.com/questions/1164582/logging-firefox-request-urls-to-stdout-or-a-file GET https://winterbash2016.stackexchange.com/api/current-hats?callback=winterBashCurrentHatsNaN&userids=NaN&host=superuser.com&_=1483822104591
https://chat.stackexchange.com/rooms/114/ask-a-super-user-moderator POST https://chat.stackexchange.com/events