I'm building an app that relies on a web scraper built in Node/Express, which operating on the same localhost as my front-facing Node/Express app.
I'm trying to optimise as much as possible and I'm completely new to all of this so please forgive if it's a basic question. Let's use this flow as an example:
- The front-facing app generates a list of about 10,000 domain names
- The front-facing app sends this data to my scraping app's api
- The scraper app does its magic then saves a whole lotta data to MongoDB
In terms of optimisation and best practice, is it better if my front-facing app sends one request to the scraper API per domain name? Or is it better to send an array of 10,000 strings to the scraper in one POST
and parse through it within the app itself?