0

I'm building an app that relies on a web scraper built in Node/Express, which operating on the same localhost as my front-facing Node/Express app.

I'm trying to optimise as much as possible and I'm completely new to all of this so please forgive if it's a basic question. Let's use this flow as an example:

  1. The front-facing app generates a list of about 10,000 domain names
  2. The front-facing app sends this data to my scraping app's api
  3. The scraper app does its magic then saves a whole lotta data to MongoDB

In terms of optimisation and best practice, is it better if my front-facing app sends one request to the scraper API per domain name? Or is it better to send an array of 10,000 strings to the scraper in one POST and parse through it within the app itself?

Jascination
  • 101
  • 3
  • 1
    This isn't a duplicate of the question linked to. The linked question talks about monitoring and tooling for load/capacity planning. This question is about *application optimization*. – Mendhak Aug 28 '15 at 09:38
  • @Mendhak: The direction is the same: We can't tell what's the best approach in the OPs specific case. He has to test what performs best in his case. – Sven Aug 28 '15 at 09:42
  • 1
    We can tell, actually - in this case it would be better to chunk the requests together due to the overhead of opening/closing a connection (do many at once). The linked thread talks of capacity planning, load testing (heavy traffic). They aren't at all the same. – Mendhak Aug 28 '15 at 09:45

0 Answers0