1

I am making an app that will receive images and then needs to store them in a SQL DB using Nodejs. I need to store a lot of images and query them a lot as well (I need to query the db to see if that image is in there)

From what I have read its better to save the files in the file system and put the path in the DB (cheaper).

I learned to develop using Heroku and Mlab and saving URLs to my DB this will be my first time using the FileSystem.

I read about azure storage options and a rep told me that I can use their blob services and it would be a good fit. But it seems like there is a lot of learning involved to use their services. I tried azure and aws before for another project and everything seems more complicated with them compared to just using git to deploy to heroku, just picking which services and stuff to use and how to use them.

But it seems like the pricing is A LOT cheaper using azure blobs than using e.g. Heroku, although even the pricing is difficult for me to understand with all these services (Heroku DYNOS...)

The thing is as far as I can tell with Heroku I can just make my app, upload the files to the server using the fs module and then save that path to my DB. Whereas to use azure blobs I also have to learn how to use azures API and store the image in the blob. I have read their documentation: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-nodejs-v10 and cant find exactly how I would know the location of the file on the blob so that I could save it to my DB and then how to retrieve that file.

So my questions are as follows:

  1. Is my best option (cheapest most efficient) to use azure and azure blobs or are there better options for what I need?
  2. Is it as I see it that their is another layer of complexity in using azure storage and blobs and is this the norm when using azure & AWS (are these cloud services whereas the others arent? Heroku also mentions cloud in their web page but these seem different)????
  3. Is it worth the hassle to learn about blobs or is it better to just use something more simple to start off?
  4. Is saving the images to cloudinary and saving the urls to my SQL DB viable or is that too expensive or inefficient???

Thanks for the help in advance

Alexander Tolkachev
  • 4,513
  • 3
  • 14
  • 23
Jack
  • 113
  • 3

3 Answers3

1

The thing is as far as I can tell with Heroku I can just make my app, upload the files to the server using the fs module and then save that path to my DB.

You can, but those files will go away a) on the next deploy and b) once daily when the dynos restart. The filesystem on Heroku is ephemeral - changes made to it are not permanent.

If you need user uploads (or any other sort of permanent file that doesn't exist in your Git repository), you need some sort of off-Heroku storage like Azure or AWS's S3 to store them.

ceejayoz
  • 32,469
  • 7
  • 81
  • 105
  • Thanks a lot for this info man, I did not know about that guess Heroku is definitely a no go. – Jack Jan 29 '19 at 13:08
  • @Jack Don't dismiss it out of hand. Doing it this way helps with scaling - you can replace servers without losing data, run multiple servers to handle heavy load, etc. The extra complexity can be well worthwhile. – ceejayoz Jan 29 '19 at 13:34
  • what do you mean didn't you say that I need some off-heroku storage? What do you mean by this way??? I was planning now on making the nodejs app on heroku and then connect to azure blobs is that what you are saying? – Jack Jan 29 '19 at 17:28
  • Your statement "guess Heroku is definitely a no go" implied you were getting rid of Heroku entirely, hence my response. – ceejayoz Jan 29 '19 at 17:32
  • Oh is what I am doing what you were suggesting??? I would like to understand more about this stuff, when you said "Doing it this way helps with scaling - you can replace servers without losing data, run multiple servers to handle heavy load,etc" do you mean by this method?? if so compared to what? how come? lol sorry all the questions just trying to learn :D – Jack Jan 29 '19 at 17:35
0
  1. Azure storage blobs are the cheapest image storing option in Azure
  2. You really dont want to save anything local disk or store images in the db (who the hell does that?)
  3. blob storage is dead simple
  4. no idea what is cloudinary, but you still need to be able to know image url if you store it in the blob, so you probably have to save it somewhere
4c74356b41
  • 628
  • 5
  • 10
  • 2- I was going from this post https://stackoverflow.com/questions/3748/storing-images-in-db-yea-or-nay – Jack Jan 29 '19 at 13:09
  • 4- arent you saving the image in the blob and then you get the blobs url which is the image url? I am now working on this and I upload the image and can see it on the azure dashboard but I cant seem to download it. – Jack Jan 29 '19 at 13:10
  • downloading is straightforward if they are public, just invoke URL. if they are not public you can use SDK to download (or can handle Auth on your own). – 4c74356b41 Jan 29 '19 at 13:22
  • oh didn't really see anything about them being public or private in the quickstart code for NodeJS, do you mean the containers/blobs or both? How do I do this programatically. If I check in the blobs shared access setting they al have read and list permission checked dont see anything on public vs private. – Jack Jan 29 '19 at 13:38
  • I dont have a nodeJs code, but I have this: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-manage-access-to-resources, https://docs.microsoft.com/en-us/azure/storage/common/storage-security-guide#data-plane-security. these should help you get started. you can set container access when you create it, or after you've created it you can modify it. this creates public container in nodejs: https://docs.microsoft.com/en-us/azure/storage/blobs/storage-quickstart-blobs-nodejs#create-a-container – 4c74356b41 Jan 29 '19 at 13:41
0

I have not tried Heroku, but I can comment on Azure Blobstorage.

The best practice I see is:

Storing images to blobstorages, with anonymous read-only access to container and saving URLs to database.

As you have only URLs, you app will download images from Azure saving traffic from your server. Generating random string names instead of (1,2,3,... or a,b,c,... or john,doe,jane,...) for each image will restrict user from accessing unauthorized images.

Creating blob storage on Azure is very simple.

  1. Create a resource (+)
  2. Select Storage Account
  3. Create

You will get Storage Account Name and Key from Access Keys tab.

In Storage Account you'll see: Blobs, Files, Tables, Queues.

Go to Blobs:

  1. Create a new container
  2. Be careful about Public Access Level for container (I would recommend container)
  3. Click on container properties (More options icon ...) and get Container URL.

As mentioned in the docs you read, you will need to define

  • Storage Account Name
  • Key
  • Container URL

in your code and use them.

Community
  • 1
r_D
  • 151
  • 4
  • Hi r_D thanks for the info, what I am trying to do is store images for stores, I planed on having a table for the stores name and then another table for the staff and one for the customers. Where the staff and customer table have a reference to the store ID. And then I would organize the blobs so that each store has its own container named by the store ID, and then the blobs would be named according to staff or customer staf001 customer001 and so. So that from the DB one could take the container = store name and the staff or customer ID and query azure on their own, or is storing URLs better? – Jack Jan 29 '19 at 13:05
  • Also for the structure I suggested above what do you think the permissions should be for containers and blobs??? I will be creating the containers and blobs programatically with code so not sure which permissions they get by default or how to change them though. – Jack Jan 29 '19 at 13:07
  • I would suggest using single container, but directories/folders inside the container for each store, and folder name can be store name. This is will avoid using multiple credentials(keys) in the code. Container is suitable for your requirements. – r_D Jan 29 '19 at 13:26
  • how do I create directories/folders inside the container, got any links to info on that??? Why multiple credential keys though dont you just use your storage account name and key for all the containers??? – Jack Jan 29 '19 at 13:34
  • Sorry for that, I confused containers with storage account. You can have multiple containers for multiple stores. For folder inside container, in the upload (in your blobUploadCode) path try with `containerURL/store1/customer1/profile/image1.jpg`, this will create `store1/customer1/profile/` folder structure in the container. – r_D Jan 29 '19 at 13:45
  • I see but now its just the same credentials no so is their an advantage to that ??? – Jack Jan 29 '19 at 17:26
  • There is no advantage. You can use any either way. – r_D Jan 29 '19 at 17:29
  • Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/88993/discussion-between-r-d-and-jack). – r_D Jan 30 '19 at 05:04