Sending gist to github via cURL and issues with new lines inside file

Question

So, let's say I have an index.php file containing this:

 <?= "Hello" ?>

 <?= echo "WORLD" ?>

And I wanted upload the contents of this file to my gists in github, which I am doing via

gist_content=$(cat 'index.php')

curl --user "GITHUB_USER"  -H "Content-Type: application/json; charset=UTF-8" -X POST -d  "{ \"description\": \"Created via API\", \"public\": \"true\", \"files\":{ \"index.php \":{ \"content\": \"$gist_content\"}}\" " https://api.github.com/gists

Now, this script does not work for some reason, and I get error response

{
  "message": "Problems parsing JSON",
  "documentation_url": "https://developer.github.com/v3/gists/#create-a-gist"
}

If I write everything in one line without tags, quotes like hello it works find

Answer 1

You've some syntax error in your JSON string. Please check and correct it. E.g.

$ echo "{ \"description\": \"Created via API\", \"public\": \"true\", \"files\":{ \"index.php \":{ \"content\": \"$gist_content\"}}\" " | python -m json.tool
Expecting ',' delimiter: line 1 column 95 (char 94)

So you're missing one of the curly brackets, you're opening 3, but closing 2.

The simplified syntax should be like:

$ echo '{"description": "Created via API", "public": "true", "files": { "index.php": { "content": "foo" } } }' | python -m json.tool
{
    "description": "Created via API",
    "files": {
        "index.php": {
            "content": "foo"
        }
    },
    "public": "true"
}

Then it's matter of escaping the quotes, but you're escaping it in the wrong way, see: How to escape single-quotes within single-quoted strings? For example:

$ echo 'abc'\''abc'
abc'abc
$ echo "abc"\""abc"
abc"abc

Since you're importing external file which consist double-quotes as well, you should double quote them as well using tools such as sed, etc. The same with new lines, you should change them into appropriate control characters (either <br> or \n) depending on the expected format.

So your final example would look like:

gist_content=$(cat index.php | sed 's/"/\\"/g' | paste -s -d '\\n' -)
curl --user "GITHUB_USER" -H "Content-Type: application/json; charset=UTF-8" -X POST -d "{"\""description"\"": "\""Created via API"\"", "\""public"\"": "\""true"\"", "\""files"\"": { "\""index.php"\"": { "\""content"\"": "\""$gist_content"\"" } } }" https://api.github.com/gists

Answer 2

You can use this solution to replace new lines, Also you have to escape double quotes in the content & description field :

#!/bin/bash

ACCESS_TOKEN="YOUR_ACCESSS_TOKEN"

description="the description for this gist. There are also some quotes 'here' and \"here\" in that description"
public="true"
filename="index.php"

desc=$(echo "$description" | sed 's/"/\\"/g' | sed ':a;N;$!ba;s/\n/\\n/g')
json=$(cat index.php | sed 's/"/\\"/g' | sed ':a;N;$!ba;s/\n/\\n/g')

curl -v -H "Content-Type: text/json; charset=utf-8" \
        -H "Authorization: Token $ACCESS_TOKEN" \
        -X POST https://api.github.com/gists -d @- << EOF
{ 
  "description": "$desc", 
  "public": "$public", 
  "files": { 
      "$filename" : { 
          "content": "$json"
       } 
   } 
}
EOF