Sending email with CURL: some chinese characters appear

I'm sending some emails using SMTP and CURL over TOR but i have an issue: i receive some chinese characters over the ones i should receive, any idea?

This is how i send the mail using a batch script (i have a file variable):

curl.exe --socks5-hostname 127.0.0.1:9050 smtp://onionaddre.onion:port --mail-from mail@onionaddr.onion --mail-rcpt mail@onionaddr.onion --user mail@onionaddr.onion:password --upload-file %filetosend%

This is the original message:

0asf345£%$&fdasdhhs#@deadfga

This is what i get:

聣聭聤聭聯聶聥耠職聮聯職耠職聩聢聴聹耠聣聬聩耍耛聳聢耍聲

zipomojiv

Posted 2018-03-23T12:01:13.060

Reputation: 11

I don’t really understand what you’re doing here. But when you see behavior like that it is usually an encoding mismatch. How are you encoding the data and what does the recipient expect? Unicode/Base64/ASCII, etc. – Appleoddity – 2018-03-23T12:33:31.680

I have no clue. Basically i have a powershell program that generate characters according to time and date (like an encryption key) and then i need to send this key over tor. I don't know what encoding is used, how do i verify that? The TOR mail server i'm using is TorBox with SquirrelMail 1.4.22 – zipomojiv – 2018-03-23T12:40:14.433

You can force Powershell to encode the text to UTF8 by doing this: $UTF8 = [System.Text.Encoding]::UTF8 $Message = "blah" $UTF8Message = $UTF8.GetBytes($Message) – shinjijai – 2018-03-23T15:54:27.823

@shinjijai this should be added in the powershell code that generates the file right? – zipomojiv – 2018-03-23T15:59:40.467

If you're using Powershell to generate the file, you can save the file as UTF8 by using the -Encoding parameter and specifying UTF8. For example Out-File -Path "file.ext" -Encoding "UTF8". IF you are loading the file in with another Powershell script using Get-Content, remember to use -Encoding "UTF8" there as well.But since you're using --upload-file, the file being saved with UTF8 should suffice. I hope that make sense. – shinjijai – 2018-03-23T16:04:18.613

@shinjijai to generate the file save path i use: 'function Start-EncKey($path="$env:enc.key")' ( later i move this enc.key to mail.txt with cmd move command), so i should change it to: 'function Start-EncKey($path="$env:enc.key") -Encoding "UTF8"' right? – zipomojiv – 2018-03-23T16:10:56.490

@shinjijai even if later i already have: ' if ($success) { [System.IO.File]::AppendAllText($Path, $myvalue, [System.Text.Encoding]::Unicode) ' – zipomojiv – 2018-03-23T16:12:16.847

1Change the [System.Text.Encoding]::Unicode to [System.Text.Encoding]::UTF8. I think that's your issue, the file is unicode and cURL might be posting as UTF8, I don't know off-hand what the defaults are for cURL. – shinjijai – 2018-03-23T16:14:38.350

Perfect, it works! Add it to the answers so i can accept it – zipomojiv – 2018-03-23T17:27:00.400

Answers

Not really an answer yet, but maybe helpful and wouldn't fit in the comment section.

If you look at the Chinese characters closely, you will see that most of them have a similar thing (radical) on the left side. This usually means that they have a very similar codepoint.

$ perl -e 'use utf8; $foo = "聣聭聤聭聯聶聥耠職聮聯職耠職聩聢聴聹耠聣聬聩耍耛聳聢耍聲"; while ($foo =~ s/^(.)//) { printf("%04x\n", ord($1)) }'
8063
806d
8064
806d
806f
8076
8065
8020
8077
806e
806f
8077
8020
8077
8069
8062
8074
8079
8020
8063
806c
8069
800d
801b
8073
8062
800d
8072

If we get rid of the 80s, we get plain ASCII:

$ perl -e 'print "\x63\x6d\x64\x6d\x6f\x76\x65\x20\x77\x6e\x6f\x77\x20\x77\x69\x62\x74\x79\x20\x63\x6c\x69\x0d\x1b\x73\x62\x0d\x72\n"'
rmdmove wnow wibty cli

I can't make any sense of that text, but maybe you can.

sneep

Posted 2018-03-23T12:01:13.060

Reputation: 161

Probably i've mistaken the email and the sent file, i'll check again now and let you know. Is this process possible only with perl? is it possible with batch or powershell? – zipomojiv – 2018-03-23T14:51:11.930

Nah, I just used perl to demonstrate this. I don't think there's a real need for you to run these commands. (Dunno much about batch/powershell. Should be possible in powershell.) Thought it might help, but maybe it didn't. – sneep – 2018-03-23T15:00:48.200

@zipomojiv if you're using powershell, you can use send-mailmessage, is there a reason why you're doing it in curl? – shinjijai – 2018-03-23T15:05:08.417

Ok, i checked again with another file and yes, that's the problem. So for some reason the encoding changes when the file is sent. Is there any way to change the code automatically? for example i have 10 txt files in chinese and i want to "translate" them to readable characters, what shoul i do? I found this website https://r12a.github.io/apps/conversion/ for the chine to /xxxx tranformation, then i remove all the 80 in word and then i past the result here again https://r12a.github.io/apps/conversion/, having my text. Too slow for multiple file analisys

– zipomojiv – 2018-03-23T15:46:32.770

@shinjijai yes, to use TOR. Haven't found any mail CLI service that supports tor, only CURL does – zipomojiv – 2018-03-23T15:47:20.477

From the comments, the file you upload is created by another Powershell script that uses if ($success) { [System.IO.File]::AppendAllText($Path, $myvalue, [System.Text.Encoding]::Unicode) to save the content to disk.

Saving the file with character encoding UTF8 should fix the problem, so replace [System.Text.Encoding]::Unicode to [System.Text.Encoding]::UTF8.

shinjijai

Posted 2018-03-23T12:01:13.060

Reputation: 1 391