Tokenize a Stack-Based language

Question

I've been working on another stack-based golfing language called Stackgoat. In this challenge you'll be writing a Tokenizer for Stackgoat (or really any general stack-based languages).

Examples

"PPCG"23+
["PPCG", '23', '+']

'a "bc" +
['"a"', '"bc"', '+']

12 34+-"abc\"de'fg\\"
['12', '34', '+', '-', '"abc\"de'fg\\"']

"foo
['"foo"']

(empty input)
[]

' ""
['" "', '""']

Specification

The three types you'll need to handle are:

• Strings, anything within ""
• Numbers, any sequence of digits
• Operators, any other single character besides whitespace

Whitespace is essentially ignored unless it is within a string or separates two numbers.

String / char spec:

• Strings are delimited by a ", and when a \ is encountered, the next character should be escaped.
• Chars are prepended by a ' and the character after the ' should be converted into a string literal. 'a -> "a"
• ' will always have a character after it
• Closing quotes should be auto-inserted

Rules:

• No form of eval is allowed

Input / Output:

• Input can be taken through STDIN, function parameters, or your language's equivalent.
• Output should be an array or your language's closest equivalent.

Answer 1

Retina, 68 64 63 bytes

M!s`"(\\.|[^"])*"?|'.|\d+|\S
ms`^'(.)|^"(([^\\"]|\\.)*$)
"$1$2"

or

s`\s*((")(\\.|[^"])*(?<-2>")?|'.|\d+|.)\s*
$1$2¶
\ms`^'(.)
"$1"

I think this covers all the funky edge cases, even those not covered by the test cases in the challenge.

Try it online!

Answer 2

Ruby, 234 bytes

puts"[#{$stdin.read.scan(/("(?:(?<!\\)\\"|[^"])+(?:"|$))|'(.)|(\d+)|(.)/).map{|m|(m[0]?(m[0].end_with?('"')?m[0]: m[0]+'"'): m[1]?"\"#{m[1]}\"": m.compact[0]).strip}.reject(&:empty?).map{|i|"'#{/\d+|./=~i ?i: i.inspect}'"}.join', '}]"

I tried using the find(&:itself) trick that I saw... somewhere, but apparently .itself isn't actually a method. Also, I'm working on golfing the regex down, but it's already unreadable.

If we don't have to output in any fancy way (i.e. strings don't have to be quoted in the array) I can save a whole lotta bytes:

Still Ruby, 194 bytes:

p$stdin.read.scan(/("(?:(?<!\\)\\"|[^"])+(?:"|$))|'(.)|(\d+)|(.)/).map{|m|(m[0]?(m[0].end_with?('"')?m[0]: m[0]+'"').gsub(/\\(.)/,'\1'): m[1]?"\"#{m[1]}\"": m.compact[0]).strip}.reject(&:empty?)

I'm sure I can golf it more, but I'm not quite sure how.

---

Ungolfed coming soon. I started fiddling with the golfed directly at some point and I'll have to tease it out.

Answer 3

Python 3, 228 bytes

import re;L=list
print(L(map(lambda i:i+'"'if i[0]=='"'and not i[-1]=='"'else i,map(lambda i:'"%s"'%i[1]if i[0]=="'"else i,filter(None,sum([L(i)for i in re.findall('(\'.)|(".*")|(\d+)|([^\w\"\'\s\\\])|(".*"?)',input())],[]))))))

Here's a nice, long, two-liner.

---

Test it out in Python 3. Here's some examples:

$ python3 test.py
"PPCG"23+
['"PPCG"', '23', '+']

$ python3 test.py
'a "bc" +
['"a"', '"bc"', '+']

$ python3 test.py
12 34+-"abc"de'fg\"
['12', '34', '+', '-', '"abc"de\'fg\\"']

$ python3 test.py
"foo
['"foo"']

$ python3 test.py

[]

$ python3 test.py
' ""
['" "', '""']