In this specific case, if I replace \ with \\ and " with \" is it possible to trick my filter?
That's not sufficient, your filter is insecure.
E.g., one valid XSS attack vector would be </script><svg onload=alert(1)>, ending up with:
<script>
var test = {src: "test", layer: {"input": "</script><svg onload=alert(1)>", "event": "ready"}};
</script>
Since the XML (HTML) tree is parsed before any JS is evaluated, the closing script tag (</script>) will terminate the script despite being placed within a JS string.
Another problem with your filter are line breaks. If an attacker can insert a 0x0a byte, they can break your script by causing a syntax error (since a double quoted string can't span multiple lines):
<script>
var test = {src: "test", layer: {"input": "
", "event": "ready"}};
</script>
If you're using PHP, a convenient filter function to work safely with user input inside JS is json_encode(). From this answer:
With plain PHP a common and safe approach is to use
json_encode() as explained here. E.g.:
var foo = <?php echo json_encode($foo, JSON_HEX_QUOT|JSON_HEX_TAG|JSON_HEX_AMP|JSON_HEX_APOS); ?>
json_encode() returns the JSON representation of a value, hence it's
guaranteed to evaulate to a valid object in your JS code and you can
just assign it to a variable as shown. But don't omit the additional
flags. Depending on the context, an attacker could otherwise use
payloads like </script> to break out of the entire script tag.