SRFI nnn: Line directives

by Lassi Kortela

Status

Draft

Abstract

Many language-agnostic programming tools rely on specially formatted source code comments to annotate the code with metadata. Such "magic comments" are hard for both humans and computers to parse reliably, as the purpose of a comment is to be free-form text that is not interpreted by machine.

This SRFI extends the standard Scheme directive syntax (#!) to support line directives. They look like magic comments to language-agnostic tools but read as S-expressions in Scheme, combining the portability of magic comments with the well-defined syntax and easy parsing of ordinary Scheme code.

Issues

Rationale

Like other Lisp dialects, Scheme uses S-expressions for its surface syntax. S-expressions are naturally extensible: just use a particular symbol as the head of a list to indicate what's coming up in the rest of the list. Ideally we would use S-expressions not only for data, but also to write metadata such as license information, character encoding, and editor settings.

However, most programming languages have a much less regular and more ad-hoc syntax and do not have easily extensible syntactic constructs. To the extent that extensible constructs exist, the culture to use them is lacking. As a result, most languages eventually paint themselves into a corner when the time comes to augment source code with metadata. As a last resort, the metadata is snuck into comments which are ignored by the compiler. This is unfortunate as comments are meant for natural language text and deliberately have no syntactic structure to be parsed, checked, and enforced by the compiler. That makes tools dealing with magic comments very brittle, and their precise behavior difficult to understand.

For interoperability with language-agnostic tools such as text editors and license checkers, we need to play along with the mess caused by the lack of foresight in other languages. However, we can salvage a bit of rigor by parsing the directives that other languages put in a comment as a sequence of S-expressions in Scheme. Happily, the character set permitted in Scheme and Lisp symbols is so wide that most magic comments (which tend to contain alphanumeric characters and some punctuation delimited by whitespace) will naturally split into a list of symbols.

If we can turn each directive into a list of symbols, it becomes easy to programmatically find all the directives in a Scheme source file. A Scheme implementation can process the directives it understands, ignore the rest, and optionally let users install custom hooks to process directives.

Symbols and keywords

Some Scheme implementations have read syntax for keywords. The following syntaxes are known to exist:

It also varies whether or not keywords are disjoint from symbols.

The presence of keyword syntax may cause some tokens that are read as symbols in most Scheme implementations to be read as keywords in others. Language-agnostic directives tend to use colons a lot, which may cause them to be parsed differently in Scheme implementations with keywords.

Why use #! as a directive marker

In Scheme, a semicolon (;) starts a comment that runs until the end of the line. In RnRS parlance, this is called a line comment. An identifier following #! is called a directive. Combining these two terms, we arrive at line directive: a directive that runs until the end of the line.

#! is the best choice for a directive marker because most Scheme script readers already need special handling for #! appearing at the very beginning of a source file since #! is the Unix indicator that says which interpreter runs the script. (When the first line of a Unix script starts with #! it is often called the "shebang line", or the "hashbang line".)

#! is also the only syntax used for reader directives in standard Scheme, and a declaration of something like character encoding is conceptually close to such directives.

Survey of #! syntax in Lisp standards

Common Lisp

The ANSI Common Lisp standard, section 2.4.8 Syntax -> Standard Macro Characters -> Sharpsign, says the dispatch macro character #! is explicitly reserved to the user. No conforming implementation defines it.

ISLisp

The ISLisp standard (2007) uses # as a dispatch macro character like Common Lisp and Scheme. It does not appear to use #! for anything.

Scheme

Scheme standards treat #! as a magic marker generally followed by a directive that looks like an identifier but serves a special purpose distinct from an ordinary identifier with the same spelling.

DSSSL and SRFI 89 use #!key and #!optional and #!rest for lambda list markers.

Various Scheme implementations use other identifiers following #! for other purposes.

SRFI 22 (Running Scheme Scripts on Unix) recognizes #! at the very beginning of the file as starting a comment that runs until the end of the line.

Guile uses #!...!# anywhere in the file for multi-line comments.

Survey of #! syntax in non-standard Lisp dialects

AutoLisp

# is a constituent character in symbols. #! does not appear to be used for anything.

Clojure and ClojureScript

#! unofficially begins a comment that runs until the end of the line.

Source: In the Clojure codebase, open src/jvm/clojure/lang/LispReader.java and look for dispatchMacros['!'] = new CommentReader();.

Emacs Lisp

#! unofficially begins a comment that runs until the end of the line.

Source: In the GNU Emacs codebase, open src/lread.c and look inside read1(). The comment in the source code says #! appears at the beginning of an executable file. Skip the first line. but the code actually skips #! anywhere in the file.

Fennel

#! at the very beginning of the file starts a comment that runs until the end of the line.

Source: In the Fennel codebase, open src/fennel/parser.fnl and look inside string-stream. It replaces a #! at the start of the file with a traditional ;; comment prefix.

Hy

# begins a comment that runs until the end of the line. By extension, #! does the same.

Janet

# officially begins a comment that runs until the end of the line. By extension, #! does the same.

Lisp Flavored Erlang

# is a constituent character in symbols. Otherwise # does not appear to be used for anything, and causes an illegal character syntax error.

NewLisp

# officially begins a comment that runs until the end of the line. By extension, #! does the same.

PicoLisp

# officially begins a comment that runs until the end of the line. By extension, #! does the same.

As a minor exception, # does not begin a comment when it is part of a symbol.

Rep (librep)

#!...!# at the very beginning of the file is treated like a comment.

Elsewhere in the file #!key and #!optional and #!rest are read in as symbols, and #! followed by anything else is a syntax error.

Source: In the librep codebase, open src/lisp.c and look inside readl().

Conclusion

Both major Lisp standards, Common Lisp and Scheme, are easily amenable to adding a #! syntax. While Common Lisp reserves #! for users, customizing the readtable is quite rare in practice, and the pervasive use of #! in Unix scripts makes it unlikely that Common Lisp users would configure #! to do anything other than read a comment.

Of the surveyed non-standard Lisp dialects, all except AutoLisp, Fennel, Lisp Flavored Erlang, and Rep handle #! anywhere in the file akin to a comment. Both Fennel and Rep handle #! at the beginning of the file as a comment, which means their syntax could easily be amended to handle #! elsewhere in the file the same way. AutoLisp and LFE permit # in symbols. LFE does not permit # to start a symbol; perhaps AutoLisp doesn't either. If so, adding special meaning to #! would not present a conflict in these dialects either.

Guile and Rep use #!...!# comments, i.e. they require a !# at the end. We can expect !# to cause a read error in most Scheme implementations. However, using ;!# causes it to be treated as an ordinary comment that runs until the end of the line in those Schemes.

Specification

Scheme's read syntax is extended such that the two-character sequence #! followed immediately by either a newline, the end of the file, or one or more horizontal whitespace charcters, causes a line directive to be read as follows:

  1. Start accumulating elements into an empty list.
  2. Skip whitespace and comments.
  3. If at end of line or no longer on the original line, return.
  4. Read one form and accumulate at the end of the list.
  5. If at end of line or no longer on the original line, return.
  6. Go to step 2.

The accumulated list is returned as the Scheme representation of the line directive. The list is not evaluated as Scheme code, but may be either ignored or passed to an implementation-defined facility for processing.

Recursive line directives

It is an error to write things like #! #! foo or #! outer (#! inner).

Equivalence of #! r6rs with a space and #!r6rs without

It is implementation-defined whether #! r6rs (with whitespace), once read, is processed the same way or in a different way compared to #!r6rs (without whitespace).

Extending the formal syntax of R7RS

R7RS section 7.1. Formal syntax add a new case to the definition of <directive>:

#! <intraline whitespace>+ <directive datums>

where

<directive datums> = | <datum> <intraline whitespace>* <directive datums>

and <datum> in this case contains no newlines in the datum or surrounding whitespace.

Examples

For clarity, all Scheme symbols in the examples are written using the |vertical-bar| syntax from R7RS.

Unix script interpreter

Line directive:

#! /usr/bin/env fantastic-scheme

Read as S-expression:

(|/usr/bin/env| |fantastic-scheme|)

License information

Line directives:

#! Copyright © 2019, 2020 Just A. Schemer <schemer@example.org>
#! SPDX-License-Identifier: GPL-3.0-or-later

Read as S-expressions:

(|Copyright| |©| 2019 (|unquote| 2020)
 |Just| |A.| |Schemer| |<schemer@example.org>|)
(|SPDX-License-Identifier:| |GPL-3.0-or-later|)

Text editor settings for Emacs

Line directive:

#! -*- mode: scheme -*-

Read as S-expression:

(|-*-| |mode:| |scheme| |-*-|)

Line directives:

#! Local Variables:
#! mode: scheme
#! coding: utf-8
#! comment-column: 0
#! End:

Read as S-expressions:

(|Local| |Variables:|)
(|mode:| |scheme|)
(|coding:| |utf-8|)
(|comment-column:| 0)
(|End:|)

Text editor settings for Vim

Line directive:

#! vim: ft=lisp tw=60 ts=2 expandtab fileencoding=euc-jp :

Read as S-expression:

(|vim:| |ft=lisp| |tw=60| |ts=2| |expandtab| |fileencoding=euc-jp| |:|)

Text editor settings for both Emacs and Vim

Line directive:

#! -*- mode: scheme -*- vim: set ft=scheme :

Read as S-expression:

(|-*-| |mode:| |scheme| |-*-| |vim:| |set| |ft=scheme| |:|)

Implementation

A patch to the Gauche reader is attached. The patch makes it write all line directives read from source files to the current error port.

Acknowledgements

References

Copyright

Copyright (C) Lassi Kortela (2021).

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.