字句。
MIME と HTTP でやや定義が異なる。差は「{」「}」の有無。
RFC 2045 token = qr/[\x21\x23-\x27\x2A\x2B\x2D\x2E\x30-\x39\x41-\x5A\x5E-\x7E]+/;
RFC 2616 token = qr/[\x21\x23-\x27\x2A\x2B\x2D\x2E\x30-\x39\x41-\x5A\x5E-\x7A\x7C\x7E]+/;
CHAR = alpha | digit | separator | "!" | "#" | "$" |
"%" | "&" | "'" | "*" | "+" | "-" | "." | "`" |
"^" | "_" | "{" | "|" | "}" | "~" | CTL
CTL = <any control character>
separator = "(" | ")" | "<" | ">" | "@" | "," | ";" | ":" |
"\" | <"> | "/" | "[" | "]" | "?" | "=" | "{" |
"}" | SP | HT
token = 1*<any CHAR except CTLs or separators>[3] この定義を ASCII に当てはめると RFC 2045 の定義と一致します。 (なぜか RFC 2616 の定義ではありません。)