[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]
Re: Dynamic token kinds
From: |
Akim Demaille |
Subject: |
Re: Dynamic token kinds |
Date: |
Wed, 19 Dec 2018 07:34:34 +0100 |
Hi Frank, Hi all,
> Le 16 déc. 2018 à 10:02, Frank Heckenbach <address@hidden> a écrit :
>
> So to make it safe, we might need something like this:
>
> static inline
> symbol_type
> make_symbol (token_type type, b4_locations_if([const location_type& l, ])T&&
> v);
>
> auto-generated for each semantic type T of any token (plus one
> without the "v" parameter for untyped tokens) that checks (at
> runtime) the "type" parameter against the (statically known) valid
> token types for T.
I like this idea. I have a draft for it in my repo, as "make-symbol".
Please, try it and report about it.
There are a few issues:
- make_symbol will collide if the user has a token named symbol
Any idea of a better name?
- In the signature of make_symbol, I've had to use int for the
token type, instead of the enum token_type, and then
to convert the int into token_type. I don't like that, but I
probably don't have much of a choice. (A template would be
overkill IMHO).
- I'm not sure I should emit the assert only when parse.assert
was set, maybe we should always do it. After all, if the user
does not want the check, he can pass -DNDEBUG. parse.assert
was made to check what Bison generates, more than to check what
the user does.
It's used like this:
yy::parser::symbol_type yylex ()
{
static char const input[] = "12";
static size_t toknum = 0;
int res = input[toknum++];
if (res == '1')
return yy::parser::make_symbol (res, std::make_unique<int> (10));
else if (res == '2')
return yy::parser::make_symbol (res, std::make_pair<int, int> (21, 22));
else
return yy::parser::make_symbol (res);
}
The generated code looks like this (I had a token named
m4_define to make sure everything was properly quoted):
Input:
%token <pair<int, int>> FOO '+' BAR "BAR" m4_define
%nterm <pair<int, int>> exp exp1
Output:
inline
parser::symbol_type
parser::make_symbol (int tok)
{
assert (tok == 0);
return symbol_type (token_type (tok));
}
inline
parser::symbol_type
parser::make_symbol (int tok, pair<int, int> v)
{
assert (tok == token::FOO || tok == 43 || tok == token::BAR || tok ==
token::m4_define);
return symbol_type (token_type (tok), std::move (v));
}
commit 1f86c28ea2ff69f68a569568ad6890dee3ac9684
Author: Akim Demaille <address@hidden>
Date: Tue Dec 18 07:16:55 2018 +0100
c++: provide symbol constructors per type
On
%token <int> FOO BAR
we currently generate make_FOO(int) and make_bar(int). However, in
order to factor their scanners, some users would also like to have
make_symbol(tok, int), where tok is FOO or BAR. To ensure type
safety, add assertions that do check that value type and token type
match. Bind this assertion to the parse.assert %define variable.
Suggested by Frank Heckenbach.
http://lists.gnu.org/archive/html/bug-bison/2018-12/msg00034.html
Should also match expectations from Аскар Сафин.
http://lists.gnu.org/archive/html/bug-bison/2018-12/msg00023.html
* data/variant.hh: Use b4_token_visible_if where applicable.
(_b4_type_constructor_declare, _b4_type_constructor_define): New.
Use them.
diff --git a/data/variant.hh b/data/variant.hh
index 9696cf7b..75811b0d 100644
--- a/data/variant.hh
+++ b/data/variant.hh
@@ -335,6 +335,16 @@ m4_define([b4_symbol_value_template],
## ------------- ##
+# _b4_includes_tokens(SYMBOL-NUM...)
+# ----------------------------------
+# Expands to non-empty iff one of the SYMBOL-NUM denotes
+# a token.
+m4_define([_b4_is_token],
+ [b4_symbol_if([$1], [is_token], [1])])
+m4_define([_b4_includes_tokens],
+ [m4_map([_b4_is_token], address@hidden)])
+
+
# _b4_symbol_constructor_declare(SYMBOL-NUM)
# ------------------------------------------
# Declare make_SYMBOL for SYMBOL-NUM. Use at class-level.
@@ -358,12 +368,38 @@ b4_join(b4_symbol_if([$1], [has_type],
])])
+# _b4_type_constructor_declare(SYMBOL-NUM...)
+# -------------------------------------------
+# Declare a unique make_symbol for all the SYMBOL-NUM (they
+# have the same type). Use at class-level.
+m4_define([_b4_type_constructor_declare],
+[m4_ifval(_b4_includes_tokens($@),
+[#if 201103L <= YY_CPLUSPLUS
+ static
+ symbol_type
+ make_symbol (dnl
+b4_join([int tok],
+ b4_symbol_if([$1], [has_type],
+ [b4_symbol([$1], [type]) v]),
+ b4_locations_if([location_type l])));
+#else
+ static
+ symbol_type
+ make_symbol (dnl
+b4_join([int tok],
+ b4_symbol_if([$1], [has_type],
+ [const b4_symbol([$1], [type])& v]),
+ b4_locations_if([const location_type& l])));
+#endif
+])])
+
+
# b4_symbol_constructor_declare
# -----------------------------
-# Declare symbol constructors for all the value types.
-# Use at class-level.
+# Declare symbol constructors. Use at class-level.
m4_define([b4_symbol_constructor_declare],
[ // Symbol constructors declarations.
+b4_type_foreach([_b4_type_constructor_declare])
b4_symbol_foreach([_b4_symbol_constructor_declare])])
@@ -401,6 +437,50 @@ b4_join(b4_symbol_if([$1], [has_type],
])])
+# _b4_type_constructor_define(SYMBOL-NUM...)
+# ------------------------------------------
+# Declare a unique make_symbol for all the SYMBOL-NUM (they
+# have the same type). Use at class-level.
+m4_define([_b4_type_clause],
+[b4_symbol_if([$1], [is_token],
+ [b4_symbol_if([$1], [has_id],
+ [tok == token::b4_symbol([$1], [id])],
+ [tok == b4_symbol([$1], [user_number])])])])
+
+m4_define([_b4_type_constructor_define],
+[m4_ifval(_b4_includes_tokens($@),
+[#if 201103L <= YY_CPLUSPLUS
+ inline
+ b4_parser_class_name::symbol_type
+ b4_parser_class_name::make_symbol (dnl
+b4_join([int tok],
+ b4_symbol_if([$1], [has_type],
+ [b4_symbol([$1], [type]) v]),
+ b4_locations_if([location_type l])))
+ {b4_parse_assert_if([
+ assert (m4_join([ || ], m4_map_sep([_b4_type_clause], [, ],
address@hidden)));])[
+ return symbol_type (]b4_join([token_type (tok)],
+ b4_symbol_if([$1], [has_type], [std::move
(v)]),
+ b4_locations_if([std::move (l)])));
+ }
+#else
+ inline
+ b4_parser_class_name::symbol_type
+ b4_parser_class_name::make_symbol (dnl
+b4_join([int tok],
+ b4_symbol_if([$1], [has_type],
+ [const b4_symbol([$1], [type])& v]),
+ b4_locations_if([const location_type& l])))
+ {b4_parse_assert_if([
+ assert (m4_join([ || ], m4_map_sep([_b4_type_clause], [, ],
address@hidden)));])[
+ return symbol_type (]b4_join([token_type (tok)],
+ b4_symbol_if([$1], [has_type], [v]),
+ b4_locations_if([l])));
+ }
+#endif
+])])
+
+
# b4_basic_symbol_constructor_declare(SYMBOL-NUM)
# -----------------------------------------------
# Generate a constructor declaration for basic_symbol from given type.
@@ -451,4 +531,5 @@ m4_define([b4_basic_symbol_constructor_define],
# Define the overloaded versions of make_symbol for all the value types.
m4_define([b4_symbol_constructor_define],
[ // Implementation of make_symbol for each symbol type.
+b4_type_foreach([_b4_type_constructor_define])
b4_symbol_foreach([_b4_symbol_constructor_define])])
diff --git a/tests/types.at b/tests/types.at
index 2289f751..adc8ec6a 100644
--- a/tests/types.at
+++ b/tests/types.at
@@ -327,6 +327,26 @@ m4_foreach([b4_skel], [[yacc.c], [glr.c], [lalr1.cc],
[glr.cc]],
[10, 21, 22],
[AT_REQUIRE_CXX_STD(14, [echo "$at_std not supported"; continue])])
+ # Type-based token constructors on move-only types, and types with commas.
+ AT_TEST([%skeleton "]b4_skel["
+ %code requires { #include <memory> }
+ %define api.value.type variant
+ %define api.token.constructor],
+ [[%token <std::unique_ptr<int>> '1';
+ %token <std::pair<int, int>> '2';]],
+ ['1' '2' { std::cout << *$1 << ", "
+ << $2.first << ", "
+ << $2.second << '\n'; }],
+ ["12"],
+ [[if (res == '1')
+ return yy::parser::make_symbol ('1', std::make_unique<int>
(10));
+ else if (res == '2')
+ return yy::parser::make_symbol ('2', std::make_pair<int, int>
(21, 22));
+ else
+ return yy::parser::make_symbol (0)]],
+ [10, 21, 22],
+ [AT_REQUIRE_CXX_STD(14, [echo "$at_std not supported"; continue])])
+
])
])
- Re: Porting to typed C++ parser (was: Dynamic token kinds), (continued)
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Frank Heckenbach, 2018/12/17
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Hans Åberg, 2018/12/17
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Frank Heckenbach, 2018/12/18
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Hans Åberg, 2018/12/18
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Frank Heckenbach, 2018/12/19
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Hans Åberg, 2018/12/19
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Frank Heckenbach, 2018/12/22
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Hans Åberg, 2018/12/22
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Frank Heckenbach, 2018/12/22
- Re: Porting to typed C++ parser (was: Dynamic token kinds), Hans Åberg, 2018/12/22
Re: Dynamic token kinds,
Akim Demaille <=
Re: Dynamic token kinds, Frank Heckenbach, 2018/12/21
Re: Dynamic token kinds, Akim Demaille, 2018/12/22
Re: Dynamic token kinds, Frank Heckenbach, 2018/12/22
Re: Dynamic token kinds, Akim Demaille, 2018/12/23
Re: Dynamic token kinds, Frank Heckenbach, 2018/12/23
Re: Dynamic token kinds, Akim Demaille, 2018/12/23
Re: Dynamic token kinds, Frank Heckenbach, 2018/12/23