pyexpander package

Submodules

pyexpander.lib module

The main pyexpander library.

class pyexpander.lib.BeginBlock(previous=None, filename=None)[source]

Bases: pyexpander.lib.Block

implements a $begin .. $end block.

This block is simply a variable scope, so it is derived from Block where the constructor is called with new_scope=True.

class pyexpander.lib.Block(previous=None, new_scope=False, filename=None, parse_list=None, external_definitions=None)[source]

Bases: object

class that represents a block in the expander language.

Each block has a parent-pointer, which may be None. Blocks are used to represent syntactical blocks in the expander language. They manage the global variable dictionary and some more properties that are needed during execution of the program.

add_macro(name, macro_block)[source]

add a new macro block.

eval_(st)[source]

perform eval with the global variable dictionary of the block.

Here is an example:

>>> b= Block(parse_list=[])
>>> b.exec_("a=2")
>>> b.eval_("3*a")
6
exec_(st)[source]

perform exec with the global variable dictionary of the block.

Here is an example:

>>> b= Block(parse_list=[])
>>> b.exec_("a=1")
>>> b["a"]
1
>>> b= Block(b,True)
>>> b["a"]
1
>>> b.exec_("a=2")
>>> b["a"]
2
>>> b.previous["a"]
1
export_symbols(lst)[source]

appends items to the export_symbols list.

This list is used by the pop() method in order to copy values from the current global variable dictionary to the global variable dictionary of the previous block.

Here is an example:

>>> b= Block(parse_list=[])
>>> b.export_symbols(["a","b"])
>>> b.exported_syms
['a', 'b']
>>> b.export_symbols(["d","e"])
>>> b.exported_syms
['a', 'b', 'd', 'e']
extend(lst)[source]

adds items to the list of expander functions or variables.

Here is an example:

>>> a=1
>>> b=2
>>> def t(x):
...   return x+1
...
>>> block= Block(parse_list=[],external_definitions=globals())
>>> block.extend(["a","b","t"])
>>> _pr_set(block.direct_vars)
set(['a', 'b'])
>>> _pr_set(block.direct_funcs)
set(['t'])
format_text(text, indent_start)[source]

currently does indent one or more lines.

get_block_list()[source]

returns all blocks of the list.

The list is returned with the oldest block first.

get_indent()[source]

gets the indent.

parse_elm()[source]

get the current parse element.

parse_loop()[source]

loop across the items in the Block.

returns False if there is nothing more to parse in the current block.

pop()[source]

removes the current block and returns the previous one.

Here is an example:

>>> b= Block(parse_list=[])
>>> b["a"]=1
>>> b["b"]=2
>>> b= Block(b,True)
>>> b["a"]=10
>>> b["b"]=20
>>> b.export_symbols(["a"])
>>> b= b.pop()
>>> b["a"]
10
>>> b["b"]
2
posmsg(msg=None, pos=None)[source]

return a message together with a position.

This method is mainly used for error messages. We usually want to print filename and the line and column number together with a message. This method returns the given message string <msg> together with this additional information.

print_block_list()[source]

print all blocks in the list.

The list is returned with the oldest block first.

Here is an example:

>>> b= Block(parse_list=[])
>>> print(b)
Block{
    has parent          : False
    filename            = None
    template            = None
    template_path       = None
    has template_parselist_cache: False
    exported_syms       = []
    direct_vars         = set([])
    direct_funcs        = set([])
    macros              : []
    new_scope           = False
    skip                = False
    indent              = 0
    new_parse_list      = True
    lst_pos             = -1
    start_pos           = 0
}
>>> b= Block(b,True)
>>> print(b)
Block{
    has parent          : True
    filename            = None
    template            = None
    template_path       = None
    has template_parselist_cache: False
    exported_syms       = []
    direct_vars         = set([])
    direct_funcs        = set([])
    macros              : []
    new_scope           = True
    skip                = False
    indent              = 0
    new_parse_list      = False
    lst_pos             = -1
    start_pos           = 0
}
>>> b.print_block_list()
Block{
    has parent          : False
    filename            = None
    template            = None
    template_path       = None
    has template_parselist_cache: False
    exported_syms       = []
    direct_vars         = set([])
    direct_funcs        = set([])
    macros              : []
    new_scope           = False
    skip                = False
    indent              = 0
    new_parse_list      = True
    lst_pos             = -1
    start_pos           = 0
}
Block{
    has parent          : True
    filename            = None
    template            = None
    template_path       = None
    has template_parselist_cache: False
    exported_syms       = []
    direct_vars         = set([])
    direct_funcs        = set([])
    macros              : []
    new_scope           = True
    skip                = False
    indent              = 0
    new_parse_list      = False
    lst_pos             = -1
    start_pos           = 0
}
set_indent(value)[source]

sets the indent.

set_substfile(filename, tp)[source]

set substitution filename.

Used for the “$subst”, “$template” and “$pattern” commands.

setdefault(name, val)[source]

return a value from globals, set to a default if it’s not defined.

Here are some examples:

>>> b= Block(parse_list=[])
>>> b["a"]
Traceback (most recent call last):
    ...
NameError: name 'a' is not defined at unknown position
>>> b.setdefault("a",10)
10
>>> b["a"]
10
>>> b["a"]=11
>>> b.setdefault("a",10)
11
str_block_list()[source]

returns a string representation of all blocks in the list.

The list is returned with the oldest block first.

str_eval(st)[source]

perform eval with the global variable dictionary of the block.

Here is an example:

>>> b= Block(parse_list=[])
>>> b.exec_("a=2")
>>> b.str_eval("3*a")
'6'
substfile_parselist(include_paths)[source]

return the parselist for a given file.

This method manages a cache of that assigns filenames to parselists. This can speed things up if a file is included several times.

class pyexpander.lib.ForBlock(previous=None, new_scope=False, value_list=None, var_expr='')[source]

Bases: pyexpander.lib.Block

implements a $for .. $endfor block.

next_loop()[source]

performs next loop.

returns:
True when the loop is not yet finished.
set_loop_var()[source]

set the loop variable to a new value.

class pyexpander.lib.IfBlock(previous=None, condition=True)[source]

Bases: pyexpander.lib.Block

implements a $if .. $else .. $endif block.

An $if block never has a variable scope, so the base Block object is called with new_scope=False.

enter_elif(condition)[source]

enter the “elif” part in the if..endif block.

enter_else()[source]

this should be called when $else is encountered.

class pyexpander.lib.IncludeBlock(previous=None, new_scope=False, filename=None, include_paths=None)[source]

Bases: pyexpander.lib.Block

implements a $include(filename) block.

This block is simply a variable scope, so it is derived from Block where the constructor is called with new_scope=True.

class pyexpander.lib.MacBlock(previous=None, declaration_block=None, parameter_list=None)[source]

Bases: pyexpander.lib.Block

implements a $macro...$endmacro block.

pop()[source]

override pop() from base class.

class pyexpander.lib.PatternBlock(previous=None, filename=None, include_paths=None, heading=None, lines=None)[source]

Bases: pyexpander.lib.Block

implements a $pattern(parameters) block.

This block is simply a variable scope, so it is derived from Block where the constructor is called with new_scope=True.

def_vars()[source]

define all the given variables in the PatternBlock.

pop()[source]
class pyexpander.lib.ResultText[source]

Bases: object

basically a list of strings with a current column property.

append(text)[source]

append some text.

column()[source]

return current column.

static current_column(st)[source]

find current column if the string is printed.

Note: With a string ending with ‘n’ this returns 1.

Here are some examples: >>> current_column(“”) -1 >>> ResultText.current_column(“ab”) -1 >>> ResultText.current_column(“nab”) 3 >>> ResultText.current_column(“nabn”) 1 >>> ResultText.current_column(“nabna”) 2

list_()[source]

return internal list.

class pyexpander.lib.SubstBlock(previous=None, filename=None, include_paths=None, external_definitions=None)[source]

Bases: pyexpander.lib.Block

implements a $subst(parameters) block.

This block is simply a variable scope, so it is derived from Block where the constructor is called with new_scope=True.

class pyexpander.lib.WhileBlock(previous=None, new_scope=False, while_expr='')[source]

Bases: pyexpander.lib.Block

implements a $while .. $endwhile block.

next_loop()[source]

performs next loop.

returns:
True when the loop is not yet finished.
pyexpander.lib.expand(st, filename=None, external_definitions=None, allow_nobracket_vars=False, auto_continuation=False, auto_indent=False, include_paths=None)[source]

returns the global dict.

pyexpander.lib.expandFile(filename, external_definitions=None, allow_nobracket_vars=False, auto_continuation=False, auto_indent=False, include_paths=None, no_stdin_warning=False)[source]

returns the global dict.

pyexpander.lib.find_file(filename, include_paths)[source]

find a file in a list of include paths.

include_paths MUST CONTAIN “” in order to search the local directory.

pyexpander.lib.keyword_check(identifiers)[source]

indentifiers must not be identical to keywords.

This function may raise an exception.

pyexpander.lib.parseFile(filename, no_stdin_warning)[source]

parse a file.

pyexpander.lib.parseString(st)[source]

parse a string.

pyexpander.lib.processToList(parse_list, filename=None, external_definitions=None, allow_nobracket_vars=False, auto_continuation=False, auto_indent=False, include_paths=None)[source]

run pyexpander, return a list of strings.

pyexpander.lib.processToPrint(parse_list, filename=None, external_definitions=None, allow_nobracket_vars=False, auto_continuation=False, auto_indent=False, include_paths=None)[source]

returns the global dict.

pyexpander.parser module

implement the parser for pyexpander.

class pyexpander.parser.IndexedString(st)[source]

Bases: object

a string together with row column information.

Here is an example:

>>> txt='''01234
... 67
... 9abcd'''
>>> l=IndexedString(txt)
>>> l.rowcol(0)
(1, 1)
>>> l.rowcol(1)
(1, 2)
>>> l.rowcol(4)
(1, 5)
>>> l.rowcol(5)
(1, 6)
>>> l.rowcol(6)
(2, 1)
>>> l.rowcol(7)
(2, 2)
>>> l.rowcol(8)
(2, 3)
>>> l.rowcol(9)
(3, 1)
>>> l.rowcol(13)
(3, 5)
>>> l.rowcol(14)
(3, 6)
>>> l.rowcol(16)
(3, 8)
rowcol(pos)[source]

calculate (row,column) from a string position.

st()[source]

return the raw string.

exception pyexpander.parser.ParseException(value, pos=None, rowcol=None)[source]

Bases: Exception

used for Exceptions in this module.

class pyexpander.parser.ParsedCommand(idxst, start, end, ident)[source]

Bases: pyexpander.parser.ParsedItem

class of a pyexpander command with arguments.

A command has the form “$name(argument1, argument2, ...)”.

args()[source]

return the arguments of the command.

class pyexpander.parser.ParsedComment(idxst, start, end)[source]

Bases: pyexpander.parser.ParsedItem

class of a parsed comment.

A comment in pyexpander starts with ‘$#’.

class pyexpander.parser.ParsedEval(idxst, start, end)[source]

Bases: pyexpander.parser.ParsedItem

class of an pyexpander expression.

A pyexpander expression has the form “$(expression)” e.g. “$(a+1)”. This is different from ParsedVar where the string within the brackets is a simple identifier.

class pyexpander.parser.ParsedItem(idxst, start, end)[source]

Bases: object

base class of parsed items.

end()[source]

return the end of the ParsedItem in the source string.

positions()[source]

return start and end of ParsedItem in the source string.

rowcol(pos=None)[source]

calculate (row,column) from a string position.

start()[source]

return the start of the ParsedItem in the source string.

string()[source]

return the string that represents the ParsedItem.

class pyexpander.parser.ParsedLiteral(idxst, start, end)[source]

Bases: pyexpander.parser.ParsedItem

class of a parsed literal.

A literal is a substring in the input that shouldn’t be modified by pyexpander.

class pyexpander.parser.ParsedPureCommand(idxst, start, end)[source]

Bases: pyexpander.parser.ParsedItem

class of a pyexpander command without arguments.

A pure command has the form “$name”. Such a command has no arguments which would be enclosed in round brackets immediately following the name.

class pyexpander.parser.ParsedVar(idxst, start, end)[source]

Bases: pyexpander.parser.ParsedItem

class of a parsed variable.

A variable in pyexpander has the form “$(identifier)”.

pyexpander.parser.change_linesep(sep)[source]

change line separator, this is here just for tests.

pyexpander.parser.parseAll(idxst, pos)[source]

parse everything.

>>> def test(st,pos):
...     idxst= IndexedString(st)
...     pprint(parseAll(idxst,pos))
...
>>> test("abc",0)
('ParsedLiteral', (0, 2), 'abc')
>>> test("abc$xyz",0)
('ParsedLiteral', (0, 2), 'abc')
('ParsedPureCommand', (4, 6), 'xyz')
>>> test("abc${xyz}efg",0)
('ParsedLiteral', (0, 2), 'abc')
('ParsedPureCommand', (5, 7), 'xyz')
('ParsedLiteral', (9, 11), 'efg')
>>> test("abc$xyz(2*4)",0)
('ParsedLiteral', (0, 2), 'abc')
('ParsedCommand', (8, 10), '2*4', 'xyz')
>>> test("abc$(2*4)ab",0)
('ParsedLiteral', (0, 2), 'abc')
('ParsedEval', (5, 7), '2*4')
('ParsedLiteral', (9, 10), 'ab')
>>> test("abc\\$(2*4)ab",0)
('ParsedLiteral', (0, 2), 'abc')
('ParsedLiteral', (4, 4), '$')
('ParsedLiteral', (5, 11), '(2*4)ab')
>>> test("ab$func(1+2)\\\nnew line",0)
('ParsedLiteral', (0, 1), 'ab')
('ParsedCommand', (8, 10), '1+2', 'func')
('ParsedLiteral', (14, 21), 'new line')
>>> test("ab$func(1+2)\nnew line",0)
('ParsedLiteral', (0, 1), 'ab')
('ParsedCommand', (8, 10), '1+2', 'func')
('ParsedLiteral', (12, 20), '\nnew line')
>>> test("ab$(xyz)(56)",0)
('ParsedLiteral', (0, 1), 'ab')
('ParsedVar', (4, 6), 'xyz')
('ParsedLiteral', (8, 11), '(56)')
>>> test(r'''
... Some text with a macro: $(xy)
... an escaped dollar: \$(xy)
... a macro within letters: abc${xy}def
... a pyexpander command structure:
... $if(a=1)
... here
... $else
... there
... $endif
... now a continued\
... line
... from here:$# the rest is a comment
... now an escaped continued\\
... line
... ''',0)
('ParsedLiteral', (0, 24), '\nSome text with a macro: ')
('ParsedVar', (27, 28), 'xy')
('ParsedLiteral', (30, 49), '\nan escaped dollar: ')
('ParsedLiteral', (51, 51), '$')
('ParsedLiteral', (52, 83), '(xy)\na macro within letters: abc')
('ParsedPureCommand', (86, 87), 'xy')
('ParsedLiteral', (89, 124), 'def\na pyexpander command structure:\n')
('ParsedCommand', (129, 131), 'a=1', 'if')
('ParsedLiteral', (133, 138), '\nhere\n')
('ParsedPureCommand', (140, 143), 'else')
('ParsedLiteral', (144, 150), '\nthere\n')
('ParsedPureCommand', (152, 156), 'endif')
('ParsedLiteral', (157, 172), '\nnow a continued')
('ParsedLiteral', (175, 189), 'line\nfrom here:')
('ParsedComment', (192, 214), ' the rest is a comment\n')
('ParsedLiteral', (215, 238), 'now an escaped continued')
('ParsedLiteral', (239, 239), '\\')
('ParsedLiteral', (241, 246), '\nline\n')
pyexpander.parser.parseBackslash(idxst, pos)[source]

parses a backslash.

>>> def test(st,pos,sep=None):
...     if sep:
...         change_linesep(sep)
...     idxst= IndexedString(st)
...     (p,elm)= parseBackslash(idxst,pos)
...     print("Parsed: %s" % elm)
...     print("rest of string:%s" % repr(st[p:]))
...     change_linesep(os.linesep)
...
>>> test(r"\abc",0)
Parsed: ('ParsedLiteral', (0, 0), '\\')
rest of string:'abc'
>>> test("\\",0)
Parsed: ('ParsedLiteral', (0, 0), '\\')
rest of string:''
>>> test("\\\rab",0,"\r")
Parsed: None
rest of string:'ab'
>>> test("\\\rab",0,"\n")
Parsed: ('ParsedLiteral', (0, 0), '\\')
rest of string:'\rab'
>>> test("\\\rab",0,"\r\n")
Parsed: ('ParsedLiteral', (0, 0), '\\')
rest of string:'\rab'
>>> test("\\\nab",0,"\r")
Parsed: ('ParsedLiteral', (0, 0), '\\')
rest of string:'\nab'
>>> test("\\\nab",0,"\n")
Parsed: None
rest of string:'ab'
>>> test("\\\nab",0,"\r\n")
Parsed: ('ParsedLiteral', (0, 0), '\\')
rest of string:'\nab'
>>> test("\\\r\nab",0,"\r")
Parsed: None
rest of string:'\nab'
>>> test("\\\r\nab",0,"\n")
Parsed: ('ParsedLiteral', (0, 0), '\\')
rest of string:'\r\nab'
>>> test("\\\r\nab",0,"\r\n")
Parsed: None
rest of string:'ab'
pyexpander.parser.parseBracketed(idxst, pos)[source]

parse an identifier in curly brackets.

Here are some examples:

>>> def test(st,pos):
...     idxst= IndexedString(st)
...     (a,b)= parseBracketed(idxst,pos)
...     print(st[a:b])
...
>>> test(r'{abc}',0)
{abc}
>>> test(r'{ab8c}',0)
{ab8c}
>>> test(r'{c}',0)
{c}
>>> test(r'{}',0)
Traceback (most recent call last):
    ...
ParseException: command enclosed in curly brackets at line 1, col 1
>>> test(r'{abc',0)
Traceback (most recent call last):
    ...
ParseException: command enclosed in curly brackets at line 1, col 1
>>> test(r'x{ab8c}',1)
{ab8c}
pyexpander.parser.parseCode(idxst, pos)[source]

parse python code, it MUST start with a ‘(‘.

Here are some examples:

>>> def test(st,pos):
...     idxst= IndexedString(st)
...     (a,b)= parseCode(idxst,pos)
...     print(st[a:b])
...
>>> test(r'(a+b)',0)
(a+b)
>>> test(r'(a+(b*c))',0)
(a+(b*c))
>>> test(r'(a+(b*c)+")")',0)
(a+(b*c)+")")
>>> test(r"(a+(b*c)+''')''')",0)
(a+(b*c)+''')''')
>>> test(r"(a+(b*c)+''')'''+# comment )\n)",0)
Traceback (most recent call last):
    ...
ParseException: end of bracket expression not found at line 1, col 1
>>>
>>> test("(a+(b*c)+''')'''+# comment )\n)",0)
(a+(b*c)+''')'''+# comment )
)
pyexpander.parser.parseComment(idxst, pos)[source]

parse a python comment.

Here are some examples:

>>> def test(st,pos,sep=None):
...     if sep:
...         change_linesep(sep)
...     idxst= IndexedString(st)
...     (a,b)= parseComment(idxst,pos)
...     print(repr(st[a:b]))
...     change_linesep(os.linesep)
...
>>> test("#abc",0)
'#abc'
>>> test("#abc\nef",0,"\n")
'#abc\n'
>>> test("#abc\r\nef",0,"\r\n")
'#abc\r\n'
>>> test("xy#abc",2)
'#abc'
>>> test("xy#abc\nef",2,"\n")
'#abc\n'
>>> test("xy#abc\nef",3)
Traceback (most recent call last):
    ...
ParseException: start of comment not found at line 1, col 4
pyexpander.parser.parseDollar(idxst, pos)[source]

parse things that follow a dollar.

Here are some examples:

>>> def test(st,pos):
...   idxst= IndexedString(st)
...   (p,elm)= parseDollar(idxst,pos)
...   print("Parsed: %s" % elm)
...   print("rest of string:%s" % st[p:])
...
>>> test("$abc",0)
Parsed: ('ParsedPureCommand', (1, 3), 'abc')
rest of string:
>>> test("$abc%&/",0)
Parsed: ('ParsedPureCommand', (1, 3), 'abc')
rest of string:%&/
>>> test("$abc(2*3)",0)
Parsed: ('ParsedCommand', (5, 7), '2*3', 'abc')
rest of string:
>>> test(" $abc(2*sin(x))",1)
Parsed: ('ParsedCommand', (6, 13), '2*sin(x)', 'abc')
rest of string:
>>> test(" $abc(2*sin(x))bn",1)
Parsed: ('ParsedCommand', (6, 13), '2*sin(x)', 'abc')
rest of string:bn
>>> test(" $# a comment\nnew line",1)
Parsed: ('ParsedComment', (3, 13), ' a comment\n')
rest of string:new line
>>> test("$(abc)",0)
Parsed: ('ParsedVar', (2, 4), 'abc')
rest of string:
>>> test("$(abc*2)",0)
Parsed: ('ParsedEval', (2, 6), 'abc*2')
rest of string:
>>> test(" $(2*x(y))abc",1)
Parsed: ('ParsedEval', (3, 8), '2*x(y)')
rest of string:abc
pyexpander.parser.parseStringLiteral(idxst, pos)[source]

parse a python string literal.

returns 2 numbers, the index where the string starts and the index of the first character after the string

Here are some examples:

>>> def test(st,pos):
...     idxst= IndexedString(st)
...     (a,b)= parseStringLiteral(idxst,pos)
...     print(st[a:b])
...
>>> test(r'''"abc"''',0)
"abc"
>>> test("'''ab'c'd'''",0)
'''ab'c'd'''
>>> test("'''ab'cd''''",0)
'''ab'cd'''
>>> test(r'''U"abc"''',0)
U"abc"
>>> test(r'''xU"abc"''',1)
U"abc"
>>> test(r'''xUr"abc"''',1)
Ur"abc"
>>> test(r'''xUr"ab\\"c"''',1)
Ur"ab\\"
>>> test(r'''xUr"ab\"c"''',1)
Ur"ab\"c"
>>> test(r'''xUr"ab\"c"''',0)
Traceback (most recent call last):
    ...
ParseException: start of string expected at line 1, col 1
>>> test(r'''"ab''',0)
Traceback (most recent call last):
    ...
ParseException: end of string not found at line 1, col 1
>>> test(r"'''ab'",0)
Traceback (most recent call last):
    ...
ParseException: end of string not found at line 1, col 1
>>> test(r'''"ab\"''',0)
Traceback (most recent call last):
    ...
ParseException: end of string not found at line 1, col 1
pyexpander.parser.pprint(parselist)[source]

pretty print a parselist.

pyexpander.parser.scanPyIdentList(st)[source]

scan a list of python identifiers.

Here are some examples:

>>> scanPyIdentList("a,b")
['a', 'b']
>>> scanPyIdentList("a,b.d, c")
['a', 'b.d', 'c']
>>> scanPyIdentList("a,b.d, c&")
Traceback (most recent call last):
    ...
ParseException: list of python identifiers expected
pyexpander.parser.scanPyIn(st)[source]

scan a python “in” statement.

Here are some examples:

>>> scanPyIn(" (a,b) in k.items() ")
('(a,b)', 'in', 'k.items()')

Module contents