NAME

OSSP var -- Variable Expansion

SYNOPSIS

Types:

var_rc_t, var_t, var_config_t, var_syntax_t, var_cb_value_t, var_cb_operation_t.

Functions:

var_create, var_destroy, var_config, var_unescape, var_expand, var_formatv, var_format, var_strerror.

Variables:

var_id.

DESCRIPTION

OSSP var is a flexible, full-featured and fast variable construct expansion library. It supports a configurable variable construct syntax very similar to the style found in many scripting languages (like "@" name, ${ name"}", $( name")", etc.) and provides both simple scalar (${ name"}") and array (${name"[" index"]}") expansion, plus optionally one or more post-operations on the expanded value (${ name":" op[":"op...]]"}"). The supported post-operations are length determination, case conversion, defaults, positive and negative alternatives, sub-strings, regular expression based substitutions, character translations, and padding. Additionally, a meta-construct plus arithmetic expressions for index and range calculations allow (even nested) iterations over array variable expansions (.."["..${ name"[#+1]}".."]"..). The actual variable value lookup is performed through a callback function, so OSSP var can expand arbitrary values.

SYNTAX CONSTRUCTS

A string expanded through OSSP var can consist of arbitrary text characters plus one or more of the following special syntax constructs which are expanded by OSSP var.

"\"NNN

Character with the octal value NNN ( N: 0,...,7).

"\x"NN, "\x{"NNMM.."}"

Character with the hexadecimal value NN or the characters denoted by grouped hexadecimal numbers NNMM... (N, M: 0,...,9,["aA"],...,["fF"]).

"\t", "\r", "\n"

Tabulator (TAB), Carriage Return (CR) and Newline (NL) character.

"\\", "\"x

Ordinary character "\" and x.

"$"name, ${name"}"

Contents of scalar variable name.

${name"["index"]""}"

Contents of array variable name at position index. For index full arithmetic expressions are allowed.

${name":#}"

Length of "$"name.

${name":l}", ${name":u}"

"$"name, converted to all lower-case or all upper-case.

${name":-"word"}"

If "$"name is not empty string and not undefined, then "$" name, else word (Default Value).

${name":+"word"}"

If "$"name is empty string, then empty string, else word (Positive Alternative).

${name":*"word"}"

If "$"name is not empty string, then empty string, else word (Negative Alternative).

${name":o"start","[length]"}"

Substring of "$"name starting at position start with length characters.

${name":o"start"-"[end]"}"

Substring of "$"name starting at position start and ending at position end (inclusive).

${name":s/"pattern"/"string"/"["itg"]"}"

"$"name after replacing characters matching pattern with string. By default, case-sensitive regular expression matching is performed and only the first occurrence of pattern is replaced. Flag ""i"" switches to case insensitive matching; flag ""t"" switches to plain text pattern; flag ""g"" switches to replacements of all occurrences; flag ""m"" switches to multi-line matching (That is, change ""^"" and ""$"" from matching the start or end of the string to matching the start or end of any line).

${name":y/"ochars"/"nchars"/}"

"$"name after replacing all characters found in the ochars character class by the corresponding character in the nchars character class.

${name":p/"width"/"string"/"{"l","c","r"}"}"

"$"name after padding to width with string. Original contents of name is either left justified (flag ""l""), centered (flag ""c""), or right justified (flag ""r"").

${name":%"func["("arg")"]"}"

"$"name after passing it to an application-supplied function func. The optional argument arg is passed to the function, too. By default no such functions are defined.

"["body"]", "["body"]""{" start","step","end"}"

Repeat expansion of body as long as at least one array variable does not expand to the empty string (first variant) or exactly ( end- start)/step times (second variant). In both cases the character ""#"" is expanded in body as the current loop index (0,... for first variant and start,...,end with stepping step for second variant). The ""#"" is usually used in the index of array variable lookups. For start, step and end, full arithmetic expressions are allowed. This loop construct can be nested, too. In this case an inner loop is fully repeated for each iteration of the outer loop. Additionally, arithmetic expressions are supported in both start, step, end and index parts of variable constructs in body.

SYNTAX CONSTRUCTS (GRAMMAR)

All the variable syntax constructs supported by OSSP var follow the same grammatical form. For completeness and reference reasons, the corresponding grammar is given in an extended BNF:

 input       ::= ( TEXT
                 ⎪ variable 
                 ⎪ INDEX_OPEN input INDEX_CLOSE (loop_limits)?
                 )*

 variable    ::= DELIM_INIT (name⎪expression)

 name        ::= (NAME_CHARS)+

 expression  ::= DELIM_OPEN 
                 (name⎪variable)+
                 (INDEX_OPEN num_exp INDEX_CLOSE)? 
                 (':' command)*
                 DELIM_CLOSE

 command     ::= '-' (TEXT_EXP⎪variable)+
               ⎪ '+' (TEXT_EXP⎪variable)+
               ⎪ 'o' NUMBER ('-'⎪',') (NUMBER)?
               ⎪ '#'
               ⎪ '*' (TEXT_EXP⎪variable)+
               ⎪ 's' '/' (TEXT_PATTERN)+ 
                     '/' (variable⎪TEXT_SUBST)* 
                     '/' ('m'⎪'g'⎪'i'⎪'t')*
               ⎪ 'y' '/' (variable⎪TEXT_SUBST)+ 
                     '/' (variable⎪TEXT_SUBST)* 
                     '/'
               ⎪ 'p' '/' NUMBER 
                     '/' (variable⎪TEXT_SUBST)* 
                     '/' ('r'⎪'l'⎪'c')
               ⎪ '%' (name⎪variable)+ 
                     ('(' (TEXT_ARGS)? ')')?
               ⎪ 'l'
               ⎪ 'u'

 num_exp     ::= operand
               ⎪ operand ('+'⎪'-'⎪'*'⎪'/'⎪'%') num_exp

 operand     ::= ('+'⎪'-')? NUMBER
               ⎪ INDEX_MARK
               ⎪ '(' num_exp ')'
               ⎪ variable

 loop_limits ::= DELIM_OPEN 
                 (num_exp)? ',' (num_exp)? (',' (num_exp)?)? 
                 DELIM_CLOSE

 NUMBER      ::= ('0'⎪...⎪'9')+

 TEXT_PATTERN::= (^('/'))+
 TEXT_SUBST  ::= (^(DELIM_INIT⎪'/'))+
 TEXT_ARGS   ::= (^(DELIM_INIT⎪')'))+
 TEXT_EXP    ::= (^(DELIM_INIT⎪DELIM_CLOSE⎪':'))+
 TEXT        ::= (^(DELIM_INIT⎪INDEX_OPEN⎪INDEX_CLOSE))+

 DELIM_INIT  ::= '$'                    
 DELIM_OPEN  ::= '{'   
 DELIM_CLOSE ::= '}' 
 INDEX_OPEN  ::= '['    
 INDEX_CLOSE ::= ']'
 INDEX_MARK  ::= '#'
 NAME_CHARS  ::= 'a'⎪...⎪'z'⎪'A'⎪...⎪'Z'⎪'0'⎪...⎪'9'

Notice that the grammar definitions of DELIM_INIT, DELIM_OPEN, DELIM_CLOSE, INDEX_OPEN, INDEX_CLOSE, INDEX_MARK and NAME_CHARS correspond to the default syntax configuration only. They can be changed through the API (see var_syntax_t).

APPLICATION PROGRAMMING INTERFACE (API)

The following is a detailed description of the OSSP var ISO-C language Application Programming Interface (API): TYPES The OSSP var API consists of the following ISO-C data types:

var_rc_t

This is an exported enumerated integer type describing the return code of all API functions. On success, every API function returns "VAR_OK". On error, it returns "VAR_ERR_XXX". For a list of all possible return codes see var.h. Their corresponding describing text can be determined with function var_strerror.

var_t

This is an opaque data type representing a variable expansion context. Only pointers to this abstract data type are used in the API.

var_config_t

This is an exported enumerated integer type describing configuration parameters for function var_config. Currently "VAR_CONFIG_SYNTAX" for configuring the syntax via var_syntax_t, "VAR_CONFIG_CB_VALUE" for configuring the callback for value lookups via var_cb_value_t, and "VAR_CONFIG_CB_OPERATION" for configuring the callback for custom value operation functions via var_cb_operation_t are defined.

var_syntax_t

This is an exported structural data type describing the variable construct syntax. It is passed to var_config on "VAR_CONFIG_SYNTAX" and consists of the following members (directly corresponding to the upper-case non-terminals in the grammar above):

 char  escape;       /* default: '\\' */
 char  delim_init;   /* default: '$' */
 char  delim_open;   /* default: '{' */
 char  delim_close;  /* default: '}' */
 char  index_open;   /* default: '[' */
 char  index_close;  /* default: ']' */
 char  index_mark;   /* default: '#' */
 char *name_chars;   /* default: "a-zA-Z0-9_" */
    

All members are single character constants, except for name_chars which is a character class listing all valid characters. As an abbreviation the construct " x"-"y" is supported which means all characters from x to y (both included) in the underlying character set.

var_cb_value_t

This is an exported function pointer type for variable value lookup functions. Such a callback function cb has to be of the following prototype:

var_rc_t * cb(var_t *var, void *ctx, const char *var_ptr, size_t var_len, int var_idx, const char **val_ptr, size_t * val_len, size_t *val_size);

This function will be called by var_expand internally whenever it has to resolve the contents of a variable. Its parameters are:

var_cb_operation_t

This is an exported function pointer type for variable value operation functions. Such a callback function cb has to be of the following prototype:

var_rc_t * cb(var_t *var, void *ctx, const char *op_ptr, size_t op_len, const char *arg_ptr, size_t arg_len, const char * val_ptr, size_t val_len, const char ** out_ptr, size_t * out_len, size_t *out_size);

This function will be called by var_expand internally whenever a custom operation is used. Its parameters are:

FUNCTIONS The OSSP var API consists of the following ISO-C functions:

var_rc_t var_create(var_t **var);

Create a new variable expansion context and store it into var.

var_rc_t var_destroy(var_t *var);

Destroy the variable expansion context var.

var_rc_t var_config(var_t *var, var_config_t mode, ...);

Configure the variable expansion context var. The variable argument list depends on the mode identifier:

var_rc_t var_unescape(var_t *var, const char *src_ptr, size_t src_len, char *dst_ptr, size_t dst_len, int all);

This expands escape sequences found in the input buffer src_ptr/src_len. The dst_ptr/dst_len point to a output buffer, into which the expanded data is copied if processing is successful. The size of this buffer must be at least src_len+1 characters. The reason is that var_unescape always adds a terminating "NUL" ('"\0"') character at the end of the output buffer, so that you can use the result comfortably with other C library routines. The supplied dst_ptr either has to point to a pre-allocated buffer or is allowed to point to src_ptr (because the unescaping operation is guaranteed to either keep the size or reduce the size of the input).

The parameter all is a boolean flag that modifies the behavior of var_unescape. If is set to true (any value except zero), var_unescape will expand any escape sequences it sees, even those that it does not know about. This means that ""\1"" will become "1", even though ""\1"" has no special meaning to var_unescape. If all is set to false (the value zero), such escape sequences will be copied verbatim to the output buffer.

The quoted pairs supported by var_unescape are ""\t"" (tabulator), ""\r"" (carriage return), ""\n"" (line feed), ""\NNN"" (octal value), ""\xNN"" (hexadecimal value), and ""\x{NNMM..}"" (grouped hexadecimal values).

var_rc_t var_expand(var_t *var, const char *src_ptr, size_t src_len, char **dst_ptr, size_t *dst_len, int force_expand);

This is the heart of OSSP var. It expands all syntax constructs in src_ptr/src_len and stores them in an allocated buffer returned in dst_ptr/dst_len.

The output buffer dst_ptr/dst_len is allocated by var_expand using the system call malloc(3), thus it is the caller's responsibility to free(3) that buffer once it is no longer used anymore. The output buffer for convenience reasons is always "NUL"-terminated by var_expand, but this "NUL" character is not counted for dst_len. The dst_len pointer can be specified as "NULL" if you are not interested in the output buffer length.

The force_expand flag determines how var_expand deals with undefined variables (indicated by the callback function through the return code "VAR_ERR_UNDEFINED_VARIABLE"). If it is set to true (any value except zero), var_expand will fail with error code "VAR_ERR_UNDEFINED_VARIABLE" whenever an undefined variable is encountered. That is, it just passes-through the return code of the callback function. If set to false (value zero), var_expand will copy the expression it failed to expand verbatim into the output buffer, in order to enable you to go over the buffer with an additional pass. Generally, if you do not plan to use multi-pass expansion, you should set force_expand to true in order to make sure no unexpanded variable constructs are left over in the buffer.

If var_expand fails with an error, dst_ptr will point to src_ptr and dst_len will contain the number of characters that have been consumed from src_ptr before the error occurred. In other words, if an error occurs, dst_ptr/dst_len point to the last parsing location in src_ptr/src_len before the error occurred. The only exceptions for this error semantics are: on "VAR_ERR_INVALID_ARGUMENT" and "VAR_ERR_OUT_OF_MEMORY" errors, dst_ptr and dst_len are undefined.

var_rc_t var_formatv(var_t *var, char **dst_ptr, int force_expand, const char *fmt, va_list ap);

This is a high-level function on top of var_expand which expands simple printf(3)-style constructs before expanding the complex variable constructs. So, this is something of a combination between sprintf(3) and var_expand.

It expands simple "%s" (string, type ""char *""), "%d" (integer number, type ""int"") and "%c" (character, type ""int"") constructs in fmt. The values are taken from the variable argument vector ap. After this expansion the result is passed through var_expand by passing through the var, dst_ptr and force_expand arguments. The final result is a malloc(3)'ed buffer provided in dst_ptr which the caller has to free(3) later.

var_rc_t var_format(var_t *var, char **dst_ptr, int force_expand, const char *fmt, ...);

This is just a wrapper around var_formatv which translates the variable argument list into "va_list".

var_rc_t var_strerror(var_t *var, var_rc_t rc, char ** str);

This can be used to map any var_rc_t return codes (as returned by all the OSSP var API functions) into a clear-text message describing the reason for failure in prose. Please note that errors coming from the callback, such as "VAR_ERR_CALLBACK" and those based on it, cannot be mapped and will yield the message ""unknown error"".

 

VARIABLES The OSSP var API consists of the following ISO-C exported variables:

var_id

This is just a pointer to the constant string ""OSSP var"". It is used as the first argument in ex_trow calls if OSSP var is built with OSSP ex support. It then allows the application to determine whether a caught exception was thrown by OSSP var. See EXCEPTION HANDLING below for more details.

COMBINING UNESCAPING AND EXPANSION

For maximum power and flexibility, you usually want to combine var_unescape and var_expand. That is, you will want to use var_unescape to turn all escape sequences into their real representation before you call var_expand for expanding variable constructs. This way the user can safely use specials like ""\n"" or ""\t"" throughout the template and achieve the desired effect. These escape sequences are particularly useful if search-and-replace or transpose actions are performed on variables before they are expanded. Be sure, though, to make the first var_unescape pass with the all flag set to false, or the routine will also expand escape sequences like ""\1"", which might have a special meaning (regular expression back-references) in the var_expand pass to follow. Once all known escape sequences are expanded, expand the variables with var_expand. After that, you will want to have a second pass with var_unescape and the flag all set to true, to make sure all remaining escape sequences are expanded. Also, the var_expand pass might have introduced now quoted pairs into the output text, which you need to expand to get the desired effect.

EXCEPTION HANDLING

OSSP var can be optionally built with support for exception handling via OSSP ex (see http://www.ossp.org/pkg/lib/ex/). For this it has to be configured with the GNU Autoconf option "--with-ex". The difference then is that the OSSP var API functions throw exceptions instead of returning "VAR_ERR_XXX" return codes. The thrown exceptions can be identified as OSSP var exceptions by checking the exception attribute ex_class. It is the OSSP var API symbol var_id for all OSSP var exceptions. The ex_object attribute is always "NULL". The ex_value attribute is the var_rc_t which forced the throwing of the exception. Exception throwing can be suppressed with ex_shield only.

EXAMPLE (DEVELOPER)

The following simple but complete program illustrates the full usage of OSSP var. It accepts a single argument on the command line and expands this in three steps (unescaping known escape sequences, expanding variable constructs, unescaping new and unknown escape sequences). The value lookup callback uses the process environment to resolve variables.

 #include <stdio.h>
 #include <stdlib.h>
 #include <string.h>

 #include "var.h"

 static var_rc_t lookup(
     var_t *var, void *ctx,
     const char  *var_ptr, size_t  var_len, int     var_idx,
     const char **val_ptr, size_t *val_len, size_t *val_size)
 {
     char tmp[256];

     if (var_idx != 0)
         return VAR_ERR_ARRAY_LOOKUPS_ARE_UNSUPPORTED;
     if (var_len > sizeof(tmp) - 1)
         return VAR_ERR_OUT_OF_MEMORY;
     memcpy(tmp, var_ptr, var_len);
     tmp[var_len] = '\0';
     if ((*val_ptr = getenv(tmp)) == NULL)
         return VAR_ERR_UNDEFINED_VARIABLE;
     *val_len = strlen(*val_ptr);
     *val_size = 0;
     return VAR_OK;
 }

 static void die(const char *context, var_t *var, var_rc_t rc)
 {
     char *error;

     var_strerror(var, rc, &error);
     fprintf(stderr, "ERROR: %s: %s (%d)\n", context, error, rc);
     exit(1);
 }

 int main(int argc, char *argv[])
 {
     var_t *var;
     var_rc_t rc;
     char *src_ptr;
     char *dst_ptr;
     size_t src_len;
     size_t dst_len;
     var_syntax_t syntax = { '\\', '$', '{', '}', '[', ']', '#', "a-zA-Z0-9_" };

     /* command line handling */
     if (argc != 2)
         die("command line", NULL, VAR_ERR_INVALID_ARGUMENT);
     src_ptr = argv[1];
     src_len = strlen(src_ptr);
     fprintf(stdout, "input:     \"%s\"\n", src_ptr);

     /* establish variable expansion context */
     if ((rc = var_create(&var)) != VAR_OK)
         die("create context", NULL, rc);
     if ((rc = var_config(var, VAR_CONFIG_SYNTAX, &syntax)) != VAR_OK)
         die("configure syntax", var, rc);
     if ((rc = var_config(var, VAR_CONFIG_CB_VALUE, lookup, NULL)) != VAR_OK)
         die("configure callback", var, rc);

     /* unescape known escape sequences (in place) */
     if ((rc = var_unescape(var, src_ptr, src_len, src_ptr, src_len+1, 0)) != VAR_OK)
         die("unescape known escape sequences", var, rc);
     src_len = strlen(src_ptr);
     fprintf(stdout, "unescaped: \"%s\"\n", src_ptr);

     /* expand variable constructs (force expansion) */
     if ((rc = var_expand(var, src_ptr, src_len, &dst_ptr, &dst_len, 1)) != VAR_OK) {
         if (rc != VAR_ERR_INVALID_ARGUMENT && rc != VAR_ERR_OUT_OF_MEMORY) {
             fprintf(stdout, "parsing:   \"%s\"\n", dst_ptr); 
             fprintf(stdout, "             %*s\n", dst_len, "^"); 
         }
         die("variable expansion", var, rc);
     }
     fprintf(stdout, "expanded:  \"%s\"\n", dst_ptr);

     /* unescape new and unknown escape sequences (in place) */
     if ((rc = var_unescape(var, dst_ptr, dst_len, dst_ptr, dst_len+1, 1)) != VAR_OK)
         die("unescape new and unknown escape sequences", var, rc);
     fprintf(stdout, "output:    \"%s\"\n", dst_ptr);
     free(dst_ptr);

     /* destroy variable expansion context */
     if ((rc = var_destroy(var)) != VAR_OK)
         die("destroy context", var, rc);

     return 0;
 }

Copy & paste the source code into a file var_play.c (or use the version already shipped with the OSSP var source distribution), compile it with

 $ cc `var-config --cflags` \
   -o var_play var_play.c \
   `var-config --ldflags --libs`

and use it to play with the various OSSP var variable expansion possibilities.

EXAMPLE (USER)

The following are a few sample use cases of OSSP var variable expansions. They all assume the default syntax configuration and the following variable definitions: "$foo=foo" (a scalar), "$bar=<bar1,bar2,bar3,>" (an array), "$baz=<baz1,baz2,baz3,>" (another array), "$quux=quux" (another scalar), "$name=<foo,bar,baz,quux>" (another scalar) and "$empty=""" (another scalar).

 Input                         Output
 ----------------------------- --------------
 $foo                          foo
 ${foo}                        foo
 ${bar[0]}                     bar1
 ${${name[1]}[0]}              bar1
 ${foo:u:y/O/U/:s/(.*)/<\1>/}  <FUU>
 ${foo:u:y/O/U/:s/(.*)/<\1>/}  <FUU>
 ${empty:-foo}                 foo
 ${foo:+yes}${foo:*no}         yes
 ${empty:+yes}${empty:*no}     no
 ${foo:p/6/./l}                foo...
 ${foo:p/6/./r}                ...foo
 [${bar[#]}${bar[#+1]:+,}]     bar1,bar2,bar3
 [${bar[#-1]:+,}${bar[#]}]     bar1,bar2,bar3
 [${bar[#]}]{2,1,3}            bar2bar3
 [${bar[#]}]{1,2,3}            bar1bar3
 [${foo[#]}[${bar[#]}]]{1,,2}  foo1bar1bar2bar3foo2bar1bar2bar3

SEE ALSO

pcre(3), regex(7), OSSP val (Value Access), OSSP ex (Exception Handling).

HISTORY

OSSP var was initially written by Peter Simons <simons@crypt.to> in November 2001 under contract with the OSSP sponsor Cable & Wireless. Its API and internal code structure was revamped in February 2002 by Ralf S. Engelschall <rse@engelschall.com> to fully conform to the OSSP library standards. Before its initial public release, Ralf S. Engelschall in March 2002 finally added support for custom operations, the formatting functionality, optional multi-line matching, etc.