NAME

perl5420delta - what is new for perl v5.42.0

DESCRIPTION

This document describes differences between the 5.42.0 release and the 5.40.0 release.

Core Enhancements

    class Point {
        field $x :reader :writer :param;
        field $y :reader :writer :param;
    }
    my $p = Point->new( x => 20, y => 40 );
    $p->set_x(60);

    use v5.42;
    use feature 'keyword_all';
    no warnings 'experimental::keyword_all';
    my @numbers = ...
    if ( all { $_ % 2 == 0 } @numbers ) {
        say "All the numbers are even";
    }

  no feature "apostrophe_as_package_separator";

  my $symref = "My'Module'Var";
  # default features
  my $x = $My'Module'Var; # fine
  no feature "apostrophe_as_package_separator";
  no strict "refs";
  my $y = $$symref;       # like $My::Module::Var
  my $z = $My'Module'Var; # syntax error

  # no switch here
  use v5.10;
  # switch here
  use v5.36;
  # no switch here
  use feature "switch";
  # switch here

  # smartmatch here
  use v5.41;
  # no smartmatch here
  use feature "smartmatch";
  # smartmatch here

Security

  $ perl -e '$_ = "\x{FF}" x 1000000; tr/\xFF/\x{100}/;'
  Segmentation fault (core dumped)

Incompatible Changes

Performance Enhancements

• Constant-folded strings are now shareable via the Copy-on-Write mechanism. [GH #22163 <https://github.com/Perl/perl5/pull/22163>]

  The following code would previously have allocated eleven string buffers, each containing one million "A"s:

      my @scalars; push @scalars, ("A" x 1_000_000) for 0..9;
      

  Now a single buffer is allocated and shared between a CONST OP and the ten scalar elements of @scalars.

  Note that any code using this sort of constant to simulate memory leaks (perhaps in test files) must now permute the string in order to trigger a string copy and the allocation of separate buffers. For example, "("A" x 1_000_000).time" might be a suitable small change.

• "tr///" now runs at the same speed regardless of the internal representation of its operand, as long as the only characters being translated are ASCII-range, for example "tr/A-Z/a-z/". Previously, if the internal encoding was UTF-8, a slower, more general implementation was used.
• Code that uses the "indexed" function from the builtin module to generate a list of index/value pairs out of an array or list which is then passed into a two-variable "foreach" list to unpack those again is now optimised to be more efficient.

      my @array = (...);
      foreach my ($idx, $val) (builtin::indexed @array) {
          ...
      }
      

      foreach my ($idx, $val) (builtin::indexed LIST...) {
          ...
      }
      

  In particular, a temporary list twice the size of the original is no longer generated. Instead, the loop iterates down the original array or list in-place directly, in the same way that "foreach (@array)" or "foreach (LIST)" would do.

• The peephole optimizer recognises the following zero-offset "substr" patterns and swaps in a new dedicated operator ("OP_SUBSTR_LEFT"). [GH #22785 <https://github.com/Perl/perl5/issues/22785>]

      substr($x, 0, ...)
      substr($x, 0, ..., '')
      

• The stringification of integers by "print" in perlfunc and "say" in perlfunc, when coming from an "SVt_IV", is now more efficient. [GH #22927 <https://github.com/Perl/perl5/issues/22927>]
• String reversal from a single argument, when the string buffer is not "swiped", is now done in a single pass and is noticeably faster. The extent of the improvement is compiler & hardware dependent. [GH #23012 <https://github.com/Perl/perl5/issues/23012>]

Modules and Pragmata

Documentation

Diagnostics

The following additions or changes have been made to diagnostic output, including warnings and fatal error messages. For the complete list of diagnostic messages, see perldiag.

Utility Changes

Configuration and Compilation

• Fix compilation on platforms (e.g. "Gentoo Prefix") with only a C locale [GH #22569 <https://github.com/Perl/perl5/issues/22569>] Bug first reported downstream bugs.gentoo.org/939014 <https://bugs.gentoo.org/939014>
• The (mostly undocumented) configuration macro "PERL_STRICT_CR" has been removed. When enabled (e.g. with "./Configure -A ccflags=-DPERL_STRICT_CR"), it would make the perl parser throw a fatal error when it encountered a CR (carriage return) character in source files. The default (and now only) behavior of the perl parser is to strip CRs paired with newline characters and otherwise treat them as whitespace.

  ("PERL_STRICT_CR" was originally introduced in perl 5.005 to optionally restore backward compatibility with perl 5.004, which had made CR in source files an error. Before that, CR was accepted, but retained literally in quoted multi-line constructs such as here-documents, even at the end of a line.)

• Similarly, the (even less documented) configuration macro "PERL_CR_FILTER" has been removed. When enabled, it would install a default source filter to strip carriage returns from source code before the parser proper got to see it.

Testing

Tests were added and changed to reflect the other additions and changes in this release. Furthermore, these significant changes were made:

• A new t/run/todo.t test script was added as a place for TODO tests for known unfixed bugs. Patches are welcome to add to this file.
• Added testing of the perl headers against the C++ compiler corresponding to the C compiler perl is being built with. [GH #22232 <https://github.com/Perl/perl5/issues/22232>]

Platform Support

Internal Changes

• The ""sv_strftime_ints"" in perlapi function is introduced. This is an enhanced version of ""my_strftime"" in perlapi, which is retained for backwards compatibility. Both are to call strftime(3) when you have the year, month, hour, etc. The new function handles UTF8ness for you, and allows you to specify if you want the possibility of daylight savings time to be considered. "my_strftime" never considers DST.
• The "bytes_to_utf8", "bytes_from_utf8", and "bytes_from_utf8_loc" functions are no longer experimental.
• Calls to call_argv() with the "G_DISCARD" flag set also ensure the SV parameters constructed from the "argv" parameter are released before call_argv() returns. Previously they were released on the next FREETMPS. [GH #22255 <https://github.com/Perl/perl5/issues/22255>]
• When built with the "-DDEBUGGING" compile option, perl API functions that take pointers to distinct types of SVs (AVs, HVs or CVs) will check the SvTYPE() of the passed values to ensure they are valid. Additionally, internal code within core functions that attempts to extract AVs, HVs or CVs from reference values passed in will also perform such checks.

  While this has been entirely tested by normal Perl CI testing, there may still be some corner-cases where these constraints are violated in otherwise-valid calls. These may require further investigation if they are found, and specific code to be adjusted to account for it.

• The op_dump() function has been expanded to include additional information about the recent "OP_METHSTART" and "OP_INITFIELD" ops, as well as for "OP_ARGCHECK" and "OP_ARGELEM" which had not been done previously.
• op_dump() now also has the facility to print extra debugging information about custom operators, if those operators register a helper function via the new "xop_dump" element of the "XOP" structure. For more information, see the relevant additions to perlguts.
• New API functions are introduced to convert strings encoded in UTF-8 to their ordinal code point equivalent. These are safe to use by default, and generally more convenient to use than the existing ones.

  ""utf8_to_uv"" in perlapi and ""utf8_to_uv_or_die"" in perlapi replace ""utf8_to_uvchr"" in perlapi (which is retained for backwards compatibility), but you should convert to use the new forms, as likely you aren't using the old one safely.

  To convert in the opposite direction, you can now use ""uv_to_utf8"" in perlapi. This is not a new function, but a new synonym for ""uvchr_to_utf8"" in perlapi. It is added so you don't have to learn two sets of names.

  There are also two new functions, ""strict_utf8_to_uv"" in perlapi and ""c9strict_utf8_to_uv"" in perlapi which do the same thing except when the input string represents a code point that Unicode doesn't accept as legal for interchange, using either the strict original definition ("strict_utf8_to_uv"), or the looser one given by Unicode Corrigendum #9 <https://www.unicode.org/versions/corrigendum9.html> ("c9strict_utf8_to_uv"). When the input string represents one of the restricted code points, these functions return the Unicode "REPLACEMENT CHARACTER" instead.

  Also ""extended_utf8_to_uv"" in perlapi is a synonym for "utf8_to_uv", for use when you want to emphasize that the entire range of Perl extended UTF-8 is acceptable.

  There are also replacement functions for the three more specialized conversion functions that you are unlikely to need to use. Again, the old forms are kept for backwards compatibility, but you should convert to use the new forms.

  ""utf8_to_uv_flags"" in perlapi replaces ""utf8n_to_uvchr"" in perlapi.

  ""utf8_to_uv_errors"" in perlapi replaces ""utf8n_to_uvchr_error"" in perlapi.

  ""utf8_to_uv_msgs"" in perlapi replaces ""utf8n_to_uvchr_msgs"" in perlapi.

  Also added are the inverse functions ""uv_to_utf8_flags"" in perlapi and ""uv_to_utf8_msgs"" in perlapi, which are synonyms for the existing functions, ""uvchr_to_utf8_flags"" in perlapi and ""uvchr_to_utf8_flags_msgs"" in perlapi respectively. These are provided only so you don't have to learn two sets of names.

• Three new API functions are introduced to convert strings encoded in UTF-8 to native bytes format (if possible). These are easier to use than the existing ones, and they avoid unnecessary memory allocations. The functions are ""utf8_to_bytes_overwrite"" in perlapi which is used when it is ok for the input string to be overwritten with the converted result; and ""utf8_to_bytes_new_pv"" in perlapi and ""utf8_to_bytes_temp_pv"" in perlapi when the original string must be preserved intact. "utf8_to_bytes_temp_pv" returns the result in a temporary using perlapi/"SAVEFREEPV" that will automatically be destroyed. With "utf8_to_bytes_new_pv", you are responsible for freeing the newly allocated memory that is returned if the conversion is successful.

  The latter two functions are designed to replace ""bytes_from_utf8"" in perlapi which creates memory unnecessarily, or unnecessarily large.

• New API functions valid_identifier_pve(), valid_identifier_pvn() and valid_identifier_sv() have been added, which test if a string would be considered by Perl to be a valid identifier name.
• When assigning from an SVt_IV into a SVt_NV (or vice versa), providing that both are "bodyless" types, Perl_sv_setsv_flags will now just change the destination type to match the source type. Previously, an SVt_IV would have been upgraded to a SVt_PVNV to store an NV, and an SVt_NV would have been upgraded to a SVt_PVIV to store an IV. This change prevents the need to allocate - and later free - the relevant body struct.
• Two new API functions are introduced to convert strings encoded in native bytes format to UTF-8. These return the string unchanged if its UTF-8 representation is the same as the original. Otherwise, new memory is allocated to contain the converted string. This is in contrast to the existing ""bytes_to_utf8"" in perlapi which always allocates new memory. The new functions are ""bytes_to_utf8_free_me"" in perlapi and ""bytes_to_utf8_temp_pv"" in perlapi. ""bytes_to_utf8_temp_pv"" in perlapi arranges for the new memory to automatically be freed. With "bytes_to_utf8_free_me", you are responsible for freeing any newly allocated memory.
• The way that subroutine signatures are parsed by the parser grammar has been changed.

  Previously, when parsing individual signature parameters, the parser would accumulate an "OP_ARGELEM" optree fragment for each parameter on the parser stack, collecting them in an "OP_LIST" sequence, before finally building the complete argument handling optree itself, in a large action block defined directly in perly.y.

  In the new approach, all the optree generation is handled by newly-defined functions in op.c which are called by the action blocks in the parser. These do not keep state on the parser stack, but instead in a dedicated memory structure referenced by the main "PL_parser" structure. This is intended to be largely opaque to other code, and accessed only via the new functions.

  This new arrangement is intended to allow more flexible code generation and additional features to be developed in the future.

• Three new API functions have been added to interact with the regexp global match position stored in an SV. These are sv_regex_global_pos_get(), sv_regex_global_pos_set() and sv_regex_global_pos_clear(). Using these API functions avoids XS modules needing to know about or interact directly with the way this position is currently stored, which involves the "PERL_MAGIC_regex_global" magic type.
• New "SvVSTRING" API macro

  A new API macro has been added, which is used to obtain the second string buffer out of a "vstring" SV, in a manner similar to the "SvPV" macro which obtains the regular string buffer out of a regular SV.

      STRLEN len;
      const char *vstr_pv = SvVSTRING(sv, vstr_len);
      

  See ""SvVSTRING"" in perlapi.

Selected Bug Fixes

• Fix null pointer dereference in S_SvREFCNT_dec [GH #16627 <https://github.com/Perl/perl5/issues/16627>].
• Fix feature 'class' Segmentation fault in DESTROY [GH #22278 <https://github.com/Perl/perl5/issues/22278>].
• "chdir" now returns real booleans (as its documentation describes), not integers. This means the result of a failed "chdir" now stringifies to '', not '0'.

  [GH #22365 <https://github.com/Perl/perl5/issues/22365>]

• Compound assignment operators return lvalues that can be further modified:

      ($x &= $y) += $z;
      # Equivalent to:
      #  $x &= $y;
      #  $x += $z;
      

  However, the separate numeric/string bitwise operators provided by the "bitwise" feature, "&= ^= |= &.= ^.= |.=", did not return such lvalues:

      use feature qw(bitwise);
      ($x &= $y) += $z;
      # Used to die:
      #  Can't modify numeric bitwise and (&) in addition (+) at ...
      

  This has been corrected. [GH #22412 <https://github.com/Perl/perl5/issues/22412>]

• Starting in v5.39.8, ""strftime"" in POSIX would crash or produce odd errors (such as "Out of memory in perl:util:safesysmalloc") when given a format string that wasn't actually a string, but a number, "undef", or an object (even one with overloaded string conversion).

  Now "strftime" stringifies its first argument, as before. [GH #22498 <https://github.com/Perl/perl5/issues/22498>]

  Also, fix POSIX::strftime() [GH #22369 <https://github.com/Perl/perl5/issues/22369>].

• "pack("p", ...)" and "pack("P", ...)" now SvPV_force() the supplied SV unless it is read only. This will remove CoW from the SV and prevents code that writes through the generated pointer from modifying the value of other SVs that happen the share the same CoWed string buffer.

  Note: this does not make "pack("p",... )" safe, if the SV is magical then any writes to the buffer will likely be discarded on the next read. [GH #22380 <https://github.com/Perl/perl5/issues/22380>]

• Enforce "no feature "bareword_filehandles"" for bareword file handles that have strictness removed because they are used in open() with a "dup" mode, such as in "open my $fh, ">&", THISHANDLE". [GH #22568 <https://github.com/Perl/perl5/issues/22568>]
• Using "goto" to tail call, or using the call_sv() and related APIs to call, any of trim(), refaddr(), reftype(), ceil(), floor() or stringify() in the "builtin::" package would crash or assert due to a "TARG" handling bug. [GH #22542 <https://github.com/Perl/perl5/issues/22542>]
• Fix sv_gets() to accept a "SSize_t" append offset instead of "I32". This prevents integer overflows when appending to a large "SV" for "readpipe" aka "qx//" and "readline". <https://www.perlmonks.org/?node_id=11161665>
• Fixed an issue where "utf8n_to_uvchr" failed to correctly identify certain invalid UTF-8 sequences as invalid. Specifically, sequences that start with continuation bytes or unassigned bytes could cause unexpected behavior or a panic. This fix ensures that such invalid sequences are now properly detected and handled. This correction also resolves related issues in modules that handle UTF-8 processing, such as "Encode.pm".
• The perl parser would erroneously parse some POD directives as if they were "=cut". Some other POD directives whose names start with cut, prematurely terminating an embedded POD section. The following cases were affected: cut followed by a digit (e.g. "=cut2studio"), cut followed by an underscore (e.g. "=cut_grass"), and in string "eval", any identifier starting with cut (e.g. "=cute"). [GH #22759 <https://github.com/Perl/perl5/issues/22759>]
• Builds with "-msse" and quadmath on 32-bit x86 systems would crash with a misaligned access early in the build. [GH #22577 <https://github.com/Perl/perl5/issues/22577>]
• On threaded builds on POSIX-like systems, if the perl signal handler receives a signal, we now resend the signal to the main perl thread. Previously this would crash. [GH #22487 <https://github.com/Perl/perl5/issues/22487>]
• Declaring a lexically scoped array or hash using "state" within a subroutine and then immediately returning no longer triggers a "Bizarre copy of HASH/ARRAY in subroutine exit" error. [GH #18630 <https://github.com/Perl/perl5/issues/18630>]
• builtin::trim() didn't properly clear "TARG" which could result in out of date cached numeric versions of the value being used on a second evaluation. Properly clear any cached values. [GH #22784 <https://github.com/Perl/perl5/issues/22784>]
• "shmread" in perlfunc and "shmwrite" in perlfunc are no longer limited to 31-bit values and can use all the available bits on a platform for their POS and SIZE arguments. [GH #22895 <https://github.com/Perl/perl5/issues/22895>]
• "shmread" in perlfunc is now better behaved if VAR is not a plain string. If VAR is a tied variable, it calls "STORE" once; previously, it would also call "FETCH", but without using the result. If VAR is a reference, the referenced entity has its refcount properly decremented when VAR is turned into a string; previously, it would leak memory. [GH #22898 <https://github.com/Perl/perl5/issues/22898>]
• The $SIG{__DIE__} and $SIG{__WARN__} handlers can no longer be invoked recursively, either deliberately or by accident, as described in "%SIG" in perlvar. That is, when an exception (or warning) triggers a call to a $SIG{__DIE__} (or $SIG{__WARN__}) handler, further exceptions (or warnings) are processed directly, ignoring %SIG until the original $SIG{__DIE__} (or $SIG{__WARN__}) handler call returns. [GH #14527 <https://github.com/Perl/perl5/issues/14527>], [GH #22984 <https://github.com/Perl/perl5/issues/22984>], [GH #22987 <https://github.com/Perl/perl5/issues/22987>]
• The ObjectFIELDS() for an object and "xhv_class_fields" for the object's stash weren't always NULL or not-NULL, confusing sv_dump() (and hence Devel::Peek's Dump()) into crashing on an object with no defined fields in some cases. [GH #22959 <https://github.com/Perl/perl5/issues/22959>]
• When comparing strings when using a UTF-8 locale, the behavior was previously undefined if either or both contained an above-Unicode code point, such as 0x110000. Now all such code points will collate the same as the highest Unicode code point, U+10FFFF. [GH #22989 <https://github.com/Perl/perl5/issues/22989>]
• In regexes, the contents of "\g{...}" backreferences are now properly validated. Previously, "\g{1 FOO}" was silently parsed as "\g{1}", ignoring everything after the first number. [GH #23050 <https://github.com/Perl/perl5/issues/23050>]
• A run-time pattern which contained a code block which recursed back to the same bit of code which ran that match, could cause a crash. [GH #22869 <https://github.com/Perl/perl5/issues/22869>]

  For example:

      my $r = qr/... (?{ foo() if ... }) .../;
      sub foo { $string =~ $r }
      foo()
      

• In some cases an "eval" would not add integer parts to the source lines saved by the debugger. [GH #23151 <https://github.com/Perl/perl5/issues/23151>]
• In debugging mode, perl saves source lines from all files (plus an indication of whether each line is breakable) for use by the debugger. The internal storage format has been optimized to use less memory. This should save 24 bytes per stored line for 64-bit systems, more for "-Duselongdouble" or "-Dusequadmath" builds. Discussed in [GH #23171 <https://github.com/Perl/perl5/issues/23171>].
• Ensure cloning the save stack for fork emulation doesn't duplicate freeing the RExC state. [GH #23022 <https://github.com/Perl/perl5/issues/23022>]
• Smartmatch against a code reference that uses a loop exit such as "last" would crash perl. [GH #16608 <https://github.com/Perl/perl5/issues/16608>]
• Class initializers and "ADJUST" blocks, per perlclass, that called "last" or other loop exits would crash perl. Same cause as for [GH #16608 <https://github.com/Perl/perl5/issues/16608>].
• Exceptions thrown and caught entirely within a "defer {}" or "finally {}" block no longer stop the outer run-loop.

  Code such as the following would stop running the contents of the "defer" block once the inner exception in the inner "try"/"catch" block was caught. This has now been fixed, and runs as expected. ([GH #23064 <https://github.com/Perl/perl5/issues/23064>]).

      defer {
          try { die "It breaks\n"; }
          catch ($e) { warn $e }
          say "This line would never run";
      }
      

• "readline" in perlfunc now clears the error flag if an error occurs when reading and that error is "EAGAIN" or "EWOULDBLOCK". This allows code that depended on "readline" to clear all errors to ignore these relatively harmless errors. [GH #22883 <https://github.com/Perl/perl5/issues/22883>]
• "open" automatically creates an anonymous temporary file when passed "undef" as a filename:

      open(my $fh, "+>", undef) or die ...
      

  This is supposed to work only when the undefined value is the one returned by the "undef" function.

  In perls before 5.41.3, this caused a problem due to the fact that the same undefined value can be generated by lookups of non-existent hash keys or array elements, which can lead to bugs in user-level code (reported as [GH #22385 <https://github.com/Perl/perl5/issues/22385>]).

  In 5.41.3, additional checks based on the syntax tree of the call site were added, which fixed this issue for some number of common cases, though not all of them, at the cost of breaking the ability of APIs that wrap "open" to expose its anonymous file mode. A notable example of such an API is autodie.

  This release reverts to the old problem in preference to the new one for the time being.

Obituaries

Acknowledgements

Perl 5.42.0 represents approximately 13 months of development since Perl 5.40.0 and contains approximately 280,000 lines of changes across 1,600 files from 65 authors.

Excluding auto-generated files, documentation and release tools, there were approximately 94,000 lines of changes to 860 .pm, .t, .c and .h files.

Perl continues to flourish into its fourth decade thanks to a vibrant community of users and developers. The following people are known to have contributed the improvements that became Perl 5.42.0:

Aaron Dill, Andrei Horodniceanu, Andrew Ruthven, Antanas Vaitkus, Aristotle Pagaltzis, Branislav Zahradník, brian d foy, Chad Granum, Chris 'BinGOs' Williams, Craig A. Berry, Dabrien 'Dabe' Murphy, Dagfinn Ilmari Mannsåker, Dan Book, Daniel Dragan, Dan Jacobson, David Cantrell, David Mitchell, E. Choroba, Ed J, Ed Sabol, Elvin Aslanov, Eric Herman, Erik Huelsmann, Gianni Ceccarelli, Graham Knop, hbmaclean, H.Merijn Brand, iabyn, James E Keenan, James Raspass, Johan Vromans, Karen Etheridge, Karl Williamson, Leon Timmermans, Lukas Mai, Marek Rouchal, Marin Tsanov, Mark Fowler, Masahiro Honma, Max Maischein, Paul Evans, Paul Johnson, Paul Marquess, Peter Eisentraut, Peter John Acklam, Philippe Bruhat (BooK), pyrrhlin, Reini Urban, Richard Leach, Robert Rothenberg, Robin Ragged, Russ Allbery, Scott Baker, Sergei Zhmylev, Sevan Janiyan, Sisyphus, Štěpán Němec, Steve Hay, TAKAI Kousuke, Thibault Duponchelle, Todd Rinaldo, Tony Cook, Unicode Consortium, Vladimír Marek, Yves Orton.

The list above is almost certainly incomplete as it is automatically generated from version control history. In particular, it does not include the names of the (very much appreciated) contributors who reported issues to the Perl bug tracker.

Many of the changes included in this version originated in the CPAN modules included in Perl's core. We're grateful to the entire CPAN community for helping Perl to flourish.

For a more complete list of all of Perl's historical contributors, please see the AUTHORS file in the Perl source distribution.

Reporting Bugs

If you find what you think is a bug, you might check the perl bug database at <https://github.com/Perl/perl5/issues>. There may also be information at <https://www.perl.org/>, the Perl Home Page.

If you believe you have an unreported bug, please open an issue at <https://github.com/Perl/perl5/issues>. Be sure to trim your bug down to a tiny but sufficient test case.

If the bug you are reporting has security implications which make it inappropriate to send to a public issue tracker, then see "SECURITY VULNERABILITY CONTACT INFORMATION" in perlsec for details of how to report the issue.

Give Thanks

If you wish to thank the Perl 5 Porters for the work we had done in Perl 5, you can do so by running the "perlthanks" program:

    perlthanks

This will send an email to the Perl 5 Porters list with your show of thanks.

SEE ALSO

The Changes file for an explanation of how to view exhaustive details on what changed.

The INSTALL file for how to build Perl.

The README file for general stuff.

The Artistic and Copying files for copyright information.