Featured

Happy new year!

This year’s advent calendar is over, and it leaves us lots of articles on metaprogramming, applications, useful Raku modules, how to migrate from Perl, programming techniques and even how to work with Raku inside containers.

No more articles until next year, and the call is already open: if you want to tell something about Raku and its surroundings, add your name to the list with possible title, and you’re in!

Meanwhile, you’ve got raku.org and docs.raku.org for all your Raku needs.

Raku means enjoyment. So we wish you all a lot of Raku for next year!

Day 24: The Grinch of Raku, Part 2: Hold Your Horses

In 2017, the Grinch ruined Christmas by showing off some of the naughty things you can do with Raku’s features. Unfortunately, while his heart grew by three sizes that year, there’s more than one Grinch! This Grinch will be doing something extra naughty this year, taking some inspiration from the JavaScript community.

You may have heard of JSFuck, which is a tool that allows you to write any JavaScript code using only the characters [, ], (, ), +, and !. This is something you’d only expect to be possible in a language like JavaScript, right? That’s not entirely true! To prove this, let’s port it to Raku. Since this can’t be implemented using the exact same set of characters, our restrictions will be that only non-alphanumeric ASCII characters may be used in the translated code, and string literals must not be used.

Generating Primitives

The first thing we’ll need to do is find a way to generate some primitives. The ones from JavaScript that are of interest to us are booleans, numbers, and strings; any other type of primitive can be represented through other means. These are generated mainly through type coercion on empty arrays, which also happens to be possible to do in Raku.

True and False can be generated in Raku using the ! prefix operator, similarly to how you can in JavaScript:

say ![];  # OUTPUT: True
say !![]: # OUTPUT: False

Using this in combination with the + prefix operator, we can generate any whole number, which is also the case in JavaScript:

say +[];         # OUTPUT: 0
say +![];        # OUTPUT: 1
say +![] + +![]; # OUTPUT: 2

In JavaScript, + also happens to be used to concatenate strings. When used with two empty arrays, + will coerce both to strings and concatenate them, which results in an empty string. + doesn’t behave like this in Raku, so we’ll need to use the ~ operator instead:

say (~[]).perl; # OUTPUT: ""

What about strings that aren’t empty though? In JavaScript, strings are iterable, which allows for certain characters to be used when stringifying values other than empty arrays. This isn’t the case in Raku! It’s time to start getting creative.

String bitwise operators allow you to perform the same bitwise operations you can perform on numbers on codepoints in strings. Using the ~^ infix operator, we can generate a null character given 0 and 0:

say ord +[] ~^ +[]; # OUTPUT: 0

We can’t generate the characters we need very easily with the ~+, ~|, and ~^ operators alone though. There is a way to do this using that null character, but we need a lowercase letter of some sort first. We can grab the letter "e" from "True" if we use a regex:

say ~(![] ~~ /...(.)/)[+[]]; # OUTPUT: e

Using an infinite sequence with these two characters, we can generate most of the characters in ASCII:

my Str:D @chars = (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
say @chars[65..90];  # OUTPUT: (A B C D E F G H I J K L M N O P Q R S T U V W X Y Z)
say @chars[97..122]; # OUTPUT: (a b c d e f g h i j k l m n o p q r s t u v w x y z)

Now that we can generate the characters in the string "&chr", we’ll be able to generate any Unicode string after the next step.

Evaluating Code

Most of the JavaScript code that can be generated depends on the Function constructor in order to work. Using it, you can arbitrarily generate a function at runtime. As far as I know, it’s not possible to generate code like this in Raku without using &EVAL. There’s a problem we need to solve if we are to use it, though.

We can use string literals with &EVAL just fine:

say EVAL "'Hello, world!'"; # OUTPUT: Hello, world!

But if we try to use a value that is unknown at compile-time with it, we’ll get an exception warning about the security implications of what we’re doing, telling us to use the MONKEY-SEE-NO-EVAL pragma:

say EVAL my $ = "'Hello, world!'"; # Throws X::SecurityPolicy::Eval

That’s not good in our case! We can’t set this pragma without alphanumeric characters. It’s time to get naughty. What happens if we try to use &EVAL using an indirect symbol lookup?

say ::('&EVAL')(my $ = "'Hello world!'"); # OUTPUT: Hello, world!

Perfect! Along with this, using indirect symbol lookup we can also call the &chr routine to generate a string for any Unicode codepoint. In combination, this allows us to translate any valid Raku code.

Hold Your Horses

We’re ready to start writing code for our port of JSFuck. This will simply be a script that takes some Raku code as input and outputs its translation. All of the subroutines used (apart from &MAIN) will be pure. Now, let’s give this port a bit of a nicer name than the obvious choice and call it Hold Your Horses instead.

Our first subroutine will be &from-uint, which will translate numbers. We could just add 1 to 0 repeatedly until we get the number we’re looking for, but this will generate huge amounts of code for larger codepoints. One way we can shorten the code this generates is if we represent numbers as being products of prime numbers. This can be further shortened by representing prime numbers greater than 5 as being a sum of products of prime numbers:

use Prime::Factor;

sub from-uint(UInt:D $x, Int:D $remainder = 0 --> Str:D) is pure {
    proto sub translate(UInt:D --> Str:D) is pure {*}
    multi sub translate(0 --> '+[]') { }
    multi sub translate(1 --> '+![]') { }
    multi sub translate(UInt:D $x --> Str:D) {
        join ' + ', '+![]' xx $x
    }

    if $x <= 5 {
        my Str:D $translation = $x.&translate;
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    } elsif $x.is-prime {
        from-uint $x - 1, $remainder + 1
    } else {
        my Str:D $translation = $x.&prime-factors».&from-uint.fmt: '(%s)', ' * ';
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    }
}

Now we can implement &from-str, which will parse code input by the user. This needs to map each codepoint in the given code to a Hold Your Horses number, which can be done by looking up a character in the sequence of characters from earlier if it is within its range, otherwise &chr can be called. Since we’re using this sequence every time we see a character that is included by it, this will be stored in $_ by our next subroutine. Since translating a single codepoint can be quite intensive, let’s use the experimental is cached trait with our helper subroutine that handles this to avoid having to do it more than once for any given codepoint:

use experimental :cached;

sub from-str(Str:D $code --> Str:D) is pure {
    my Int:D constant LIMIT = 'z'.ord.succ;

    proto sub translate(UInt:D --> Str:D) is pure is cached {*}
    multi sub translate(UInt:D $codepoint where 0..^LIMIT --> Str:D) {
        sprintf '.[%s]', $codepoint.&from-uint
    }
    multi sub translate(UInt:D $codepoint where LIMIT..* --> Str:D) {
        sprintf '::(%s)(%s)',
                '&chr'.ords».&translate.join(' ~ '),
                $codepoint.&from-uint
    }

    sprintf '::(%s)(%s)',
            '&EVAL'.ords».&translate.join(' ~ '),
            $code.ords».&translate.join(' ~ ')
}

Now we can implement &hold-your-horses, which will handle the full translation of code input by the user. All this needs to do is store the sequence from earlier in $_ before calling &from-str:

sub hold-your-horses(Str:D $code --> Str:D) is pure {
    Qc:to/TRANSLATION/.chomp
    $_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
    {$code.&from-str};
    TRANSLATION
}

With &MAIN added, our script is now complete:

use v6.d;
use experimental :cached;
use Prime::Factor;
unit sub MAIN(Str:D $code) {
    say hold-your-horses $code
}

sub from-uint(UInt:D $x, Int:D $remainder = 0 --> Str:D) is pure {
    proto sub translate(UInt:D --> Str:D) is pure {*}
    multi sub translate(0 --> '+[]') { }
    multi sub translate(1 --> '+![]') { }
    multi sub translate(UInt:D $x --> Str:D) {
        join ' + ', '+![]' xx $x
    }

    if $x <= 5 {
        my Str:D $translation = $x.&translate;
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    } elsif $x.is-prime {
        from-uint $x - 1, $remainder + 1
    } else {
        my Str:D $translation = $x.&prime-factors».&from-uint.fmt: '(%s)', ' * ';
        $translation ~= ' + ' ~ $remainder.&from-uint if $remainder;
        $translation
    }
}

sub from-str(Str:D $code --> Str:D) is pure {
    my Int:D constant LIMIT = 'z'.ord.succ;

    proto sub translate(UInt:D --> Str:D) is pure is cached {*}
    multi sub translate(UInt:D $codepoint where 0..^LIMIT --> Str:D) {
        sprintf '.[%s]', $codepoint.&from-uint
    }
    multi sub translate(UInt:D $codepoint where LIMIT..* --> Str:D) {
        sprintf '::(%s)(%s)',
                '&chr'.ords».&translate.join(' ~ '),
                $codepoint.&from-uint
    }

    sprintf '::(%s)(%s)',
            '&EVAL'.ords».&translate.join(' ~ '),
            $code.ords».&translate.join(' ~ ')
}

sub hold-your-horses(Str:D $code --> Str:D) is pure {
    Qc:to/TRANSLATION/.chomp
    $_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
    {$code.&from-str};
    TRANSLATION
}

Now, does this actually work? For brevity’s sake, let’s say it works as intended if say "Hello, world! 👋" can be translated and run:

bastille% raku hold-your-horses.raku 'say "Hello, world! 👋"' > hello-world.raku
bastille% raku hello-world.raku
Hello, world! 👋

Perfect! This is the script’s output:

$_ := (+[] ~^ +[]...~(![] ~~ /...(.)/)[+[]]...*);
::(.[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])])(.[(+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![]] ~ .[((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ .[((+![] + +![]) * (+![] + +![] + +![]) + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![])] ~ .[(+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![])] ~ ::(.[(+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![]) * (+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) + +![])] ~ .[(+![] + +![]) * (+![] + +![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])])((+![] + +![] + +![] + +![] + +![]) * (+![] + +![] + +![] + +![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * ((+![] + +![]) * (+![] + +![] + +![] + +![] + +![]) + +![]) + +![]) + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) * (+![] + +![] + +![]) + +![])) ~ .[(+![] + +![]) * ((+![] + +![]) * (+![] + +![]) * (+![] + +![]) * (+![] + +![]) + +![])]);

Wrapping Up

Raku is quite a large language with an extensive set of features. These can be combined in some very interesting ways! Here, using a combination of type coercion, string bitwise operators, regexen, sequences, indirect symbol lookup, and a loophole with &EVAL, we were able to be naughty Grinches again this year and port JSFuck from JavaScript. If you’re tempted to say something is impossible to write in Raku, hold your horses; it may very well be possible to do with the right tools.

Day 23 – A Raku Advent Helper

Introduction

I have been writing Raku Advent articles annually since 2016, and it’s always been a struggle for me to get a reliable transformation of my source file into the Raku Advent WordPress (WP) website without something getting changed by WP. Then, the menus are terrible and editing can be troublesome. In this article I hope to show how the situation can be improved.

Background

The great name change to Raku this year unfortunately happened late in the year and there was not a lot of time to get a new Raku Advent website ready. Consequently, theme selection and tweaking, confusion over the actual Raku Advent website link, and unfortunate article cancellations were wrinkles in the normally smoother process. However, we plan to improve the website before the 2020 Advent season, and also get commitments earlier with concrete drafts available sooner. In the meantime, in this hastily prepared stand-in article, I will go into a bit of detail on some help we hope to offer.

Article creation

Since my first experience with WordPress, I have found these things that make it awkward for me to use WP:

  • Small editing window
  • Noticeable lag time while editing increases fumble-finger errors
  • Enter the desired schedule time in the website time zone (TZ) but see it displayed in your local TZ (with little or no hint as to what you are seeing) [See Note 1]
  • Confusing editing contexts and widget placements

I’m sure most of my problems with WP are self-induced, but I do prefer a more TeX-like document production work flow.

Prior years

In past years I’ve created the articles in Gihub-flavored markdown, manually (with the assistance of my Emacs editor) converted each paragraph to single, long lines, and then posted it in a Github gist. After that, I used the tool p6advent-md2html.p6, developed by @zoffix [Note 2] and modified by @SimonProctor, to extract the html from Github’s representation of the markdown which results in a nice highlighting of code blocks. Finally, that html is copied and pasted into WP and a publishing schedule set up. That original process is outlined here:

  1. Write the post in Github-favored Markdown text
  2. Collapse each paragraph to one long line
  3. Paste the source into a Github gist
  4. Use the existing Advent tool to extract the resulting html representation to one’s local computer
  5. Copy the html and paste it into the blank, html view of the selected WP editor
  6. View the finished product and check for errors

If errors are found:

  1. Correct the errors in the WP editor

OR

  1. Correct the errors in the source
  2. Repeat steps 2 through 6 again

That process is not so bad the first time through it, but when, inevitably, errors are found, one has the choice of either manually editing it on WP or modifying the source and going through the entire process again! Neither choice is very good.

2019’s Advent goal: reduce WP pain

I decided this year to help my article-creation situation so I created a Raku tool to eliminate some of the problems. It’s available to the public as of today:

$ zef install RakuAdvent::WordPress

That module provides the tool make-wp-input. So my new steps as of this year:

  1. Write the post in raw html
  2. Run my new Advent tool (make-wp-input) to format the source into WP-acceptable html
  3. Copy the html and paste it into the blank, html view of the selected WP editor
  4. View the finished product and check and correct for errors

If errors are found:

  1. Correct the errors in the WP editor

OR, preferably,

  1. Correct the errors in the source
  2. Repeat steps 2 through 4 again

Thus, in my new process, I’ve eliminated a couple of steps, but I still have to copy/paste my clean WP source into the WordPress editor—but that’s because I have not taken advantage of the available APIs from WordPress and Github to do the drudge work.

However, in spite of other limitations, the new tool has been a huge help in easing the use of live code examples in an article. In my sandbox where I write my article, I create the code samples in their own files and then add the

<!-- insert file-name lang -->

lines as needed in the location needed. That way, I can edit the live code and test it to make sure it works, but don’t have to change the source using that code.

Tips for Raku authors

Here are some ideas I’ve found helpful while developing articles for the Raku Advent:

  • See help for the WP scheduling calendar in this video
  • Take advantage of your default personal WP website to experiment
  • View the finished product as printed PDF, and check and correct for errors (this has been a very good way for me to look over my article at leisure while sipping an egg nog with my BF “by the fire.” 😊; see Ref. 1 for an excellent html-to-pdf converter)

Wish lists

Here are some things I hope to do with make-wp-input in the New Year:

  1. Convert html source to Github-flavored markdown
  2. Handle html tables
  3. Allow paragraphs in the source html to be recognized by either blank lines above and below the text or a line with a closing tag on the line before the text or an opening tag on the line following the text
  4. Use Github’s APIs [Ref. 2] to manipulate markdown source to a Github gist and get html results back from it
  5. Use WordPress’s APIs [Ref. 3] to manipulate one’s article on WordPress (including setting or updating the publication schedule)

And here are some things I hope the community can do (or at least agree upon) for the Raku Advent website:

  1. Improve the theme and code styling.
  2. Use the old Perl 6 Advent theme?
  3. Sign up for article slots earlier in the year, and start the article (at least in skeleton form) as a scheduled one on the Raku Advent website.

Summary

This year has seen a lot of changes in the Raku community, especially with the name change, and not all are done yet. One area that still needs work is improving the new Raku Advent website. We also hope to make it easier to create and post Raku Advent articles as well as get more participation. Note the 2020 schedule is open now, so you can get your slot early and avoid last minute shopping, er, Raku Adventing!

I ❤️ Raku! 😊

Merry Christmas and a Happy, Blessed New Year to all!


APPENDIX


Notes

  1. I have filed an issue with WordPress to help with time zone identification in the scheduling calendar.
  2. Names preceded by @ are IRC or Github aliases.

References

  1. wkhtmltopdf (available as a Debian package)
  2. Github API
  3. WordPress API

Raku modules used

  • RakuAdvent::WordPress (v.0.0.2)

Day 22: Off Course

You may not have heard about my Perl 6 Courses, and I don’t blame you.

It has been quite a journey.

It started in September 2018 with my Perl6 In 45+45 Minutes introduction to Perl 6 at the Nordic Perl Workshop in Oslo. The very first time a held a presentation at a conference…

I got positive feedback, and wondered if I could build on it. The idea of a full blown course matured, and I started working on the accompanying textbook first.

The book and course is meant as an introduction to Raku, for people already familiar with programming.

I pitched the course to PerlCon 2019 in Riga, and they accepted it. The organiser asked me to promote it, and the result was my Perl 6 blog Perl 6 Musings (at the absolutely fantastic address «perl6.eu»).

Unfortunately that didn’t work out, and the course was cancelled due to too few participants.

  Beginning Raku, 1. Edition (December 2019)  

Pages: 370

File size: ~ 11 Mbyte (pdf)


arnesom.github.io/Beginning-v1.00.pdf

I am giving away this first version of the book for free. I do reserve the right to print the book and sell it. You are free to distribute the pdf file or print it. You are also free to distribute printed copies, but you may not get paid for it.

Feel free to use the code samples, either as they are or as inspiration for your own work. Atribution would be nice, but isn’t required.

I will be grateful for feedback, and do so at the Github page for the book – or by email to the address shown in the book. I intend to publish a revised version of the book if I receive feedback that warrants an update.

Not Complete

The next course, «Advanced Raku» continues where this one ends. As the book is meant as a reference, I have chosen to make a combined book for both courses, called «Raku Explained». The second half (the «Advanced Raku» part) is unfinished, but I have published a preliminary Table of Contents and Index so that you can see what the whole book intends to cover.


  Raku Explained, v0.01 (December 2019)  

Pages: 30 (Table of Contents & Index only)

File size: ~ 5 Mbyte (pdf)


arnesom.github.io/Explained-v0.01.pdf

I am also interested in feedback on the topics in the second part (chapter 18 – 32).

Day 21: Searching for a Red gift

Alabaster Snowball, the elf, was searching for a gift for the person he had drawn on the North Pole’s Secret Santa. He had the great honour to draw Santa! What to give for the one who gives everyone’s presents? So he was searching on the internet for some keywords he knew Santa would like:

  • automatic letter reader
  • resistant boots
  • red sleigh accessories
  • red cap that does not fly off in wind
  • red jacket
  • red

Wait a minute! Is Red going to :api<2>?!! Alabaster Snowball has already read about that ORM for Raku. But it seems this new :api<2> version is taking it to the next level.

That’s it! I’ll give Santa a Red:api<2> PoC as gift! I know he has been playing with Raku, and I think it would be great to change all that collection of SQL strings on the NiceList model to a well made set of ORM classes.

Reading the documentation, Snowball learned that it would be very easy to create it’s first model:

use Red:api<2>;

unit model Child;

has UInt $!id              is id;
has Str  $.name            is column;
has Str  $.country         is column;

He started using Red:api<2> and creating a new model that represents a table child with 3 columns (id, name and country). As easy as that.

Alabaster could now just connect into a database, create the table, and start inserting children:

use Red:api<2>;
red-defaults default => database "SQLite";

Child.^create-table: :unless-exists;

Child.^create: :name<Fernanda>, :country<England> ;
Child.^create: :name<Sophia>,   :country<England> ;
Child.^create: :name<Dudu>,     :country<Scotland>;
Child.^create: :name<Rafinha>,  :country<Scotland>;
Child.^create: :name<Maricota>, :country<Brazil>  ;
Child.^create: :name<Lulu>,     :country<Brazil>  ;

And to list all children created:

.say for Child.^all.sort: *.name;

And that would run this query:

SELECT
   child.id, child.name, child.country 
FROM
   child
ORDER BY
   child.name

And prints:

Child.new(name => "Dudu", country => "Scotland")
Child.new(name => "Fernanda", country => "England")
Child.new(name => "Lulu", country => "Brazil")
Child.new(name => "Maricota", country => "Brazil")
Child.new(name => "Rafinha", country => "Scotland")
Child.new(name => "Sophia", country => "England")

If it’s needed, Santa can classify children by country:

my %by-country := Child.^all.classify: *.country;

And to discover what countries have children registered:

say %by-country.keys;

That would run:

SELECT
   DISTINCT(child.country) as "data_1"
FROM
   child

And that would return:

(England Scotland Brazil)

If he needs to get all children from England:

.say for %by-country<England>;

That would run:

SELECT
   child.id, child.name, child.country 
FROM
   child
WHERE
   child.country = ?

-- BIND: ["England"]

That would return:

Child.new(name => "Fernanda", country => "England")
Child.new(name => "Sophia", country => "England")

It’s working great! How about storing the gifts? Is there a way to store what a child asked by year?

# Gift.pm6
use Red:api<2>;

unit model Gift;

has UInt $!id            is serial;
has Str  $.name          is column{ :unique };

has      @.asked-by-year is relationship( *.gift-id, :model<ChildAskedOnYear> );

method child-asked-on-year(UInt $year = Date.today.year) {
    @!asked-by-year.grep(*.year == $year)
}

method asked-by(UInt $year) {
    self.child-asked-on-year(|($_ with $year)).map: *.child
} 
# Child.pm6
use Red:api<2>;

unit model Child;

has UInt $!id              is id;
has Str  $.name            is column;
has Str  $.country         is column;

has      @.asked-by-year   is relationship( *.child-id, :model<ChildAskedOnYear> );

method asked(UInt $year = Date.today.year) {
    @!asked-by-year.grep: *.year == $year
}
# ChildAskedOnYear.pm6
use Red:api<2>;

unit model ChildAskedOnYear;

has UInt $!id       is serial;
has UInt $.year     is column = Date.today.year;
has UInt $!child-id is referencing(*.id, :model<Child>);
has UInt $!gift-id  is referencing(*.id, :model<Gift>);

has      $.child    is relationship( *.child-id, :model<Child> );
has      $.gift     is relationship( *.gift-id,  :model<Gift>  );

Alabaster Snowball thought that way he could get all information he would need. Creating new gifts is easy!

for <doll ball car pokemon> -> $name {
    Gift.^create: :$name;
}

How about searching? Alabaster Snowball writes a new line:

.say for Gift.^all

And it returns all the gifts. But what if we want only the gifts that end with “ll”?

.say for Gift.^all.grep: *.name.ends-with: "ll"

That will run a query like:

SELECT
   gift.id, gift.name 
FROM
   gift
WHERE
   gift.name like '%ll'

Snowball wondered if it is possible to find what a child has asked:

.say for Child.^find(:name<Fernanda>).asked.map: *.gift

That runs:

SELECT
   child_asked_on_year_gift.id, child_asked_on_year_gift.name 
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
WHERE
   child_asked_on_year.child_id = ? AND child_asked_on_year.year = 2019

And what if we want to know the last year’s gift?

.say for Child.^find(:name<Fernanda>).asked(2018).map: *.gift
SELECT
   child_asked_on_year_gift.id, child_asked_on_year_gift.name 
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
WHERE
   child_asked_on_year.child_id = ? AND child_asked_on_year.year = '2018'

How do we know how many of each gift should be built?

say ChildAskedOnYear.^all.map(*.gift.name).Bag
SELECT
   child_asked_on_year_gift.name as "data_1", COUNT('*') as "data_2"
FROM
   child_asked_on_year
    LEFT JOIN gift as child_asked_on_year_gift ON child_asked_on_year.gift_id = child_asked_on_year_gift.id
GROUP BY
   child_asked_on_year_gift.name

The documentation for Red is on https://fco.github.io/Red/ and some examples used here can be found on https://github.com/FCO/Red/blob/join/examples/xmas/index.p6

Day 20 – Raku from Perl: Transforming Old Perl Code (Part 2)

Spoiler Alert!

When I started this two-part post, I was blissfully unaware of a similar topic in the Raku community; however, I was awakened to that fact when I first read Elizabeth Mattijsen’s Raku Weekly Blog on Monday, 9 Dec, and saw that the famous Perl and Raku expert, Jeff Goff, had written a multi-part series on porting a very complex Perl module, to Raku. The only saving grace for me is that the posts are highly technical, and its audience is for serious Raku hackers who want to produce native Raku code for the most complicated Perl modules in existence: a highly recommended read for the accomplished programmer who relishes reverse-engineering and all its tribulations! So, with a tip of my hat to Jeff, I will continue to slog along here in my old Perl garden which is so overgrown with newby, self-taught Perl.

One more side note: the author of the module Jeff is porting, John McNamara, is my favorite Perl module author, whose wonderful modules enabling Perl users to slice and dice Microsoft Excel were life-savers for me in my last job. I have had several e-mail discussions with John over the years, and he is a kind and very talented man who has contributed so much for Perl user. Many thanks, John!

Introduction

In this post we will continue to port old Perl code to Raku. In order to follow this Part 2 you should have read the Part 1 post. Make sure you clone the exercise code from Github so you can follow along since much of the actual porting problems and solutions are shown as git commits in each stage-N branch.

The next step in our porting adventure is to start transforming Perl modules to Raku. During the transition from Perl to Raku we will strive to duplicate (port) every Perl module to Raku.

In our previous article we continued to ensure the transformed program ran (without arguments) while we moved all its Perl subroutines to a Perl module. Now, while we port Perl modules, we will also check actual operation of the driver program. We expect to find many more challenges as we continue working through my old code, so let’s dig in!

Stage-2: where we left off

At the end of Part 1 I said there was one more thing I would do to the the driver program (manage-web-site.raku): replace the if/else blocks with when blocks, so I’ll do that now. Please go the exercise repository directory for 2019 and ensure you are in the branch stage-2 with no changes uncommitted.

In order to use the when blocks we must use the implicit topic variable ($_) instead of the $arg variable; however, at the moment we still need it so we remove it from the loop variable and temporarily declare it at the top of the block (see Note 1): my $arg = $_. Now let’s execute the program again, this time with the -h (help) option:

$ ./manage-web-site.raku -h
Use of uninitialized value of type Any in numeric context
  in block  at ./manage-web-site.raku line 184
Use of uninitialized value of type Any in numeric context
  in block  at ./manage-web-site.raku line 185
Use of uninitialized value of type Any in numeric context
  in block  at ./manage-web-site.raku line 186
No such method 'Int' for invocant of type 'Any'
  in block  at ./manage-web-site.raku line 186

Here are the offending lines:

# lines 181-187:
my $arg = $_;                    # <= new decl of $arg, set to topic variable's value
my $val;
my $idx = index $arg, '=';       # = 0) {                 # <= this is where the problem first surfaces
    $val = substr $arg, $idx+1;  # <= and then here
    $arg = substr $arg, 0, $idx; # <= and here
}

What has happened is another syntax change between Perl and Raku has surfaced: the index routine in Perl returns a ‘-1’ if the needle is not found, but in Raku it returns undefined so our existing Perl test for an integer throws an error. The solution is to test the return value for definedness. As in Perl, We can’t just test for zero with an if because that is a valid value but it isn’t truthy. So, we change our code to this:

# lines 184-187:
if $idx.defined {                # <= this is the only change needed
    $val = substr $arg, $idx+1;
    $arg = substr $arg, 0, $idx;
}

After a few more cleanups we’re ready to start porting a module. Ensure your git repo is on branch stage-2> and clean with no uncommitted changes. Then execute:

$ git checkout -b stage-3

Porting a Perl module

First, let’s look at a few of the common issues we will meet:

  1. Converting Perl’s export methods to Raku’s much simpler syntax
  2. Converting Perl-style calls to Raku’s function signatures
  3. Converting Perl’s foreach loops to Raku’s for @arr -> { loops
  4. Converting Perl’s for (my $i...) {...} loops to Raku’s loop (...) {...}

There will definitely be more issues, and I’m sure the changes I make may not be the ones a person more knowledgeable of modern Perl may make but remember, a lot of my old Perl predates modern Perl, and I’m not always taking the time to improve the code but just getting a first cut at a working solution.

Before we look at the old Perl we will look at working examples of handling export and signatures. The following Perl module (P5.pm) is using Damian Conway’s Perl6::Export::Attrs module for simpler and Raku-like export handling. Note the three types of argument passing.

package P5;

use feature 'say';
use strict;
use warnings;

# This module does NOT affect exporting to Raku, it only affects
# exporting to Perl programs. See program `usep5.pl` for examples.
use Perl6::Export::Attrs; # [from CPAN] by Damian Conway

our $VERSION = '1.00';

# Always exported:
# sub passing all args via @_ array:
sub sayAB :Export(:MANDATORY) {
    my $a = shift;
    my $b = shift;
    $b = 0 if !defined $b;
    say "\$a = '$a'; \$b = '$b'";
}

# Export sayABC when explicitly requested or when the ':ALL' export is set
# sub passing args via a hash:
sub sayABC :Export(:sayABC) {
    my $href = shift;
    my $a = $href->{a};
    my $b = $href->{b};
    my $c = $href->{c};
    $a = 0 if !defined $a;
    $b = 0 if !defined $b;
    $c = 0 if !defined $c;
    say "\$a = '$a'; \$b = '$b'; \$c = '$c'";
}

# Always exported:
# sub pass rw args via a ref
sub changeC :Export(:MANDATORY) {
    my $cref = shift;
    my $newc = shift;
    $${cref} = $newc;
}

1; # mandatory true value 

The following Perl program (usep5.pl) shows how to use the three types of argument passing. It also shows how the export-restricted sub (sayABC) is used.

#!/usr/bin/env perl
use feature 'say';
use strict;
use warnings;

use lib qw(.);
use P5 qw(:sayABC);

sayAB(1);             # using the @_ for passing arguments
sayABC({a=>1, b=>2}); # using a hash for passing arguments

my $c = 1;
say "\$c = $c";
changeC(\$c, 2); # using a reference to pass a read/write variable
say "\$c = $c";

We exercise the Perl script:

$ ./usep5.pl
$a = '1'; $b = '0'
$a = '1'; $b = '2'; $c = '0'
$c = 1
$c = 2

We then see the Raku version of the Perl script and module demonstrating the changes necessary to get the same output result (but see Note 2).

unit module P6;

#| Always exported:
#| ported from a Perl sub passing all args via @_ array:
sub sayAB($a, $b = 0) is export {
    say "\$a = '$a'; \$b = '$b'";
}

#| Export &sayABC when explicitly requested or when the ':ALL' export is set
#| ported from a Perl sub passing args via a hash:
sub sayABC(:$a, :$b, :$c = 0) is export(:sayABC) {
    say "\$a = '$a'; \$b = '$b'; \$c = '$c'";
}

#| Always exported:
#| ported from a Perl sub passing a read/write arg:
sub changeC($c is rw, $newc) is export {
    $c = $newc;
}
#!/usr/bin/env perl6

use lib ;
use P6 :ALL; #= <= ensures all exportable object are exported

sayAB 1;
sayABC :b, :a;

my $c = 1;
say "\$c = $c";
changeC $c, 2;
say "\$c = $c";

Executing the Raku script should show the same results:

$ ./usep6.raku
$a = '1'; $b = '0'
$a = '1'; $b = '2'; $c = '0'
$c = 1
$c = 2

Voila! I hope you can see the Raku module’s version is cleaner and simpler looking as compared to that of Perl.

To lead us to the first module we want to port we will execute the main option in the driver program:

$ ./manage-web-site.raku -gen
Collecting names by CS...
Unable to open file 'usafa-template1-letter.ps': No such file or directory
  in method call-args at /usr/local/rakudo.d/share/perl6/site/sources/ACCE801FB16076DAD1F96BE316DBFEDD148902C8 (Inline::Perl5) line 430
  in sub  at /usr/local/rakudo.d/share/perl6/site/sources/ACCE801FB16076DAD1F96BE316DBFEDD148902C8 (Inline::Perl5) line 935
  in block  at ./manage-web-site.raku line 449

If we look at the line (449) that apparently caused the failure we see the following:

build_montage(%CL::mates, $genS); #=  <= in module ./PicFuncs.pm5

and we see that module PicFuncs.pm5 is the entry point. So we will select that module to start with and copy it to PicFuncs.pm6. We open the new file in our favorite editor and see some changes we can make immdediately (starting at the first line):

  • change package to unit module
  • remove the next file lines of code
  • change all sub lines reading :Export(:DEFAULT) to is export
  • move sub build_montage to the top of the module to ease porting one sub at a time
  • remove Perl code for true return at the bottom of the file

Now let’s take a look at some other things we notice:

  • the module is using the G.pm which is in Perl format, so we will have to either convert that next or change the way we use it inside the PicFuncs.pm6 module. I choose to convert the G.pm module to G.pm6
  • to ease the port we want to end the PicFuncs.pm6 module after the first sub by use of the =finish pod feature
  • we have to change the way we use the PicFuncs module inside the manage-web-site.raku file

You can probably guess that soon we will be in a situation where we can’t get much farther without converting the rest of the Perl modules to Raku. But on to tending to the list above…

To get a check on our progress we execute the driver program again:

$ ./manage-web-site.raku -gen
===SORRY!=== Error while compiling /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm (PicFuncs)
This appears to be Perl 5 code. If you intended it to be Perl 6 code, please use a Perl 6 style declaration like "unit package Foo;" or "unit module Foo;", or use the block form instead of the semicolon form.
at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm (PicFuncs):1
------> package PicFuncs;⏏

Whoa, looks like we need to try to separate the two types of modules since the names confuse Raku. I’ll first try rearranging some code in the driver….

The ‘use’ statement and module searches

Note there are some differences in the way Perl and Raku handle the usestatement. In Perl, modules are searched for modules in paths in the following order: paths defined in the use lib statement, paths defined in the environment variables PERL5LIB and PERLLIB (in that order), and, finally, in paths defined in the @INC array. In any list of paths, individual paths are separated by the colon character (‘:’) which is consistent with the OS path separator.

Raku is very different. First, we cannot use the use statement in a module because it will be precompiled before use and the use statement cannot be precompiled. Second, Raku uses paths searched for in the following order: paths defined in the use lib statement, paths defined in the environment variable PERL6LIB, and paths defined during Raku installation. In any list of paths, individual paths are separated by the comma character (‘,’) which is consistent with lists in Raku. Here are the search paths on my computer for this test environment (PERL6LIB=foo,bar):

$ perl6 -e 'use lib ; use MyModule'
===SORRY!===
Could not find MyModule at line 1 in:
    file#/home/tbrowde/raku-advent/raku-advent-2019
    file#/home/tbrowde/raku-advent/raku-advent-2019/foo
    file#/home/tbrowde/raku-advent/raku-advent-2019/bar
    inst#/home/tbrowde/.perl6
    inst#/usr/local/rakudo.d/share/perl6/site
    inst#/usr/local/rakudo.d/share/perl6/vendor
    inst#/usr/local/rakudo.d/share/perl6
    ap#
    nqp#
    perl5#

Resume porting…

Now try again:

$ ./manage-web-site.raku -gen
===SORRY!===
Unsupported use of 'foreach'; in Perl 6 please use 'for'
at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):48
------>     foreach⏏ my $cs (@cs) {
Other potential difficulties:
    To pass an array, hash or sub to a function in Perl 6, just pass it as is.
    For other uses of Perl 5's ref operator consider binding with ::= instead.
    Parenthesize as \(...) if you intended a capture of a single variable.
    at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):23
    ------>     U65::get_keys_by_sqdn(\⏏%sqdn, $mref);

We see now the first glimpse of some of the loop differences between Perl and Raku. I’ll start with changing the loops. One warning about Perl’s for (my $i = 0; $i < $max; ++$i) {...}. The direct translation into Raku is this: loop (my $i = 0; $i < $max; ++$i) {...}, but therein lies a surprise. The loop index variable, which in the scope of the loop in Perl, is in the outer scope of the loop’s parentheses in Raku! So I got in the habit of rewriting a Perl loop to emphasize the proper scope of the index variable and a better pointer to possible problems of duplicate declarations:

my $i;
loop (my $i = 0; $i < $max; ++$i) {...}

Now we start again with execution testing…oops, another syntax problem:

$ ./manage-web-site.raku -gen
===SORRY!===
Unsupported use of @{$sqdn{$cs}; in Perl 6 please use @($sqdn{$cs) for hard ref or @::($sqdn{$cs) for symbolic ref
at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):55
------>         my @n = @{$sqdn{$cs}⏏};
Other potential difficulties:
    To pass an array, hash or sub to a function in Perl 6, just pass it as is.
    For other uses of Perl 5's ref operator consider binding with ::= instead.
    Parenthesize as \(...) if you intended a capture of a single variable.
    at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):23
    ------>     U65::get_keys_by_sqdn(\⏏%sqdn, $mref);

A huge dfference between Perl arrays and hashes and those of Raku. I’ll tend to those, too…another problem:

$ ./manage-web-site.raku -gen
===SORRY!=== Error while compiling /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs)
Unsupported use of ->(), ->{} or ->[] as postfix dereferencer; in Perl 6 please use .(), .[] or .{} to deref, or whitespace to delimit a pointy block
at /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs):137
------> 	        my $fname = $mref->{⏏$c}{file};

This problem is from the differences in signature and passing arguments to subs from the caller as discussed earlier in the simple examples. I will work through the signature to see if we can help the situation. I’m first going to look at the calling line in the driver we looked at before:

build_montage(%CL::mates, $genS);

We see two arguments: a hash and an apparent scalar. We changed those in Part 1 to Raku from their original Perl syntax using a reference. So we are fine on the calling side at the moment. Back to the called sub where the first few lines look like this (with comments stripped out):


sub build_montage is export {
    my $mref  = shift @_; # \%CL::mates
    my $cs    = shift @_;

We’re going to change that to:

sub build_montage(%mates, $cs) is export {

and proceed with other required changes…After quite a few changes we start running into missing routines:

$ ./manage-web-site.raku -gen
===SORRY!=== Error while compiling /home/tbrowde/mydata/tbrowde-home/perl6/raku-advent/raku-advent-2019/raku-advent-extras/2019/PicFuncs.pm6 (PicFuncs)
Undeclared routines:
    convert_single_pic_to_eps used at line 159
    insert_logo used at lines 287, 297
    insert_pictures used at line 279

This looks like a good stopping place. As I said earlier, we are probably at the point where we will have to convert almost the whole project, or at least those modules which are used by the driving script as well as all the other modules they use. I checked out a stage-4 branch and did some more housekeeping and tidying, but I leave the continuation of porting for another day.

Summary

In these two articles you have seen one way to ease porting Perl code to Raku, and I hope they may help those who are considering moving to Raku see that it can be accomplished iteratively in smaller steps instead of taking great chunks of time. The results, for me, include code that is:

  • more visually appealing (cleaner and less cluttered)
  • easier to maintain
  • easier to see where to improve it

If you are interested in seeing the rest of the conversion, execute .ask tbrowder on IRC channel ‘#raku’ and I’ll keep on plugging on the repo we’ve been using (and it will help me, too!). Happy Rakuing

I ❤️ ❤️ Raku! 😊

🎅 Merry Christmas 🎅 and 🥂 Happy New Year 🎉 to all and may ✝ “God bless Us, Every One!” ✝ [Ref. 1]


APPENDIX


Notes

  1. I wasn’t sure if changing the value of the topic variable in the loop would allow the when blocks to continue to work as expected. Fortunately, it does.
  2. There are some subtle (to me) issues when using various module export options in Raku. The current discussion in the docs here (see the ordered list of five Notes) do not describe exhaustively all possible combinations of export options, and as a result I (who was the person who wrote those notes) saw an error message when I used a new combination of conditions that, in my opinion, was misleading. Consequently I first suspected a Rakudo bug but now suspect it is an LTA (Less Than Awesome) error message. Keep an eye on Rakudo issue #3341 for a resolution.

References

  1. A Christmas Carol, a short story by Charles Dickens (1812-1870), a well-known and popular Victorian author whose many works include The Pickwick Papers, Oliver Twist, David Copperfield, Bleak House, Great Expectations, and A Tale of Two Cities.

Day 19 – Functional Programming with Raku

Functional Programming in Raku

I watched a really nice video recently, Functional Programming in 40 minutes, which I thought was really good. I’ve done bits of functional programming over the years but I know lots of people find the ideas behind it quite confusing.

So the first thing I’d say is go watch that video, it really helps.

I’ll wait.

Back? OK, cool, so I’m sure you’ve got the core concept here functional programming can be broken down into 3 main things.

  • Pure Functions
  • Immutable Data Structures
  • Bridging Systems

So how easy is it to bring these ideas into Raku code? Would it surprise you to know “really really” would be the answer to that question? I didn’t think so, lets go over each section in turn.

Pure Functions

So a pure function causes no side effects. And for any given input it always gives the same output. Raku includes a is pure trait that you can use to flag a code block to say that it will return the same value and the compiler can use this as a hint to replace calls with constants. Generally if you flag a function with is pure you don’t want to cause side effect either (as your code may only be called once).

Not only that Raku has a number of ways to generate code blocks from named subs sub name($n) {"Hello $n"} through anonymous subs sub ($n) {"Hello $n"} on to pointy code blocks -> $n {"Hello $n"} and finally just plain blocks {"Hello $_"}; all of these can be called or assigned to variables and used in other subs.

Raku also allows you to use some more complex function creation techniques function composition and currying; whilst these aren’t related to the video they are kind of cool so I’m going to digress. Bear with me.

Function Composition

Function composition lets us take advantage of the idea that f(g(x)) will always return the same value for a given value of x. And as such we can create a new function h where h(x) == f(g(x)). The composition operator ( or o) lets use define h without having to formally wrap the functions :

my &h = &f ∘ &g; # or my &h = &f o &g;

Now when you call h(x) it’s the same as calling f(g(x)). Function composition lets you create complex chains of functions that you can pass around to other functions easily.

Currying

Currying functions is when you want to take a function and generate a new function made up from partially calling the original…

Ok lets explain. Lets say I’ve got a greeting function that takes a greeting and name like so :

sub greeting( Str $greeting, Str $name ) { 
    return "$greeting $name!" 
}

Ok. Now we’d like a function that only takes a name, the greeting will be set to “Hello”. We could do something like this :

sub hello( Str $name ) {
    greeting( "Hello", $name );
}

Or possibly create a closure.

my &hello = -> $greeting { 
    -> $name { 
        greeting($greeting, $name) 
    }
}("Hello");

But Raku has a built in method for currying a function assuming which simplifies things to this :

my &hello = &greeting.assuming("Hello");

And that’s fine. But what if we want to assume a later value? That’s easy unnamed questioner we can make use of whatever stars like so.

my &greet-sam = &greeting.assuming(*, "Sam");

Now we can call greet-sam("Hi") and get back “Hi Sam!”.

By combining function composition and currying you can create complex functions from simple, easily testable, parts.

Immutable Data Structures

Raku provides us with a number of ways to have Immutable Data including Maps, Lists, Sets and Bags. You can also bind values to variables or create them as unsigiled variables or constants :

my $foo := 10;
my \foobar = 12;
constant bar = 11;

Also by default all Raku Objects are Immutable when you create them the example Point class for instance :

class Point {
	has Num $.x;
	has Num $.y;
}

You can create a Point instance assigning x and y values on creation but once it’s created you can’t modify them. Sure you can create mutable objects with is rw but if you want to do functional programming then you’re all set.

Also when you pass arguments to functions they are also immutable unless you specifically mark them as is rw or is copy. The second of these gives you a new copy you can modify inside your function that doesn’t affect its original.

So Raku provides us with all we need for Immutable data.

Bridging Systems

So if you want to pass information to and from the non functional part of your code what kind of Bridge can you use? There are two obvious answer with slightly different use cases Channels and Supplies.

Supplies can be used when we want to react to something; I’d suggest when an input comes in from the outside and you want your functional system to handle it.
You can setup a react block to take the incoming data (or function and pass it into the code of your functional code). Then when the code is done you can pass the result to a Channel that feeds the data back out to the outside world.

By combining all this together you got access to most of the functionality of a purely functional language but also a powerful object system and concurrency. The best of all worlds.

Day 18: What is my concurrent or parallel Raku program doing?

Raku makes it easy to introduce a bit of parallelism into a program – at least, when the solution lends itself to that. Its asynchronous programming capabilities also shine when building concurrent applications. During the last year, I’ve enjoyed employing both of these in my work.

However, I also discovered that something was missing. It was relatively straightforward to build the parallel and concurrent things that I wanted in Raku. However, once they were built, I found it hard to reason about what they were doing. For example:

  • For data parallel operations, what degree of parallelism was being achieved?
  • What tasks executed in parallel, and were there missed opportunities for further task parallelism?
  • Where was time being spent in different stages of an asynchronous workflow? For example, if a HTTP request could trigger background work, which later led to a message being sent over a WebSocket, where did the time go between these steps?
  • Given a Cro web application is also an asynchronous pipeline, where is time going on request processing? Is some piece of middleware surprisingly eating up a lot of time?

At first, I started putting in bits of timing code that wrote results out to the console. Of course, I either had to remove it again afterwards, or guard it with an environment variable if I wanted to keep it in the code. And even then, the data it produced was quite hard to interpret. This all felt rather inefficient.

Thus, Log::Timeline was born. Using it, I learned a lot about my application’s behavior. Best of all, I didn’t just get answers to the questions I had, but also some that I hadn’t even considered asking. In this advent post, I’ll show how you can use this module also.

Trying it out

The first way we might make use of Log::Timeline doesn’t actually involve using the module at all, but rather using some library that already uses it to do logging. Since Cro::HTTP has done this (since 0.8.1), we can get our very first glimpse at Log::Timeline in action by building a Cro HTTP application.

Aside from something that produces log events, we also need some way to view the logs. At present, there are two modes of Log::Timeline output: writing a file in JSONLines format or sending them over a socket. The second of these is used by the log viewer in Comma (a Raku IDE), meaning that we can view the output live as our application runs.

Thus, assuming we already installed Cro, we can:

  1. Create a new Cro Web Application project in Comma (works much like using cro stub at the command line)
  2. Create a “Cro Service” Run Configuration (choose Edit Configurations on the Run menu)
  3. On the Run Menu, choose “Run ‘service.p6’ and show Timeline”

The service will start up, and display the URL it is running at in the console (probably http://localhost:20000). If we make a request to it, then flip to the Timeline tab, we’ll see the request logged (in fact, do it in a browser then probably two requests are logged, since a favicon.ico is automatically requested). The requests can be expanded to see how the processing time within them is spent.

cro-app-1

Cro can process requests to HTTP application in parallel. In fact, it’s both concurrent (thanks to use of asynchronous I/O) and parallel (all steps of request processing are run on the Raku thread pool). So, if we use the Apache Benchmark program to send 100 requests, 3 at a time, we’d hope to see an indication that up to 3 requests were being processed in parallel. Here’s the command:

ab -n 100 -c 3 http://localhost:20000/

And we do, indeed, see the parallelism:

cro-app-2

Similarly, if we jack it up to 10 concurrent requests:

ab -n 100 -c 10 http://localhost:20000/

Then we will see something like this:

cro-app-3

Looking a little more closely, we see that the “Process Request” task is logged as part of the HTTP Server category of the Cro module. However, that’s not all: tasks (things that happen over a period of time) can also have data logged with them. For example, the HTTP request method and target are logged too:

data

We might also notice that requests are displayed in alternating shades of color; this is to allow us to differentiate two tasks if they occur “back to back” with no time between them (or at least, none visible at our zoom level).

Adding Log::Timeline support to an application

What if we want to add Log::Timeline support to our own an application, so we can understand more about its behavior? To illustrate this, let’s add it to jsonHound. This is an application that looks through JSON files, and ensures that they are in compliance with a set of rules (its was built for checking the security of router configurations, but in principle could be used for much more).

When jsonHound is run, there are two steps:

  1. Loading the rules, which are written in Raku
  2. Checking each specified JSON file against the rules; if there is more than one file, they will be checked in parallel

We’ll create a Log::Timeline task for each of these. They go into a JsonHound::LogTimelineSchema module. The code in the module looks like this:

unit module JsonHound::Logging;

use Log::Timeline;

class LoadRules does Log::Timeline::Task["jsonHound", "Run", "Load Rules"] {
}

class CheckFile does Log::Timeline::Task["jsonHound", "Run", "Check File"] {
}

First, we use the Log::Timeline module. Then we create a class for each task that does the Log::Timeline::Task role (there is also an Event role, which can be used to log events that happen at a particular moment in time).

Next, we need to modify the program to use them. First, in the code that we want to add logging to, we need to use our task schema:

use JsonHound::LogTimelineSchema;

And now we can go to work. Loading the ruleset looks like this:

my $*JSON-HOUND-RULESET = JsonHound::RuleSet.new;
my $rule-file = $validations.IO;
CompUnit::RepositoryRegistry.use-repository:
        CompUnit::RepositoryRegistry.repository-for-spec(
                $rule-file.parent.absolute);
require "$rule-file.basename()";

We wrap it up as follows:

JsonHound::Logging::LoadRules.log: {
    my $rule-file = $validations.IO;
    CompUnit::RepositoryRegistry.use-repository:
            CompUnit::RepositoryRegistry.repository-for-spec(
                    $rule-file.parent.absolute);
    require "$rule-file.basename()";
}

When we’re not running with any logging output, the block is just executed as normal. If, however, we were to have the socket output, then it would send a message out over the socket when the block’s execution begins, and another when it ends.

The per-file analysis looks like this:

.($reporter-object) for @json-files.race(:1batch).map: -> $file {
    # Analysis code here
}

That is, we take the JSON files, and then map over them in parallel. Each produces a result, which we then invoke with the reporter. The exact details matter little; all we really need to do is wrap the analysis in our task:

.($reporter-object) for @json-files.race(:1batch).map: -> $file {
    JsonHound::Logging::CheckFile.log: {
        # Analysis code here
    }
}

Handily, the log method passes along the return value of the block.

What we’ve done so far will work, but we can go one step better. If we look at the log output, we might see a JSON input file that takes a long time to process, but we’d not know which one it is. We can annotate the log entry with whatever data we choose, simply by passing named arguments.

.($reporter-object) for @json-files.race(:1batch).map: -> $file {
    JsonHound::Logging::CheckFile.log: :$file, {
        # Analysis code here
    }
}

And with that, we’re ready! After adding a run configuration in Comma, and running it with the timeline viewer, we get a chart like this:

jsonhound

The future

While Log::Timeline can already offer some interesting insights, there’s still plenty more functionality to come. Current work in progress uses a new MoarVM API to subscribe to GC events and log them. This means it will be possible to visualize when GC runs and how much time is spent in it – and to correlate it with other events that are taking place.

I’m also interested in exposing various Rakudo-level events that could be interesting to see plotted on the timeline. For example, it would be possible to provide information on lock waiting times, or await times, or Supply contention times. Other ideas are plotting the times files or other resources are opened and closed, which may in turn help spot resource leaks.

Of course, just because there may be a wide range of things we could log does not mean they are all useful, and logging carries its own overhead. The use of the LogTimelineSchema naming convention looks forward to a further feature: being able to introspect the full set of events and tasks available. In Comma, we’ll then provide a UI to select them.

In closing…

No small amount of the time we spend solving problems with our programs goes on finding out what happens as they run. Good tooling can provide a window into the program’s behavior, and in some cases point out things that we might not even have considered. Log::Timeline is a pretty small module, and the visualizer didn’t take terribly long to implement either. However, the payback in terms of useful information makes it one of the most worthwhile thing I’ve built this year. I hope you might find it useful too.

Day 17: Maze Maker for Fun

Maze Maker

I remember when I was in school, I used to play the game where you help the mouse go through the maze and reach the target as quickly as possible. Even
when I finished my college, I would still wonder how one can create a maze. Those who create maze must be super ingenious.

It was only recently I decided to decode the mystery of maze creation. I asked my friend Google and guess what, I was presented with plenty of solutions in various languages. Some very difficult to follow and some not so difficult. Having got my head around the algorithm, I decided to do it in my first language i.e. Perl. It didn’t took me long to convert the algorithm into Perl script.

As you all know, I am still a new player in Raku. Thankfully I have plenty of support when it comes to Raku queries. For the first time, I decided to be part of Raku Advent Calendar. Thanks to JJ Merelo, I was given a slot. At that point, I didn’t have a clue what I am going to do. But I knew I have plenty of ideas that can be converted into Raku.

As you guessed it correctly, I picked my favourite maze maker script for the Raku Advent Calendar. With the help of many people on Twitter, I got my first draft ready in Raku. But it wasn’t creating the tunnel as expected. I went to my friend, Scimon Proctor, for guidance. He helped me with the Raku magic and all working smoothly now.

It was time to do some housekeeping before I hand it over to JJ Merelo. Having spent around 30 minutes, I got my final draft ready as below:

use v6;

my %opposite-of = ( 'north' => 'south', 'south' => 'north',  'west' => 'east', 'east'  => 'west' );
my @directions  = %opposite-of.keys;

sub MAIN(Int :$height = 10, Int :$width = 10) {
    my $maze;
    make-maze( 0, 0, $maze, $height, $width );
    say render-maze( $maze, $height, $width );
}

#
#
# METHODS

sub make-maze( $x, $y, $maze is rw, $height, $width ) {

    for ( @directions.pick(@directions) ) -> $direction {
        my ( $new_x, $new_y ) = ( $x, $y );

        if    ( 'east'  eq $direction ) { $new_x += 1; }
        elsif ( 'west'  eq $direction ) { $new_x -= 1; }
        elsif ( 'south' eq $direction ) { $new_y += 1; }
        else                            { $new_y -= 1; }

        if not-visited( $new_x, $new_y, $maze, $height, $width ) {
            $maze[$y][$x]{$direction} = 1;
            $maze[$new_y][$new_x]{ %opposite-of{$direction} } = 1;
            make-maze( $new_x, $new_y, $maze, $height, $width );
        }
    }
}

sub not-visited( $x, $y, $maze, $height, $width ) {

    # check the boundary
    return if $x < 0 or $y < 0;
    return if $x > $width - 1 or $y > $height - 1;

    # return false if already visited
    return if $maze[$y][$x];

    # return true
    return 1;
}

sub render-maze($maze, $height, $width) {

    my $as_string = " " ~ ( "_ " x $width );
    $as_string ~= "\n";

    for ( 0 .. $height - 1 ) -> $y {
        $as_string ~= "|";
        for ( 0 .. $width - 1 ) -> $x {
            my $cell = $maze[$y][$x];
            $as_string ~= $cell<south> ?? " " !! "_";
            $as_string ~= $cell<east>  ?? " " !! "|";
        }
        $as_string ~= "\n";
    }

    return $as_string;
}

What next?

I am thinking of converting this into a Raku Library, something like Games::Maze. I am working on it, hopefully I should have it ready before Christmas.

If you think it can be improved in any way then please do share with me. I would love to hear from you and learn from your experience. Till then have fun with it from a Perl Fan, who equally loves Raku.

Day 16: Raku powered jmp-ing to the coalface

Santa makes sure his elves can get to and from the workplace quickly. I wish it was the same for computer programming! Sometimes it takes a while to wade through a codebase to find where the real work needs to happen.

jmp is a Raku powered command-line utility I use for searching and jumping around large codebases and command output. It provides a terminal frontend to your favourite $code-searching-tool (e.g., ag, git grep, ack etc) so you can quickly jump into your favourite $EDITOR (e.g., vim, code, emacs etc).

It works like this:

jmp-demo

When jmp-ing into large codebases I often need to visit lots of different files and repositories. Sometimes this blows my short-term memory buffer. To try and stay in flow I leave a breadcrumb trail of comments throughout the code (i.e., “# COAL”).  Later when I need to get back to the coalface of file locations I do a jmp search for ‘COAL’.

shell> jmp to COAL

That works OK but it’s a manual process and requires cleaning up all the COAL markers afterwards. Can jmp help out a bit more here?

It would be cool if jmp automatically remembered where I’d visited in a codebase so I didn’t need to leave COAL markers around. Sounds like a bit more Raku is required!

To achieve this – jmp needs a memory:

# keep a file-based record of the searches and jmps
class JMP::Memory {

    has $.max-entries = 100; # keep a record of the last 100 jmps
    has $!file;
    has @!latest-jmps;

    #| get a list of the most recent JMP::File::Hits
    method get-recent-jmps ($last-n-jmps) { ... }

    #| write the memory file 
    method save ($jmp-command, $hit) { ... }

    submethod TWEAK { ... }

}

So let’s fill this class out.

TWEAK is a special submethod for helping to lazily finish building objects after an instance is created. Not all invocations of jmp need memory recall so it’s good not to process the history file if we can avoid it. The $!file and @!latest-jmps are encapsulated as private attributes so they are easy to change in future if needed.

submethod TWEAK {

    $!file = $*HOME.add('.jmp.hist').path;
    return unless $!file.IO.e;
    return without my $history = from-json($!file.IO.slurp);
    @!latest-jmps = $history.List;

}

The handy global $*HOME IO::Path object helps to create a cross-platform location for the jmp history file (e.g., ~/.jmp.hist). If the file exists we slurp the file in one go and parse the json payload into a .List of jmp records with the help of JSON::Tiny::from-json.

Whenever a search hit occurs we need to save the jmp command and the selected hit:

#| write the memory file 
method save ($jmp-command, $hit) {

    # for each incoming hit - we record two entries - the jmp command
    my %jmp-record = %(
        current-directory => $*CWD.path,
        jmp-command       => $jmp-command,
    );

    # and the selected destination
    my %hit-record = %(
        line-number   => $hit.line-number,
        file-path     => $hit.file-path,
        full-path     => $hit.full-path,
        matching-text => $hit.matching-text,
    );

    @!latest-jmps.unshift(%hit-record);
    @!latest-jmps.unshift(%jmp-record);

    my @hits = @!latest-jmps;

    @!latest-jmps = @hits.head($!max-entries);

    # dump to disk
    $!file.IO.spurt(to-json(@!latest-jmps));
}

Unshifting the record to the start of the list means more recent jmps appear at the top. Limiting the head() of the list ensures the list won’t grow too big and jmp stays fast.

When the user calls jmp with no arguments a list of recent jmps is displayed.

method get-recent-jmps ($last-n-jmps) {
    # return a list of JMP::File::Hits 
    my @recent-jmps;
    for @!latest-jmps.head($last-n-jmps) -> %hit {
        my $hit = %hit<jmp-command>:exists
                ?? JMP::Memory::Command.new(|%hit)
                !! JMP::Memory::Hit.new(|%hit);
        @recent-jmps.push($hit);
    }
    return @recent-jmps;
}

The for loop iterates through the previously recorded jmp commands and the selected search hits. Here is the full source code of JMP::Memory.

Now jmp can help you to return quickly to the coding coalface without leaving a mess behind.

To install the latest version of jmp, first install Raku, then use zef the Perl 6 module manager to install it:

shell> zef install jmp
shell> zef upgrade jmp   # if you've installed it before

Looking forward to using more Raku-powered utilities in 2020.

Happy Christmas!