F# as a scripting language

May 19, 2022 • Christophe Moinard

Context

When an employee terminates his work contract, the human ressources need to create a settlement (solde de tout compte in french). In Timmi Leaves, our leaves management software, the Settlement Module lists both pending and completed settlements. Once the settlement is completed, the employee should not be able to acquire any new leaves.

Settlement page in Timmi Leaves

One of our clients had a problem loading this page because he had 820 pending settlements, with some of them dating back from 2013! We fixed this by adding a paging in the API response, limiting the payload to 40 items per page, instead of returning all of them.

This client also wanted to ignore settlements older than 2022. Now this is already possible in Timmi Leaves: the web client lets us ignore settlements for a month (i.e. September 2021), so in order to ignore all settlements (from 2013 till 2021), we'll have to do it manually, and it would take a while... And since it's not a recurring demand, we decided to write a one-time script.

Its logic is simple:

  1. Retrieve all settlements using HTTP GET settlements/pending
  2. Filter them to keep only the older ones
  3. Ignore old settlements using HTTP POST settlements/ignore

Piece of cake right? Well no, paging got in the way! So I decided to write the script in F#.

Why F#?

I fell in love with this language 7 years ago. I love its conciseness, its expressivity, it is also a very strongly-typed functional-first static language of the .NET framework with a lot of killer features. It is a great language for backend (Saturn, Giraffe), for frontend thanks to Fable (Elmish, Feliz, Bolero (web-assembly)) and also for prototyping and scripting.

Why did I choose F# for scripting and not a true scripting language? Well, because I'm really bad at writing bash scripts and I never understood PowerShell! I also love strongly-typed languages because the compiler guides me writing valid code. C# scripts can work too, but it misses some great features F# has and I find that the object-oriented paradigm does not fit well with scripts.

Here are the steps to work with F# in VSCode:

Scripting with F# is really easy because you can work with the REPL (F# Interactive) by evaluating portions of code and get immediate result. In order to use the REPL, set the language to F# in VSCode, select some code and send it to the REPL by pressing Alt+Enter. A REPL window opens, the code is evaluated and it shows the result.

VSCode REPL with F# code

If you want to run this article's code, you can:

  • Create a Settlements.fsx file and open it with VSCode
  • Execute the backend app on my Github repository with Visual Studio or Rider

At Lucca, we only use C# for backend and Angular for frontend. I did this script in F# only because I am more fluent with it for scripts. But maybe one day, after bothering all my collegues some evangelizing, we'll start having some F# on production, who knows ? :)

HTTP calls

The settlements/pending API returns a collection of Settlement:

{
    "count": 1,
    "items": [
        {
            "id": "c4df8d6b-be6b-48db-9bfc-32b2872683ad",
            "contractEndDate": "2014-07-21T00:00:00",
            "user": {
                "id": 1,
                "firstName": "John",
                "lastName": "Zachman"
            }
        }
    ]
}

To send an HTTP request, like C#, we can use HttpClient but it is not very functional and F#-friendly. There's a funkier and more functional way to do HTTP: FsHttp

#r "nuget: FsHttp"

open FsHttp

let httpGetPendingSettlements =
    http {
        GET "http://localhost:8085/settlements/pending"
    }

Request.send httpGetPendingSettlements

If you select this code and send it to REPL, you can see 820 settlements in the response! Quick feedback, yay!

The first line imports the nuget FsHttp in the script.

open is the equivalent of the C# using or TS import.

let is the keyword to declare a value or function. You don't need to specify the type, the compiler infers the type for you.

You can create a named function with the let keyword or lambdas with the fun keyword:

// x y are the arguments of the function add
let add x y = x + y

// increment is a function based on a lambda that takes x and return x + 1
let increment =
    fun x -> x + 1

// add type is:
// int -> int -> int
// The last type on the right is the return type
// The types on the left of the return type are the arguments
// So add is a function that takes 2 int and returns an int

// increment type is:
// int -> int

Inside the let, we use a F# feature called Computation expression. Computation Expressions are like mini-languages inside F#. httpGetPendingSettlements use the HTTP Computation Expression, but there are a lot of other Computation Expressions: ones for sequences, lists, optional values, results (alternative for exceptions), asynchronous code, validation...

Parsing the response

This call returns a Response object, but now we need to parse its content and then filter all the settlements older than 2022.

One way to do this is to create some types:

open System

// This syntax means that Settlement and SettlementsResponse are records
// It is an immutable structure and has structural equality
type Settlement = {
    Id: Guid
    ContractEndDate: DateTime
}

// If you want to use something in F#, it must be declared before
// So Settlement must be declare before SettlementsResponse
// It is true for everything, also the order of files in your project!
type SettlementsResponse = {
    Count: int
    Items: Settlement list
}

But instead of declaring the types manually, we can generate these types at compile-time thanks to Type providers:

#r "nuget: FSharp.Data"

open FSharp.Data

type SettlementsResponse = JsonProvider<
    """
    {
        "count": 1,
        "items": [
            {
                "id": "c4df8d6b-be6b-48db-9bfc-32b2872683ad",
                "contractEndDate": "2014-07-21T00:00:00"
            }
        ]
    }""" >

JsonProvider evaluates its json content at compile-time and generate the type SettlementResponse. It is NOT a dynamic type, in case of a typo there won't be runtime errors, it will not compile at all! So it is really safe to use.

There are a lot of existing powerful type providers:

  • CsvProvider considers a CSV as a list of records with the columns as properties
  • HtmlProvider creates types depending on a HTML source
  • RegexProvider creates types depending on named groups
  • OpenApiProvider generates all types from an OpenApi file
  • SqlClient execute SQL queries and verifies that the SQL code is valid depending on the database source
  • ...

Thanks to the JsonProvider, the generated type has a static function Parse that can parse json to the SettlementsResponse root type. This provider can generate multiple types, the type containing Items and Count is called SettlementsResponse.Root:

#r "nuget: FsHttp"
#r "nuget: FSharp.Data"

open FsHttp
open FSharp.Data

type SettlementsResponse = JsonProvider<
    """
    {
        "count": 1,
        "items": [
            {
                "id": "c4df8d6b-be6b-48db-9bfc-32b2872683ad",
                "contractEndDate": "2014-07-21T00:00:00"
            }
        ]
    }""" >

// If you want to be explicit about the type, you can type
// let settlementResponse: SettlementResponse.Root =
let settlementResponse =
    // Calls the GET
    http {
        GET "http://localhost:8085/settlements/pending"
    }
    |> Request.send
    // Extracts the content as string
    |> Response.toString None  
    // Converts the string as SettlementsResponse.Root
    |> SettlementsResponse.Parse

|> is a very useful operator in F#, it takes the value on its left side and passes it as last argument of the function on the right side. It is like the unix |. Very useful to chain calls:

// int -> int
let increment x = x + 1
let three = increment 2
let four = 3 |> increment

// int -> int -> int
let add x y = x + y
let five = add 2 3
let six = 4 |> add 2

let startDate = DateTime(2022, 03, 29)
let daysInMarch =
    [0..6] // Creates a list from 0 to 6
    |> List.map (fun i -> startDate.AddDays(i)) // Creates a date from each int (equivalent of C# Select or TS map)
    |> List.filter (fun date -> date.Month = 3) // Filter only dates in march (equivalent of C# Where of TS filter)

Ignore old settlements

To ignore the old settlements, we have to:

  1. Distinguish the old settlements from the pending ones
  2. Call POST settlements/ignore with old settlement ids in the payload
#r "nuget: Newtonsoft.Json"

open Newtonsoft.Json

// ... (previous code with http GET)

// Explicit type for response because the compiler cannot infer properly
let getSettlementIdsToIgnore (response: SettlementsResponse.Root) =
    response.Items
    |> Array.filter (fun s -> s.ContractEndDate.Year < 2022)
    |> Array.map (fun s -> s.Id)

let idsToIgnore = getSettlementIdsToIgnore settlementResponse

printfn "%i ids to ignore" idsToIgnore.Length

http {
    POST "http://localhost:8085/settlements/ignore"
    body
    json (JsonConvert.SerializeObject(idsToIgnore))
}
|> Request.send
|> ignore

printfn "Settlements ignored successfully"

Nothing really new here, we reference Newtonsoft.Json nuget to serialize the list of ids. http calls the settlements/ignore route with a POST and a json body. Then, its result is piped in ignore function which maps the result into () (called unit, the equivalent of void). This way, the result will not appear in the REPL.

response.Items returns an Array so we have to use filter and map of the Array module instead of List module that we used earlier.

Here's the full script:

#r "nuget: FsHttp"
#r "nuget: FSharp.Data"
#r "nuget: Newtonsoft.Json"

open FsHttp
open FSharp.Data
open Newtonsoft.Json

type SettlementsResponse = JsonProvider<
    """
    {
        "count": 1,
        "items": [
            {
                "id": "c4df8d6b-be6b-48db-9bfc-32b2872683ad",
                "contractEndDate": "2014-07-21T00:00:00"
            }
        ]
    }""" >

let settlementResponse =
    http {
        GET "http://localhost:8085/settlements/pending"
    }
    |> Request.send
    |> Response.toString None
    |> SettlementsResponse.Parse
    

let getSettlementIdsToIgnore (response: SettlementsResponse.Root) =
    response.Items
    |> Array.filter (fun s -> s.ContractEndDate.Year < 2022)
    |> Array.map (fun s -> s.Id)

let idsToIgnore = getSettlementIdsToIgnore settlementResponse

printfn "%i ids to ignore" idsToIgnore.Length

http {
    POST "http://localhost:8085/settlements/ignore"
    body
    json (JsonConvert.SerializeObject(idsToIgnore))
}
|> Request.send
|> ignore

printfn "Settlements ignored successfully"

Here comes a new challenger: Paging

The settlements/withPaging route returns a paged result:

GET http://localhost:8085/settlements/withPaging?limit=40&page=1

returns

{
    "count": 820,
    "items": [
        // first 40 settlements
    ]

}

The idea is to call this paged route and iterate on the page number. In order to compute page number parameter, we need to divide the count by the limit. So withPaging needs to be called at least once to get the items count. Before fastforwarding to the solution, let's dive into another Computation Expression : seq.

seq is short for sequence, which is an alias for the C# IEnumerable<T>. It is a structure that only has the current value and a way to obtain the next one. Unlike a collection, only the current item is loaded in memory. seq can easily create efficient sequences:

// sequence of { 1; 2; 3; 4; 5; 6; 7; 8; 9; 10; 11; 12; 14; 15 }
let seq1 =
    seq {
        yield 1
        yield 2
        yield! [3..10]

        for i in [11..15] do
            if i <> 13 then
                yield i
    }

Inside the seq Computation Expression:

  • yield v returns a single int value v
  • yield! mySeq returns every int value in the int sequence mySeq

seq can be useful for this paging issue.

First of all, let's refactor the pending http request to handle paging:

let limit = 40

let settlementsAtPage pageNumber =
    // local function
    let parseResult response =
        response
        |> Response.toString None
        |> SettlementResponse.Parse

    http {
        GET "http://localhost:8085/settlements/withPaging"
        query [
            "limit", string limit
            "page", string pageNumber
        ]
    }
    |> Request.send
    |> parseResult

query is a parameter of the http Computation Expression that takes a list of pair of strings, like Key-Value pairs. Unlike a lot of other languages, , is not a separator, (almost) every , you see in F# is used to declare a tuple value.

Now, let's use seq. This sequence will get the first settlement page, then calculate the total number of pages and iterate over all of them. With this sequence, we can filter the settlements that need to be ignored:

let idsToIgnore =
    seq {
        let settlementsPage1 = settlementsAtPage 1
        yield settlementsPage1

        let totalPages = (settlementsPage1.Count / limit) + 1
        if totalPages > 1 then
            yield! 
                [2..totalPages]
                |> Seq.map settlementsAtPage
    }
    |> Seq.collect (fun p -> p.Items)
    |> Seq.filter (fun s -> s.ContractEndDate.Year < 2022)
    |> Seq.map (fun s -> s.Id)
    |> Seq.toList

Seq.toList behaves like C# ToList, it evaluates the sequence and in our case creates a list of old settlements.

Seq.collect is the equivalent of C# SelectMany or TS flatMap, it yields each item of a given lambda's property.

If the settlements are ordered by ContractEndDate, we can optimize our code by using Seq.takeWhile instead of Seq.filter. If the condition of Seq.takeWhile is met before the end of the sequence, the iteration will stop:

let idsToIgnore =
    seq {
        let settlementsPage1 = settlementsAtPage 1
        yield settlementsPage1

        let totalPages = (settlementsPage1.Count / limit) + 1
        if totalPages > 1 then
            yield! 
                [2..totalPages]
                |> Seq.map settlementsAtPage
    }
    |> Seq.collect (fun p -> p.Items)
    |> Seq.takeWhile (fun s -> s.ContractEndDate.Year < 2022)
    |> Seq.map (fun s -> s.Id)
    |> Seq.toList

Okay, now let's execute this script by calling the fsi command (F# Interactive):

dotnet fsi Settlements.fsx

Script execution in terminal

Well that was the last step! Let's take a look at the full script:

#r "nuget: FsHttp"
#r "nuget: FSharp.Data"
#r "nuget: Newtonsoft.Json"

open FsHttp
open FSharp.Data
open Newtonsoft.Json

type SettlementsResponse = JsonProvider<
    """
    {
        "count": 1,
        "items": [
            {
                "id": "c4df8d6b-be6b-48db-9bfc-32b2872683ad",
                "contractEndDate": "2014-07-21T00:00:00"
            }
        ]
    }""" >

let limit = 40

let settlementsAtPage pageNumber =
    let parseResult response =
        response
        |> Response.toString None
        |> SettlementsResponse.Parse

    http {
        GET "http://localhost:8085/settlements/withPaging"
        query [
            "limit", string limit
            "page", string pageNumber
        ]
    }
    |> Request.send
    |> parseResult
    
let idsToIgnore =
    seq {
        let settlementsPage1 = settlementsAtPage 1
        yield settlementsPage1

        let totalPages = (settlementsPage1.Count / limit) + 1
        if totalPages > 1 then
            yield! 
                [2..totalPages]
                |> Seq.map settlementsAtPage
    }
    |> Seq.collect (fun p -> p.Items)
    |> Seq.takeWhile (fun s -> s.ContractEndDate.Year < 2022)
    |> Seq.map (fun s -> s.Id)
    |> Seq.toList

printfn "%i ids to ignore" idsToIgnore.Length

http {
    POST "http://localhost:8085/settlements/ignore"
    body
    json (JsonConvert.SerializeObject(idsToIgnore))
}
|> Request.send
|> ignore

printfn "Settlements ignored successfully"

Conclusion

This script was really stimulating to make, I was able to use many powerful features that F# provides in a few lines of code:

  • Strongly-typed
  • Conciseness
  • Strictness
  • Quick feedback thanks to REPL
  • Computation Expressions
  • Type Providers

F# is an excellent language to discover functional programming.

If you reached this point of the article, I hope that you share my opinion: you don't need to master advanced functional stuff like monoids, monads, functors nor any other unfamiliar concepts to understand and use F#.

I find scripting in F# very useful when applying Domain Modeling, implementing new ideas fast or even automating small tasks.

And more generally in my opinion, functional thinking can reshape your way of thinking (with algebric data types, immutability, pure functions). It sure helped me a lot in the past and still helps me today and everyday, even on my everlasting C# journey.

If you want to know more about F#, you can visit the F# Website, the community is great and passionate. I especially recommend the awesome Scott Wlaschin website, you can check out his videos and his excellent book about DDD in F#.

About the author

Christophe Moinard

Expert Software Engineer