YAML and LiveCode

[mattmaier]

I found a reference to one or more JSON libraries for LC. Are there any libraries for YAML?

[FourthWorld]

There might be, but I don't recall recent mention of one. I was tempted to write such a library myself, but truth be told, fun as YAML is I have no actual need for it.

What are you looking to do?

[mattmaier]

I want to use a text file to record all of the steps necessary for a project. The basic format would be a long list of nodes with links to other nodes. Then that text file could be read out and displayed in a bunch of different ways, like as a graphical network, or as a step-by-step list of instructions, or certain pieces of information could be pulled out like the bill of materials could be calculated.

At first I designed the syntax as just what I needed, but as I get closer to trying to implement it I realized that I should probably try to use an existing data serialization format. YAML seems to have the same balance between human readability and machine readability that I was going for. It's important that people be able to understand and work with the raw file if they want to, but it's also important that a rendering engine be able to make the raw file easier to work with in different ways.

[FourthWorld]

Sounds like a cool project. I wish I had a YAML library, and if I stumble across one for LC I'll post a link here, but as far as I know at the moment you're on your own.

That said, if you'd be inclined to open source it I'd be happy to lend a hand if needed. It would be nice to see YamlToArray and ArrayToYaml handlers in the pool of community resources.

[icouto]

I see that it's been a couple of years since anyone posted on this thread. I'm looking for a YAML library for LiveCode, and my big hope is that someone might have written one during this time... I've googled around, but cannot find anything. Does anyone know of one?

[mwieder]

I never bothered to make a library out of two functions, but here's what I use:

Code: Select all

/**
Array-to-Yaml and Yaml-to-Array library
Mark Wieder 2016
wtf licenced : have fun

Public functions here:
arrayToYaml
yamlToArray
yamlFileToArray
*/

-- allow 4 spaces per indentation in output yaml files
-- adjust as desired
constant kIndent = 4

/*
arrayToYaml

Convert a LiveCode array to yaml format

@pArray : the array to convert
@pIndentLevel : initially empty, recursion sets this
*/
function arrayToYaml pArray, pIndentLevel
   local tYaml
   local tIndentLevel
   
   put pIndentLevel into tIndentLevel
   if tIndentLevel is empty then
      put kIndent into tIndentLevel
      put "ArrayName :" & cr into tYaml
   end if
   if pArray is an array then
      repeat for each key tKey in pArray
         if pArray[tKey] is an array then
            put indent(tIndentLevel) & tKey && ":" && pArray[tKey] & cr after tYaml
            add kIndent to tIndentLevel
            -- recurse to work through multidimensional arrays
            put arrayToYaml(pArray[tKey], tIndentLevel) after tYaml
            subtract kIndent from tIndentLevel
         else
            put indent(tIndentLevel) & tKey && ":" && pArray[tKey] & cr after tYaml
         end if
      end repeat
   else
      throw "not an array"
   end if
   return tYaml
end arrayToYaml

/*
indent

Used internally by arrayToYaml
*/
private function indent pIndentLevel
   local tIndentLevel
   
   -- return the proper number of spaces for indentation
   repeat pIndentLevel times
      put space after tIndentLevel
   end repeat
   return tIndentLevel
end indent

/*
indentationLevel

Return the indentation level of the supplied line
NOTE: changes the input line in situ to remove the indentation
Used internally by yamlToArray
*/
private function indentationLevel @pLine
   local tIndentationLevel
   local tKey, tValue
   
   put 0 into tIndentationLevel
   repeat while char 1 of pLine is space
      add 1 to tIndentationLevel
      delete char 1 of pLine
   end repeat
   return tIndentationLevel
end indentationLevel

/*
yamlToArray

Convert yaml format text to a LiveCode array
*/
function yamlToArray pYaml
   local tArray
   local tIndentation, tPrevIndent
   local tWorkingLine
   local tYamlHeading
   local tKey, tValue
   
   repeat for each line tLine in pYaml
      put tLine into tWorkingLine
      set the itemdelimiter to ":"
      put item 1 of tWorkingLine into tKey
      repeat while char -1 of tKey is in ": "
         delete char -1 of tKey
      end repeat
      put word 1 to -1 of item 2 of tWorkingLine into tValue
      put indentationLevel(tKey) into tIndentation
      if tPrevIndent is empty then
         -- first time through
         put tIndentation into tPrevIndent
         put tKey into tYamlHeading
         put tValue into tArray[tYamlHeading]
      else
         switch
            case tIndentation < tPrevIndent
               -- remove the last array index
               RemoveLastIndexFrom tYamlHeading
               RemoveLastIndexFrom tYamlHeading
               put tIndentation into tPrevIndent
               if tYamlHeading is empty then
                  do "put tValue into tArray[" & tKey & "]"
               else
                  do "put tValue into tArray" & yamlPathToArrayPath(tYamlHeading) & "[" & tKey & "]"
               end if
               break
            case tIndentation is tPrevIndent
               RemoveLastIndexFrom tYamlHeading
               if tYamlHeading is empty then
                  do "put tValue into tArray" & "[" & tKey & "]"
               else
                  do "put tValue into tArray" & yamlPathToArrayPath(tYamlHeading) & "[" & tKey & "]"
               end if
               put comma & tKey after tYamlHeading
               break
            case tIndentation > tPrevIndent
               -- add an array index
               if tYamlHeading is empty then
                  do "put tValue into tArray[" & tKey & "]"
               else
                  do "put tValue into tArray" & yamlPathToArrayPath(tYamlHeading) & "[" & tKey & "]"
               end if
               put comma & tKey after tYamlHeading
               put tIndentation into tPrevIndent
               break
         end switch
      end if
   end repeat
   return tArray
end yamlToArray

private command RemoveLastIndexFrom @pIndexString
   set the itemdelimiter to comma
   delete item -1 of pIndexString
end RemoveLastIndexFrom

private function yamlPathToArrayPath pYamlPath
   local tArrayPath
   
   set the itemdelimiter to comma
   repeat for each item tPath in pYamlPath
      put "[" & tPath & "]" after tArrayPath
   end repeat
   return tArrayPath
end yamlPathToArrayPath

/*
yamlFileToArray

Point to a yaml file, return a LiveCode array
*/
function yamlFileToArray pFilePath
   local tYaml
   
   put url ("file:" & pFilePath) into tYaml
   return yamlToArray(tYaml)
end yamlFileToArray

/*
*/
command testArrayToYaml
   local tArray
   
   put "hello" into tArray["hi"]
   put "bucko" into tArray["greeting"]["name"]
   put 1234 into tArray["greeting"]["mynumber"]
   put "goodbye" into tArray["bye"]
   put "510-555-1212" into tArray["greeting"]["phonenumber"]["landline"]
   put "510-555-1212" into tArray["greeting"]["phonenumber"]["cell"]
   put arrayToYaml(tArray)
end testArrayToYaml

command testYamlToArray
   local tYaml
   
   put "greeting :" & cr into tYaml
   put "    myNumber : 1234" & cr after tYaml
   put "    phonenumber :" & cr after tYaml
   put "        landline : 510-555-4567" & cr after tYaml
   put "            extension : 42" & cr after tYaml
   put "        cell : 510-555-1234" & cr after tYaml
   put "    name : bucko" & cr after tYaml
   put "hi : hello" & cr after tYaml
   put "bye : goodbye" & cr after tYaml
   put arrayToYaml(yamlToArray(tYaml)) & cr after msg
end testYamlToArray

[icouto]

Wow, Mark, thank you so much for sharing your code- it's truly helpful!

Your code seems almost complete, however it seems to expect that every line of the inputted YAML will always be in 'array' (key:value) format - like this:

Code: Select all

name: Mark McGwire
hr:   65
avg:  0.278

YAML is actually a superset of JSON: all valid JSON documents are also valid YAML. For instance: like JSON, the YAML specs also allow for **lists**. The following is a valid YAML document:

Code: Select all

- Mark McGwire
- Sammy Sosa
- Ken Griffey

YAML also allows for **comments**, which is any text - anywhere - preceded by a 'pound' sign:

Code: Select all

hr:  65    # Home runs
avg: 0.278 # Batting average
rbi: 147   # Runs Batted In

The following are valid YAML documents, which combine arrays, lists and comments:

Code: Select all

# Lists inside array
american:
  - Boston Red Sox
  - Detroit Tigers
  - New York Yankees
national:
  - New York Mets
  - Chicago Cubs
  - Atlanta Braves

Code: Select all

# Arrays inside list
-
  name: Mark McGwire
  hr:   65
  avg:  0.278
-
  name: Sammy Sosa
  hr:   63
  avg:  0.288

You can also use JSON's '[...]' notation for lists, and '{...}' for arrays - these are specially useful if placing short lists or arrays on a single line:

Code: Select all

- [name        , hr, avg  ]
- [Mark McGwire, 65, 0.278]
- [Sammy Sosa  , 63, 0.288]

As YAML was primarily designed to be a data-serialisation language that is human-readable and easy to understand, there are ways for you to break large chunks of text that would span multiple lines:

Code: Select all

name: Mark McGwire
accomplishment: >
  Mark set a major league
  home run record in 1998.
stats: |
  65 Home Runs
  0.278 Batting Average

Finally, there are some features of YAML that go *beyond* what you can do with JSON. This is where YAML starts to become really interesting, and where it gets a bit trickier to develop a parser for YAML than for JSON.

For instance, YAML has a few different ways to avoid duplication of data - e.g., by using '&' pointers to previously entered nodes:

Code: Select all

---
hr:
  - Mark McGwire
  # labels this node as 'SS'
  - &SS Sammy Sosa
rbi:
  - *SS # previous node reference
  - Ken Griffey

YAML also has **document delimiters**, to allow you to send more than one 'document' in a single stream. Three dashes mark the beginning of a document, and three dots mark the end. Below we see 2 yaml documents in a single stream:

Code: Select all

---
time: 20:03:20
player: Sammy Sosa
action: strike (miss)
...
---
time: 20:03:47
player: Sammy Sosa
action: grand slam
...

The full YAML spec is quite easy to understand, but it is unfortunately significantly more complex than JSON - you can read it in full, here:

http://www.yaml.org/spec/1.2/spec.html

The following is an example of a well-formed YAML document, with some of the language's basic features included:

Code: Select all

--- !<tag:clarkevans.com,2002:invoice>
invoice: 34843
date   : 2001-01-23
bill-to: &id001
    given  : Chris
    family : Dumars
    address:
        lines: |
            458 Walkman Dr.
            Suite #292
        city    : Royal Oak
        state   : MI
        postal  : 48046
ship-to: *id001
product:
    - sku         : BL394D
      quantity    : 4
      description : Basketball
      price       : 450.00
    - sku         : BL4438H
      quantity    : 1
      description : Super Hoop
      price       : 2392.00
tax  : 251.42
total: 4443.52
comments: >
    Late afternoon is best.
    Backup contact is Nancy
    Billsmer @ 338-4338.

[mwieder]

it seems to expect that every line of the inputted YAML will always be in 'array' (key:value) format - like this:

Not at all. It's the key/value hierarchy that's important, not the placement.I use these functions to deal with my ruby yaml files.

Code: Select all

action:
  accounts : 'Accounts'
  launch_scan : 'Launch Scan'
  view_scan_requests : 'View Scan Requests'
  view_scan_results : 'View Scan Results'

findings:
  day : 'Today'
  week : 'Week'
  month : 'Month'

policy_rules:
  all : 'all'
  email : 'email'
  malware : 'malware'
  network : 'network'
  web : 'web'

YAML is actually a superset of JSON

Not exactly. It's a similar format though, especially with YAML 1.2.
Anyway, glad you found some use in those functions.

[icouto]

YAML is a superset of JSON, and a YAML parser should be able to parse JSON documents, too - as well as do quite a bit more.

The latest YAML specs can be found here. Section 1.3 of the specs, titled "Relation to JSON", states:

(...)YAML can therefore be viewed as a natural superset of JSON, offering improved human readability and a more complete information model. This is also the case in practice; every JSON file is also a valid YAML file. This makes it easy to migrate from JSON to YAML if/when the additional features are required.

It would be truly useful to have a fully-compliant YAML parser in LiveCode. There are many web frameworks that use .yaml files either for configuration or data storage, or both, and LiveCode would have been a great tool to be able to produce GUIs for these.

[icouto]

Hi all! It's been another couple of years, and I'm wondering whether there is now a YAML parser for LiveCode? I've downloaded the latest version (9.0), but haven't been able to see anything in the docs about YAML...

[Mikey]

Trevor has one in his Levure framework, since Levure used YML for configuration.

[icouto]

I was unable to (easily) find the code in Levure that deals with parsing YAML. Nevertheless, with a little research I did find a command-line tool that converts YAML to JSON, which LiveCode can then convert into a native array. The tool that I am using is Mike Fara's "yq", which so far has performed quite well for my purpose.

So, for future reference, using yq, the code that allows me to use YAML in LiveCode:

Code: Select all

local tYQCommand, tJSON, tArray
put "path/to/yq r -j" && "path/to/sample.yml" into tYQCommand
put shell(tYQCommand) into tJSON
put JSONToArray(tJSON) into tArray