Removing Nulls from strings

[stam]

Sure - was just responding to the comments immediately above regarding my post that was dealing with 2 methods.

[richmond62]

There are some would-be purists 'out there' who might like to reflect on the fact that LiveCode is the spiritual grandchild of
the thing that was hyped as "for the rest of us"; and, as, in the last year I have made no money whatsoever by directs sales,
about $200 from teaching LiveCode, and (an admittedly generous) donation of $200 for my Devawriter Pro: I slot right
bang-in-the-centre of the 'for the rest of us' crowd (mind you, whether that is still a crowd is infra dig).

My approach to removing Nulls was, as are my usual efforts, based on 2 ideas:

1. Everything should be manageable inwith LiveCode, pure and simple.

2. While a spade might be clunky, it can shift earth.

I would hate to think that LiveCode and its installed base were becoming a bunch of snobs, resulting
in LIveCode becoming such a coterie fetish it is at risk of disappearing up its own . . .

I have nothing against the regex solutions, Ding An Sich, but the bottom feeders like me require something a little more obvious.

[Simon Knight]

I'm with Richmond62. I bought into Livecode because it was advertised as a tool for subject matter experts to use to get their tasks done especially when the company was too tight to supply Matlab.

So while I now understand that the ReplaceText command uses RegEx and that I should have read the dictionary I think that Livecode could have either converted the null constant to the correct RegEx code within the code of the command or posted an error along the lines of "please use a regular expression".

So that was null....... have you discovered what happens when a filename contains a question mark ?

https://quality.livecode.com/show_bug.cgi?id=23676

S

[stam]

Guys no need to get all purist “should all be doable in livecode” nonsense.

Use the tool you’re comfortable with.

Richmond: the one liner “LiveCode” way of doing it is

Code: Select all

replace null with empty in <container>

No regex involved. And the nice thing is that you are not constrained to either method.

Don’t use regex if it’s not what you need and you’re not sure how to use it. Not complicated and it’s not the first time this has arisen. But I can guarantee every time regex is mentioned someone will jump up and start moaning about code purity.

The question was asked, and 1-liner options with and without regex were discussed.
Forgive me if I disagree with your repeat until method. It’s needlessly “expensive” and complicated for me.

[FourthWorld]

You could also shell out to awk. ::ducking::

[Simon Knight]

https://www.tutorialspoint.com/awk/awk_ ... amples.htm

Oooo heck - no thanks but you go ahead

S

[stam]

So that was null....... have you discovered what happens when a filename contains a question mark ?

https://quality.livecode.com/show_bug.cgi?id=23676

erm... I cannot reproduce your issue.
I just get the entire filename with the '?' as expected on a single line (MacOS 12.2.1, LC 10.0 DP3)

I often will post suspected bugs in the bug triage forum to ensure it's reproducible and a valid bug report...

[richmond62]

I am not anti-regex or any other way to get things done quickly and efficiently

(which my repeat loop offering signally does not),

what I am against is the implication that fairly clunky ways of doing things
(which might be easier for beginners and 'the rest of us' to understand)
are to be poo-pooed and rejected.

One of the things I greatly admire about LiveCode is the ability to get things done
in a number of ways: some of which are generally quick and efficient, and some of
which are fairly clunky.

To my mind, as a teacher (and in this context, of programming to people who have
no previous programming experience nor computational thinking) the virtue of the clunkier
methods is that, on the face of things, they are generally easier for "freshers" to understand.

[stam]

Leaving regex aside - while i agree verbose ways of doing things are easier for less experienced people to grasp, it does not follow that more verbose is more understandable.

My point is that

Code: Select all

on mouseUp
   put empty into fld "FIXED"
   put fld "ProblemName" into PN
   repeat until PN is empty
      put char 1 of PN into FX
      delete char 1 of PN
      put codePointToNum(FX) into MNUM
      if MNUM is (0x0) then
         --- do nix ---
      else
         put FX after fld "FIXED"
         end if
   end repeat
end mouseUp

is much less clear and easy to grasp (and also much less of what i think of as LC) than

Code: Select all

replace null with empty in <chunk>

But anyway, anything that gets the job done is fine. Code that is easier to understand and maintain 2 years later is better

[Simon Knight]

erm... I cannot reproduce your issue.

Strange. In my case the question mark was in the middle of the filename part of the path. I have no idea how the character became part of the name as I would not use it and I believe that MacOS uses it via the Finder to indicate a problem with the file - however I am unsure that this is true.

I have just double checked and here is a screen shot of the results when the following code is actioned:

Code: Select all

on mouseUp pButtonNumber
   put empty into field "debug"
   answer folder "Please select a folder?"  
   if the result is "Cancel" then exit mouseup
   put it into tFolderPath
   
   set the defaultfolder to tFolderPath
   
   put the files into field "debug"
   
end mouseUp

Question Mark issue

(Field debug is shown in front of the Finder window in the screen shot)

I tried to attached the actual image files but it looks like the same issue has occurred : the question mark is removed and a line break added.

Next I've tried exactly what you have tried and the results are the same as yours :

2022-04-14-091254-Screenshot 2022-04-14 at 09.12.49.png (38.59 KiB) Viewed 3204 times

Looking at the original file name in a hex editor does not reveal anything odd i.e. it is just a question mark.

I have no idea why the image files are causing problems except to say that it must go beyond the text of the filename. Apple uses a resource fork so perhaps that is getting involved.

I'll add some notes to my bug report which I feel is still valid.

S

[Simon Knight]

Ho hum,

That Question Mark is not a question mark, its a new line (I think but who knows). Here is what I have added to the bug report :

Prompted by comments made on the user Forum I now realise that step four above will not reproduce the fault.

It seems that the question marks in my file names were added at some point by MacOS and by some process were incorporated within the body of the name. So this may or may not be an edge condition.

Using the terminal is revealing :
Simons-MacBook-Pro:QuestionMArk-SingleFile skids$ ls -la
total 312
drwxr-xr-x 3 skids staff 96 14 Apr 09:38 .
drwx------+ 133 skids staff 4256 14 Apr 09:38 ..
-rw-r--r-- 1 skids staff 159349 7 Mar 2021 2000-07-27-132916-16Jul00-53-Eire-Places-LandScape?LandScape-.jpg
Simons-MacBook-Pro:QuestionMArk-SingleFile skids$ ls | hexdump -c
0000000 2 0 0 0 - 0 7 - 2 7 - 1 3 2 9 1
0000010 6 - 1 6 J u l 0 0 - 5 3 - E i r
0000020 e - P l a c e s - L a n d S c a
0000030 p e \n L a n d S c a p e - . j p
0000040 g \n
0000042
Simons-MacBook-Pro:QuestionMArk-SingleFile skids$

I read that as a new line character is in the body of the filename and no, I have no idea how it got there. Ouch!

I imagine this is a tricky issue to fix so perhaps a workaround and a warning in the dictionary is the way to go.

(Step four was creating a file name with a question mark within it)
It seems that MacOS uses the question mark to indicate a character that has no symbol. How that symbol appears in the file name is unknown.

My thoughts are some defensive code that acts when "the result" returns a problem and melds the current line with the one following and tries again.

Hopefully this type of file name is rare. My own files date from the year 2000 so will have been processed by OS 8 before OS 9 and then X although adding "Landscape" would have happened much later.

S

[Simon Knight]

It seems that Unix allow all but two characters in a file path. The two not allowed are "/" and ASCII null "\0". "/" is the directory indicator and ASCII null is the terminator.

From here : https://dwheeler.com/essays/fixing-unix ... names.html. read and weep?

[stam]

My first thought when reading your previous code is that there is a LF char (ascii 10) or other funny control char that's leading to the new line.
Probably good to defend against this if dealing with old/suspect/legacy names especially if from Windows or other OS.
Quite easy to do, just only allow chars with ascii values > 31 (i.e. 32 or higher) in the filename list.

Actually this would be a good use case for @Richmond's repeat until loop.

Still not sure i'd class that as a 'bug' though... difficult for LC to know if it should strip such characters as there may be circumstances where it's intentional.

----------------------------------------------------------------------------
EDIT: simple to create a function to clean file names for you:

Code: Select all

function getCleanText pText
   # PURPOSE : Strips control characters out of a string
   local tText
   repeat for each char tChar in pText
      if nativeCharToNum(tChar) > 31 then put tChar after tText
   end repeat
   return tText
end getCleanText

untested as i don't really have ready access to text with funky chars in them but this should work... Mind you if you want to have text that has tabs or CR in it, you'd need to modify the 'if' statement above.

[richmond62]

also much less of what i think of as LC

Ha, Ha, Ha: what I think of as LC is, obviously, rather different to what you think of as LC.

That, of course, should not matter: it is lovely that LC is sufficiently elastic-sided that it can
accommodate both opinions.

[richmond62]

Just to throw a spanner in the works: the Q-Mark-Problem stack
behaved as it should at work on my Mac OS 10.7.5 machine with
LC 8.1.10:
-