Mapping handler dependencies

[mwieder]

?

Code: Select all

repeat for each line aLine in tCollect

shouldn't that read

Code: Select all

repeat for each line aLine in pScript

[SWEdeAndy]

?

Code: Select all

repeat for each line aLine in tCollect

shouldn't that read

Code: Select all

repeat for each line aLine in pScript

Yes, the latter's correct.
Thanks Bernd (& Mark) for this improvement - I wasn't aware of the token keyword! It's apparently tailor-made for LC script parsing.
I also didn't test my stripping method with a big enough script, so I didn't realise what a speed penalty it caused.

The token method will be implemented in the next version.

Concerning block comments (/* <code> */), that is harder to deal with. I guess the whole chunk between the block markers could be deleted by a replaceText regex, but then we're back at reducing speed a bit. Though it would be applied to the whole handler, not needing to go through each line. Could be a user option to turn on/off perhaps.

[bn]

?

Code: Select all

repeat for each line aLine in tCollect

shouldn't that read

Code: Select all

repeat for each line aLine in pScript

Mark,

You certainly have a way of spoiling a speed record
Now it is a little below 6 seconds instead of 21.

Thanks
Bernd

[SWEdeAndy]

With a script mass amounting to 600+ handlers, I get:
18,3 secs with comment stripping (Bernd's method)
3,6 secs without any comment stripping

So, I'm making it a user option, with a warning on speed effects.

I found an easy way to skip block comments, by the way:

Code: Select all

put replaceText(pScript,"\/\*[\s\S]*?\*\/","") into pScript

It might also be an idea to run a replaceText to ditch all lines beginning with a comment sign. I'll test if that speeds things up.
Actually, I could even start with a check on whether the handler contains any comments at all, an if not then the whole line-by-line check can be skipped.

[bn]

Hi Andy,

I found when clicking on a line number in widget "handlerCatalogueArray" with a number in the thousands which make the action inspect icon appear triggers an error in its handler
"actionInspect pPathToNode" at "do tPath". Double clicking on the partial visible number has the expected result to go to that line in the script.

I suggest to set the "separatiorRatio" of the treeview widget to 0.95 which is for me enough to diplay the number without the "icon inspect" being shown.

Additionally you could the change handler "actionInspect" of widget "handlerCatalogueArray" to

Code: Select all

on actionInspect pPathToNode
   actionDoubleClick pPathToNode
end actionInspect

Kind regards
Bernd

[SWEdeAndy]

Thanks Bernd, you are quite right. This goes right into the next version.

I've also found a very efficient way of increasing the speed of the analysis, even with comment stripping:

In function searchHandler, an early line of

Code: Select all

if pLookFor is not in tScript then return 0

does a crude check if the handler word at all exists in the script, before any slowing line-by-line check is initiated. Since most handlers are not called by most other handlers, this significantly cuts the time taken for the whole run. I should have seen this long ago...

In my tests, the comment stripping hardly registers now, so that might not need to be a user option at all. But I'll keep a tick box for it for now, so you guys can test if there is a noticeable difference in your material.

[SWEdeAndy]

OK, new version with pretty good improvements, both in speed and functionality:

v1.1.3 (27 Aug 2023)
- Fixed bug: Saving user projects and settings didn’t trigger properly at closeStack. Now triggers at closeCard instead.
- Enhancement: Handler analysis and comment stripping is now considerably faster to complete, especially in large projects
- Enhancement: If a handler is selected in the Handler dependency stats field when creating a flowchart, all the arrows leading to and from that handler node will be coloured, to make them easier to follow. On MacOS, this will also happen when clicking a handler in the flowchart (in the browser widget).
- Enhancement: Flowchart settings moved to its own settings pane, and node colours made modifiable

https://github.com/wheninspace/WIS_Scri ... ses/latest

@Paul: I'll be interested to know how your 2000 handler project performs in this version. It should cut the processing time by 80-90%, I think.

[bn]

Hi Andy,

thank you for version 1.1.3.

It is very fast again even if I uncheck "omit commented-out code" in preferences. Quite an improvement.
I would even suggest to make it the default setting and let the user opt-in.
As far as I understand the feature it is to prevent to pick up commented code parts that would be interpreted as part of the dependencies.
Since your aim is to list those dependencies the user would be surprised to find commented-out parts of scripts displayed as part of the dependencies.

The one new feature I like best is the colorization of the path when clicking on a node. That helps a lot in interpreting the diagram when drilling down the dependencies.

A very nice new version.

Kind regards
Bernd

[mwieder]

The one new feature I like best is the colorization of the path when clicking on a node. That helps a lot in interpreting the diagram when drilling down the dependencies.

Ah. I missed that feature. Yes, that's a nice touch.

[SWEdeAndy]

It is very fast again even if I uncheck "omit commented-out code" in preferences. Quite an improvement.

That's good! But I may have been a little unclear in my explanation and/or labelling:
If the box is checked, it will strip out comments. If not, it will leave them in. So if anything, it should run a little bit faster with the box unchecked (though that might depend on the amount of commented code).

However, you seem to observe the same as I did - the comment-stripping has almost no impact any more on the total time taken to complete the analysis. So, unless our 'monster script' colleagues observe differently, I will probably remove the check box in future versions.

Your understanding is quite correct: commented-out parts of scripts should never be included in the analysis, and it was a bug that they ever were. It was just because my initial, suboptimal comment stripping code made the whole run so slow, that I made it a user option, just so we could test it.

[jacque]

Just tried version 1.1.3, it's terrific. The colored lines make it so much easier to trace paths. I tried with and without commented code and the speed didn't seem any different to me, but the flowchart seemed less complex with the option to omit commented handlers turned on. My test stack has a lot of commented handlers. Some are duplicates I kept for reference in case my changes didn't work, but others are independent handlers I no longer needed. So there is a difference there. I'd leave the option available I think in case others keep extra junk in their scripts like I do. (On final release I try to remove the unused stuff but during development I always think I might need it later.)

This tool is becoming very professional. I was delighted to discover I can resize the mainstack as well as the flowchart. That makes long lists much easier to read. I can't recall which monster script I used originally but this one has 75 handlers and a couple thousand lines. I didn't get the text limit warning.

I next tested a truly monster stack with 332 handler names. It analyzed correctly but I couldn't make a flowchart. No errors, but the window didn't display anything, it just remained blank. I added an extra zero to the character limit but that didn't help. Commented handlers were omitted. I also filtered out four or five handlers but I needed to see the others.

Still, this is a great tool.

[SWEdeAndy]

Just tried version 1.1.3, it's terrific. The colored lines make it so much easier to trace paths. I tried with and without commented code and the speed didn't seem any different to me, but the flowchart seemed less complex with the option to omit commented handlers turned on. My test stack has a lot of commented handlers. Some are duplicates I kept for reference in case my changes didn't work, but others are independent handlers I no longer needed. So there is a difference there. I'd leave the option available I think in case others keep extra junk in their scripts like I do. (On final release I try to remove the unused stuff but during development I always think I might need it later.)

Yes, the standard setting should be (as it is now) that commented-out handlers are ignored. It doesn't make much sense to see them in the lists or flowchart. But I'll leave in the possibility to turn that filter off, just in case someone wants that.

This tool is becoming very professional. I was delighted to discover I can resize the mainstack as well as the flowchart. That makes long lists much easier to read.

Oh yes, I've put some effort into the responsive design, and it's been in there since the early days really. I deliver the stack in it's minimum size just in case someone opens it on a laptop, but it's definitely easier to use on a large screen.

I can't recall which monster script I used originally but this one has 75 handlers and a couple thousand lines. I didn't get the text limit warning.
I next tested a truly monster stack with 332 handler names. It analyzed correctly but I couldn't make a flowchart. No errors, but the window didn't display anything, it just remained blank. I added an extra zero to the character limit but that didn't help. Commented handlers were omitted. I also filtered out four or five handlers but I needed to see the others.

Still, this is a great tool.

I'm almost positive that I know what your blank flowchart problem is. It happened to me as well. It's actually nothing wrong at all: I bet you have unticked the Limit width box, right? And then a very large flowchart is produced, but your view in the browser widget is naturally scrolled to the top left when it loads. Enlarge the flowchart window, and/or try scrolling down and to the right, and you'll soon start seeing some nodes. Big flowcharts can have big areas of white space, it all depends on the code structure.

[jacque]

I'm almost positive that I know what your blank flowchart problem is.

You are exactly right. The chart is there but in this case it is too large to view even fullscreen on my 27" monitor. But yup, all good. I will practice scrolling.

[jacque]

For the exclusions, I want to exclude everything containing the string "hmac" but I'm not sure what to use. I tried these variations without success:

.*hmac.*
*hmac*

My regex knowlege is pretty limited. The handlers I want to exclude are all in a library substack. Is there a way to exclude the entire substack?

[SWEdeAndy]

For the exclusions, I want to exclude everything containing the string "hmac" but I'm not sure what to use. I tried these variations without success:

.*hmac.*
*hmac*

My regex knowlege is pretty limited. The handlers I want to exclude are all in a library substack. Is there a way to exclude the entire substack?

*hmac* should do the trick.
I tried adding some handlers to a stack, called "hmac", "hmac2" and "xHMACx". Putting *hmac* in the exclusion field filters them all out as expected. Putting ?hmac* filters out only "xHMACx", and putting hmac* filters out "hmac" and "hmac2" etc.
You need to rerun the mapping and analysis after you change a filter though, they're not applied live (as opposed to the Handler dependency stats filter).

Excluding whole substacks is not possible at the moment, but that seems like a useful feature to add. I'll need to find a way to cram that too into the UI then.