Square brackets in replacement string

[heyvern]

This is driving me crazy.

Below is the source from a file that needs to be changed. I am using "for each line in" to "move" the open bracket up one line so that the "camera" values are easier to handle using itemDel.

Code: Select all

camera_track
[
	keys 3
		0 1 0.1 0.5 0 0 1.7321
		1 1 0.1 0.5 0 0 1.7321
		12 1 0.1 0.5 0 0 1.7321
]
camera_zoom
[
	keys 3
		0 1 0.1 0.5 1
		1 1 0.1 0.5 0.915311
		12 1 0.1 0.5 0.915311
]
camera_roll
[
	keys 3
		0 1 0.1 0.5 0
		1 1 0.1 0.5 0.259061
		12 1 0.1 0.5 0.259061
]
camera_pan_tilt
[
	keys 3
		0 1 0.1 0.5 0 0
		1 1 0.1 0.5 -0.059462 0.216465
		12 1 0.1 0.5 -0.059462 0.216465
]

I am using a repeat structure like this for speed:

Code: Select all

local tcount
put 1 into tcount
repeat for each line i in varText
  if i contains "camera_" then
     replace "[" with "" in line tcount + 1 of varText
     replace "camera_" with "[camera_" in line tcount of varText
  end if
  add 1 to tcount
end repeat

THIS WON'T WORK.
It only changes the FIRST "camera_" line encountered. After that it won't finish the others. The odd thing is that if I change the camera text using anything that doesn't have a bracket in it AND any form of the word camera it works fine. Putting the "[" in the replace text messes up the replacement using the incremented tcount variable. Also if I use any of the letters of "camera" in that order will also break the repeat. For example if I try to replace "camera_" with "---cam" doesn't work. Anything else will work.

This DOES work:

Code: Select all

local tcount
put 1 into tcount
repeat for each line i in varText
  if i contains "camera_" then
     replace "[" with "" in line tcount + 1 of varText
     replace "camera_" with "dagnabbit" in line tcount of varText
  end if
  add 1 to tcount
end repeat

No amount of fiddling with escaping or charToNum with the bracket works. Nothing will put a freaking bracket in front of that line. That "in line tcount + 1" is not the problem either. I thought it might be incrementing the tcount variable but it is not. I've tried everything like using... put "[" before line... etc etc.

There's another issue. If I use the format below it works fine. Notice I am using the tcount to check for the line... however this makes the script very very very slow. The files I am changing are huge. The difference in time between these two is astronimcal, 2 seconds without counting the lines compared to 10 minutes using the code below.

Code: Select all

local tcount
put 1 into tcount
repeat for each line i in varText
  if line tcount of varText contains "camera_" then
     replace "[" with "" in line tcount + 1 of varText
     replace "camera_" with "dagnabbit" in line tcount of varText
  end if
  add 1 to tcount
end repeat

There appears to be some weird "offset" with brackets in the replace string. I tried using replaceText() which has the same problem. The "[" is causing no end of problems. Changing it to a char number has no effect. there doesn't seem to be a way to use square brackets in a replacement string. And what's the problem using the word "cam" as replacement text?

I am going insane on this one. Driving me crazy.

[sturgis]

If I understand what you're trying to accomplish, this should do what you want.

Code: Select all

        -- first line finds the preceeding return, plus any number of leading chars/spaces prior to [
        --then matches the escaped [.  Need the backslash prior to [ so the regex engine knows to treat it as  literal.
   put replaceText(tText,return & ".*\[",empty) into tOut 

        --Finds the word camera, and replaces it with [camera.  No preceeding \ is necessary here
        --because it dosen't have to actually process the string and hence doesn't care that there is a [ there.  
    put replaceText(tOut,"camera","[camera") into tFinal

[heyvern]

This is freaking weird. I decided to figure out why "camera" can't be replaced with certain combinations of letters. Certain replace string character combos don't work as if some weird regex thing is going on. Can't figure this out. I went with simple string replacements in the repeat that are very simple.

for example if I change the text with this random string "dooghyt" it won't work. If I use "doghyt" it works... one "o" instead of 2. What's up with that????? Any letter in front of more than 1 of another letter fails after the FIRST replacement on the repeat. This is freaking bizarre. Weirdest thing I have ever seen.

Does the "replace" use regex? I thought it didn't that was why it is faster? If you use quotes isn't that just a string? Why would certain letter combinations have problems? Why would repeating characters break it?

[heyvern]

Thanks Sturgis,

You posted while I was still... posting.

Your solution works perfectly... uh... except it strips out all the returns. I should be able to play around with and get it to not strip the returns. I need the returns.

So... what about the "[" in "replace"? Shouldn't that be a literal string? Any clues about the weird character combos that won't replace? it seems odd to me. I am happy to find alternatives just would like to know when and why I would need to and what character combos I need to avoid.

[heyvern]

Nevermind. Works great.

I was still doing it "line by line" instead of for the whole text chunk. Your solution on the whole chunk works.

The problem is I am having difficulty splitting this file format into usable chunks. There are not consistent delimiters like this and often it is "inconsistent". It's been a real chore trying to break the file apart. It will be tricky to find that spot specifically to process it which is why I used the line by line method. The final goal is to convert the entire file format to xml. I have about 90% converted so far. I have a bunch of other sections of this file format process with the same repeat structure that works perfectly. I only run into trouble trying to "move" the square bracket. Did the exact same thing for the "{" bracket which works fine. It's the stupid square brackets that seem to give trouble.

Thanks for the wicked fast response. Was going crazy.

[sturgis]

If you only care about getting rid of the [ itself, just use "\[" as the match string. This will get rid of the bracket, but leave the line intact including the preceeding spaces.

However, I made it much more difficult than it needs to be based on your data.

Just do this.

Code: Select all

   replace "[" with empty in varText
   replace "camera_" with "[camera_" in varText

Having said all this, I suspect there is something going on in your "non-working" code that has to do with the way containers are seen, but this is purely a guess. The reason the last option works but is so slow is that its having to cycle from the first line to the current line being checked every time, not very efficient. If your data matches the sample you gave throughout the whole dataset, then the 2 consecutive replaces should work fine.

[sturgis]

Out of curiosity, what OS are you on to do this? And are you the only one who needs to use your app?

If the files are as huge as it sounds, you might consider shelling out and using something like sed to do the processing. Its been a while since I used sed, but I think you can either have it modify the files where they sit (backup first of course) or create new files, but sed is REALLY fast, and really good at this sort of thing. Dunno if there is a sed for windows, (probably is) but if you're on mac or linux, it's most likely sitting there waiting for you to use.

In fact, while I hate to say it here, you can probably use sed without runrev to beat the files into whatever form you need.

Just a thought.

[heyvern]

I am going to develop for both mac and win. I am developing on win right now. Speed is not an issue. I have processed HUGE files with my "prototype" in seconds. 3-6mb.

What I am trying to develop is an external editor for another applications file format that is ascii text. The files are "simple" but the formatting is not... easy to deal with.

The snippet in this thread is only one small section of the entire file format. There are multiple different areas that use the [] to hold data. Each of those sections may be slightly different. I can't globally replace the [ in the whole file so I have to break it up into "chunks" and process either each line of each chunk or do what you did... actually I really appreciate the code you provided. It has inspired me. I am creating some simpler functions and commands to process "chunks".

There are several "chunks" or characters in the file format I can use to break up the imported file. Sometimes there are opening and closing elements but sometimes not.

Code: Select all

### animate values (or some other type of thingy)
--- animated values here in brackets
### styles
--- style values here in brackets but formatted differently than the above
[
key values 
]

{
layers
}

So I can break the file up into chunks using the "### " as delimiters. But these are actual opening and closing sections. One item starts with this and then there is a double return and the next section starts. I replace those with a new itemDel. However in only one part of the file format that "### " is not a "section" or chunk and in ONE other spot there is no return after the section, so I have to handle those differently.

Further down I can break data up using the [] and also the {}. The [] is the most common element and unfortunately the "description" for that container is always out side the brackets. So the solution you provided is going to be used a lot.

it gets VERY confusing to figure it all out along with strange inconsistencies in the application file format. Like some areas use _ instead of spaces while others don't. In one spot a single entry uses a freaking hyphen... so I have to account for that one single line difference.

When I am done the file format will be converted to xml so my application can display and interact with the file format easily. Hopefully going "backwards" to save out the modified file format should be easier since the xml has a more logical structure.

[sturgis]

Well while you're looking at all this, you might add matchText and matchChunk to your experimentation. Especially matchChunk, both work with regex expressions, and though I can't really picture the rest of the format of the file, my guess that if a file has big sections that all relate, you could use matchChunk to find the start and end of said "chunk", move that chunk to a variable, process it, and after removing that chunk from the original, proceed to grab the next section.

Regular expressions can be a tremendous pain, or a tremendous help.
I think it largely depends on the alignment of the moon and stars as to which type of tremendous it will be for that day.

Good luck.

[heyvern]

I love regex. I use jEdit for all of my coding and text editing. I had to sort of "relearn" regex using RR. It's a bit different from what I'm use to. In the past I did a ton of lua scripting for this same application I am creating my gizmo for. Lua is "almost" similar to RR. I actually use jEdit and save my RR code as lua so I get "similar" syntax highlighting. I don't know if what is it... transport? Does it have a file extension that might be in jEdit? anyway.

Yes. I am going to do exactly what you described. Break into chunks and nail them in one shot. I am actually processing the "original" stored in one variable and putting the changed chunks into another variable and removing the chunks from the original so everything matches up. At the end I just send the final xml format to that cool xml gizmo thingy I found. Works like a charm.

The animation software I am developing this for has ZERO project management tools. Nada... zip... none. So I am hoping to create something that can manage projects or groups of files for easier management. With the cool online features of RR maybe some sort of collaborative online project manager.

This all started because the new version of my favorite animation application doesn't save backwards to older versions. Anything created in the new version can't be opened in the old version. Many people would like to use both. I was able to convert new files to old files using jEdit and a bunch of regex so I thought creating a simple file conversion utility in RR would make sense. In the past only a few lines needed changing but now depending on the size of a file there could be thousands and thousands of lines to change to save as an older file format. After fiddling with it I decided all the effort would be wasted if MY application did only that "one thing". It would have taken nearly as much effort and then be stuck as a "one trick pony". By converting to xml format it can do the "save as" function really easily AND potentially do much more with the file format later on.

Fingers crossed.

[BvG]

i hate regexp, and I never learned it. Therefore I use the itemdelimiter to do open-close bracket parsing:

Code: Select all

on mouseUp
   --I assume there's no keys within the data that actually contain "camera_"
   put field 1 into theData
   set the itemdelimiter to "["
   set the linedelimiter to "]"
   repeat for each line theLine in theData
      if theLine contains "camera_" then 
         put "[" & item 1 of theLine & item 2 to -1 of theLine & "]" & return after theResult
      else
         beep
         --should not happen
         put theLine & "]" after theResult
      end if
   end repeat
   put theResult
end mouseUp

But there'd be empty lines all over the place. I think your data is actually a form of an encoded array, so what i'd do to get data out of it is actually this:

Code: Select all

on mouseUp
   --this assumes that whitespaces are not ignored
   --which means that returns are not doubled or omitted
   put field 1 into theData
   replace return & "[" & return with "[" in theData
   replace return & "]" & return with "]" in theData
   split theData by "]" and "["
   put the keys of theData into theList
   repeat for each line theLine in theList
      put "[" & theLine & return & theData[theLine] & "]" & return after theResult
   end repeat
   put theResult
end MouseUp

[heyvern]

Thanks BvG,

That is close to the code I was using to get the bracketed chunks.

Another wrinkle in the mix is the fact that the brackets can be nested within other different elements that aren't "open and close". So there could be a larger chunk separated only by a single line starting with "### " and a return. Inside that "might" be another chunk separated by a single line of "### " or maybe it might be additional nested "[]" or "{}". Or it might be a whole section of "### " that doesn't have returns etc etc.

I've decided to use a bit of "brute force" to break up the data. The file format has a consistent list of items that appear in the larger chunks. I am using a list of strings to check for each chunk group. These groups repeat many many many times through out the file and can be nested. For example each layer group has a list of generic values that can have keys. But a GROUP layer can have layers INSIDE it that also have these values. The layers are delimited using "{}". So the big issue will be making sure I don't mismatch the nested groups.

I made huge progress using Sturgis's regex. I love those two lines of code baby! . Very fast and works for many different sections by tweaking the regex. With modifications I can use it all over the place. I created a custom function that can work on each chunk. I first get the start and end of each chunk using lineOffset for the specific chunk.

For example I now that:

Code: Select all

### animated values
--- animated values here with other bracketed elements
### layers

... Is consistent in all files.

Here is a sample of the entire file format I am processing. It is very simple and has a fraction of what could be in one of these files but gives you the idea.
I have already finished handling the "header" section since it is always the same in every file. The section up to "### styles" is what I'm calling the "header" for now. From that point on things could be different for each file. In this sample there are no saved styles so there is nothing below that line. In a different file there would be style names and key groupings with "[ ]". I had thought maybe to use the leading "tab" indents for handling nesting.

Code: Select all

application/x-vnd.lm_mohodoc
version 19
### Created in Moho version 6.1, � 1999-2009 Smith Micro Software, Inc.
### Created: Sun Feb 21 13:31:50 2010

### static values
dimensions 872 486
frame_range 1 240
fps 24
back_color 234 234 234 255
noise_grain 0
allow_3d true
depth_sort false
distance_sort false
depth_of_field false 2 1 0.05
stereo_separation -1
extra_swf_frame false
soundtrack ""

### styles

### animated values
camera_track
[
	keys 1
		0 1 0.1 0.5 0 0 1.7321
]
camera_zoom
[
	keys 1
		0 1 0.1 0.5 1
]
camera_roll
[
	keys 1
		0 1 0.1 0.5 0
]
camera_pan_tilt
[
	keys 1
		0 1 0.1 0.5 0 0
]

### layers
layer_type 3
{
	### generic layer values
	name "group layer"
	quality_flags 2046
	animated_layer_effects false
	origin 0 0
	parent_bone -2
	visible true
	render_only false
	edit_only false
	scale_compensation true
	rotate_to_follow false
	face_camera false
	masking 0
	blend_mode 0
	camera_immune false
	dof_immune false
	timing_offset 0

	### transforms
	translation
	[
		keys 1
			0 1 0.1 0.5 0 0 0
	]
	scale
	[
		keys 1
			0 1 0.1 0.5 1 1 1
	]
	rotation_x
	[
		keys 1
			0 1 0.1 0.5 0
	]
	rotation_y
	[
		keys 1
			0 1 0.1 0.5 0
	]
	rotation_z
	[
		keys 1
			0 1 0.1 0.5 0
	]
	flip_h
	[
		keys 1
			0 1 0.1 0.5 false
	]
	flip_v
	[
		keys 1
			0 1 0.1 0.5 false
	]
	shear
	[
		keys 1
			0 1 0.1 0.5 0 0 0
	]

	### layer effects

	### group layer values
	expanded true
	group_mask 0
	depth_sort false
	distance_sort false
	layer_ordering
	[
		keys 1
			0 1 0.1 0.5 ""
	]

	### sub-layers
	layer_type 1
	{
		### generic layer values
		name "vector layer 2"
		quality_flags 2044
		animated_layer_effects false
		origin 0 0
		parent_bone -1
		visible true
		render_only false
		edit_only false
		scale_compensation true
		rotate_to_follow false
		face_camera false
		masking 0
		blend_mode 0
		camera_immune false
		dof_immune false
		timing_offset 0

		### transforms
		translation
		[
			keys 1
				0 1 0.1 0.5 0 0 0
		]
		scale
		[
			keys 1
				0 1 0.1 0.5 1 1 1
		]
		rotation_x
		[
			keys 1
				0 1 0.1 0.5 0
		]
		rotation_y
		[
			keys 1
				0 1 0.1 0.5 0
		]
		rotation_z
		[
			keys 1
				0 1 0.1 0.5 0
		]
		flip_h
		[
			keys 1
				0 1 0.1 0.5 false
		]
		flip_v
		[
			keys 1
				0 1 0.1 0.5 false
		]
		shear
		[
			keys 1
				0 1 0.1 0.5 0 0 0
		]

		### layer effects

		### mesh layer values
		noisy_shapes false
		noisy_lines false
		animated_noise false
		extra_sketchy true
		noise_amp 0.012346
		noise_scale 0.098765
		extra_lines 5
		gap_filling false
		exclude_lines_from_mask false
		fill_texture ""
		line_texture ""
		mesh
		[
			points 0
			curves 0
			shapes 0
			groups 0
		]
	}
	layer_type 1
	{
		### generic layer values
		name "vector layer 1"
		quality_flags 2044
		animated_layer_effects false
		origin 0 0
		parent_bone -1
		visible true
		render_only false
		edit_only false
		scale_compensation true
		rotate_to_follow false
		face_camera false
		masking 0
		blend_mode 0
		camera_immune false
		dof_immune false
		timing_offset 0

		### transforms
		translation
		[
			keys 1
				0 1 0.1 0.5 0 0 0
		]
		scale
		[
			keys 1
				0 1 0.1 0.5 1 1 1
		]
		rotation_x
		[
			keys 1
				0 1 0.1 0.5 0
		]
		rotation_y
		[
			keys 1
				0 1 0.1 0.5 0
		]
		rotation_z
		[
			keys 1
				0 1 0.1 0.5 0
		]
		flip_h
		[
			keys 1
				0 1 0.1 0.5 false
		]
		flip_v
		[
			keys 1
				0 1 0.1 0.5 false
		]
		shear
		[
			keys 1
				0 1 0.1 0.5 0 0 0
		]

		### layer effects

		### mesh layer values
		noisy_shapes false
		noisy_lines false
		animated_noise false
		extra_sketchy true
		noise_amp 0.012346
		noise_scale 0.098765
		extra_lines 5
		gap_filling false
		exclude_lines_from_mask false
		fill_texture ""
		line_texture ""
		mesh
		[
			points 0
			curves 0
			shapes 0
			groups 0
		]
	}
}

I am making progress.