Revigniter V2 sanitizing issue

Are you using LiveCode to create server scripts or CGIs?

Moderators: FourthWorld, heatherlaine, Klaus, kevinmiller, robinmiller

Post Reply
istech
Posts: 172
Joined: Thu Sep 19, 2013 10:08 am

Revigniter V2 sanitizing issue

Post by istech » Wed Nov 24, 2021 11:57 pm

Hi All,

Maybe Ralf can answer this when he is free. A final issue has popped up when testing Revigniter V2 with my project. Upon POSTING an array to a controller it now sanitizes my POST before it gets to the controller. So my POST is empty. The error I get is attached. It seems as though input lib does not like any array keys with base64 encoded data. With V1 I could POST arrays with base64 encoded data with no issues. Has this changed in V2? If so is there a way around it with not too many changes? Would love to hear your thought on this.

Many thanks

PS - Some of my errors in the attachment are from my handler picking up on the empty POST. In my logs I also get "GLOBAL POST & COOKIE WHERE SANITIZED "
Attachments
log3.zip
(454 Bytes) Downloaded 12 times

Ralf Bitter
Posts: 14
Joined: Mon Aug 26, 2013 6:49 pm

Re: Revigniter V2 sanitizing issue

Post by Ralf Bitter » Thu Nov 25, 2021 7:24 pm

Hi istech,

valid characters for POST keys are ”(?i)^[a-z0-9:_\/-]+$" (regular expression).
This has actually never changed, except for the brackets added in version 1.7.2.
The problem is that base64 encoded strings may have one or more equal signs at
the end of the line. I think you can safely remove these before posting, then
base64 encoded keys should work.
Ralf

istech
Posts: 172
Joined: Thu Sep 19, 2013 10:08 am

Re: Revigniter V2 sanitizing issue

Post by istech » Thu Nov 25, 2021 8:11 pm

Thanks for that Ralf.

Previously, I have added "a-z 0-9~%.:_\-?=+/" into gConfig["permittedUriChars"] in the config.lc file. Would this not allow the POST with the base64 to pass as is?

Many thanks

Ralf Bitter
Posts: 14
Joined: Mon Aug 26, 2013 6:49 pm

Re: Revigniter V2 sanitizing issue

Post by Ralf Bitter » Fri Nov 26, 2021 12:31 am

Hi istech,

this setting applies only to URLs and has no effect
on POST data.
If you absolutely need equal signs in POST keys you
may consider to add an equal sign to the regular
expression in function _rigCleanInputKeys() in the
Input library.
Ralf

istech
Posts: 172
Joined: Thu Sep 19, 2013 10:08 am

Re: Revigniter V2 sanitizing issue

Post by istech » Sat Nov 27, 2021 12:31 pm

Hi Ralf,

Of course you are right. I have made your recommendation to try and add "=" to the above function however I don't think this is possible as the whole site can not be reached with this error displayed. "Disallowed Key Characters. ri_session" and my POST response error is now "Disallowed Key Characters. eyJhcGkiOiB7ImtleSI6ICJhQjJRZ0pka0VrNHM5cmpzb1FDRyIsInRhc2siOiAidXBkYXRl"

Very strange, It seems as though it sees my base64 encoded POST as an array key?

If you find the time can you let me know how the Input.livecodescript gets the POST/$_POST_RAW before releasing it to the controller?

private function _rigCleanInputKeys pKey
if matchchunk(pKey, "(?i)^[a-z0-9:_\/-]+=$") is FALSE then
put "Disallowed Key Characters." && pKey
exit to top
end if

return pKey
end _rigCleanInputKeys

Ralf Bitter
Posts: 14
Joined: Mon Aug 26, 2013 6:49 pm

Re: Revigniter V2 sanitizing issue

Post by Ralf Bitter » Sat Nov 27, 2021 1:57 pm

Hi istech,

I am in a hurry right now, but will deal with it later.
There seems to be a misunderstanding on my side.
Thought your POST array keys are base64 encoded.
Now it turns out your POST array is encoded. I assume
you followed the instructions regarding dealing with arrays
in POST data in chapter Input Library of the user guide.
Ralf

Ralf Bitter
Posts: 14
Joined: Mon Aug 26, 2013 6:49 pm

Re: Revigniter V2 sanitizing issue

Post by Ralf Bitter » Sun Nov 28, 2021 3:20 pm

Hi istech,

could you provide a sample array so I can investigate
where the problem is?
Ralf

istech
Posts: 172
Joined: Thu Sep 19, 2013 10:08 am

Re: Revigniter V2 sanitizing issue

Post by istech » Mon Nov 29, 2021 1:12 pm

Hi Ralf,

Sorry for my late reply I was out all day yesterday.

I have found out the cause but do not know how to fix it as yet. The problem comes from some base64 encoded image information stored in my array. When I remove the based64 encoded image data the POST is processed fine.

The strange thing is as they are base64 encoded revigniter should not have a problem processing them. right? (I even removed the padding just to see "=")

I'll attach a sample so you can have a look today when I get back.

As always thanks for your time on this one.

PS. I am currently using an base64 encoded LC array to POST. However, If I wanted to POST a JSON array encoded with base64. Would this fail to pass the input lib because it is base64 encoded?

Ralf Bitter
Posts: 14
Joined: Mon Aug 26, 2013 6:49 pm

Re: Revigniter V2 sanitizing issue

Post by Ralf Bitter » Mon Nov 29, 2021 5:30 pm

Hi istech,

you can POST JSON base64 encoded,

you can POST JSON unencoded if you

Code: Select all

set the httpHeaders to "Content-Type: application/json”
or you can POST JSON in a base64 encoded arrayencoded LC array if you

Code: Select all

set the httpHeaders to "Content-Type: application/lc.array”
or if you are dealing with a compressed array

Code: Select all

set the httpHeaders to "Content-Type: application/lc.array.compressed”
Ralf

istech
Posts: 172
Joined: Thu Sep 19, 2013 10:08 am

Re: Revigniter V2 sanitizing issue

Post by istech » Tue Nov 30, 2021 4:44 am

Thank you for your patience Ralf,

I have attached a sample base64 encoded post for you to review. I have narrowed it down to the base64 encoded image data. But not quite sure how to get around this one as it used to POST with Revigniter V1. Would love to get some feedback. Many thanks
Attachments
sample_base64_post_removed_from_post.zip
(91.17 KiB) Downloaded 7 times

Ralf Bitter
Posts: 14
Joined: Mon Aug 26, 2013 6:49 pm

Re: Revigniter V2 sanitizing issue

Post by Ralf Bitter » Tue Nov 30, 2021 7:37 pm

Hi istech,

have tested your sample data in a POST action. There was no
problem neither by sending the base64 encoded data nor
by sending the base64 decoded (JSON) data, in this case using
a "Content-Type: application/json" header.

To verify that everything works as expected, on the server side
I converted the JSON data to an array and returned an arbitrary
value (tYourDataArray[“link"]["todo"]) of the array data. The result
was “update”.

However, it is important to note that if you post base64 encoded
data you should

Code: Select all

put false into gConfig["csrf_protection"]
If you need CSRF protection you should post the JSON data
unencoded and

Code: Select all

set the httpHeaders to "Content-Type: application/json”
Furthermore I noticed that if gConfig["globalXssFiltering"] is true
the execution of the Input library stops while sanitising data.
It seems that there is a problem with the size of your data because
if I test less data it works. So, could you please check what happens
when you

Code: Select all

put false into gConfig["globalXssFiltering"]
Please let me know if this helps.
Ralf

istech
Posts: 172
Joined: Thu Sep 19, 2013 10:08 am

Re: Revigniter V2 sanitizing issue

Post by istech » Tue Nov 30, 2021 10:21 pm

Thanks for this Ralf and I really appreciate your dedication and time.

I can confirm the config.lc file settings are false.

put false into gConfig["globalXssFiltering"]

put false into gConfig["csrf_protection"]

and also confirm the settings for the header is "set the httpHeaders to "Content-Type: application/json”

So I can only assume it has got to be a server limit for the post. I will check my server logs for any information.

Did you POST from a Livecode application/IDE? Just covering all basis.

Ralf Bitter
Posts: 14
Joined: Mon Aug 26, 2013 6:49 pm

Re: Revigniter V2 sanitizing issue

Post by Ralf Bitter » Wed Dec 01, 2021 12:46 am

Hi istech,

not that there is any misunderstanding, I thought you post
the JSON data base64 encoded. So, no need to set a
HTTP header. But previously mentioned restrictions do
not apply if you post unencoded JSON data using the
appropriate HTTP header.

And yes, I did post from the LC IDE. How do you post?
Ralf

istech
Posts: 172
Joined: Thu Sep 19, 2013 10:08 am

Re: Revigniter V2 sanitizing issue

Post by istech » Wed Dec 01, 2021 1:46 pm

Hi Ralf,

I have tried to POST with encoded and non-encoded. I have also tried both JSON and LC arrays with the same results.

I POST from my LC IDE so we have tried the same method.

I'm still investigating the issue and will update you when I have some more information. The feedback is priceless many thanks.

Post Reply

Return to “CGIs and the Server”