Problem with recursive API call

[matgarage]

Hello,

I’m having a problem with a recursive call to an API.
The purpose of this API is to allow users to retrieve lists of companies using filters such as sector of activity, region, etc.
The documentation is available here:
https://recherche-entreprises.api.gouv.fr/docs/

My script makes a standard call

Code: Select all

put URL tURL into tRawData

tURL is a web address:

Code: Select all

https://recherche-entreprises.api.gouv.fr/search?region=75&activite_principale=46.90Z&page=1&per_page=10

I receive a JSON response, which I then process.

Using a standard “REPEAT” function, I increment the “page=” argument by 1 each time until I get an empty page, which stops the script.

Everything works fine, except when the number of pages is large (in my case, 128), when I get the following error message:

The handler: ulGetFormat has reached the recursion limit of: 400000.
Execution will be terminated to prevent a hang

My problem is that I don’t have this handler in my code.
My debugging tests confirm that the error occurs during the 128th “put URL tURL” call.

Can anyone help me understand this?

[Emily-Elizabeth]

Instead of handling it in a repeat loop, look at using send

[bwmilby]

Is this something where you could post a stripped down example stack? We would just need the loop piece and would not need to do any of the post processing of the received data. If it fails at the same spot in a test stack others can try it out and see if we get the same result.

It looks like you can specify records per return. Does it change if you use 9 or 11 instead of 10?

That handler is part of liburl.

What version of the IDE are you using as well?

[SparkOut]

I'd also question whether your code is potentially sending a url request, looping and sending the same request again, without waiting or checking for a returned result or callback.
How do you ensure there's a valid response before sending the next api url request?

If nothing else, make sure the loop has time to update housekeeping tasks within the repeat structure by including a line

Code: Select all

wait 0 milliseconds with messages

That gives the engine just enough leverage to stop the loop hijacking all the processor cycles without essentially slowing down the handler.

[matgarage]

@ Emily-Elisabeth: I’m going to work on replacing the logic in my script by testing it with SEND

@ bwmilby: I tested it by changing the number of results per page to 9, and got the same error message on the 128th iteration.
When I reduced it to 5, I got an HTTP 429 error: too many requests. I’m not yet sure if this was ‘from my IP’ or if the server was temporarily overloaded.

@ SparkOut: There’s no great programming secret in my code, so I’ve posted it below so you can test it for yourselves.
Sorry, it’s in French...

[FourthWorld]

Consider using load with callbacks.
https://lessons.livecode.com/m/4071/l/7 ... o-livecode

Callbacks keep network I/O clean and efficient.

[matgarage]

I forgot to mention that, to test it, you need to select a business category from the data grid at the top and a region from the drop-down menu in the top right-hand corner.
Then click on ‘Recherche’.

@ FourthWorld: I’ll read this lesson carefully.

[matgarage]

It works.
Thanks to FourthWold’s advice and the LOAD function, the code is cleaner and doesn’t generate any errors.
Thanks to everyone.