Using Objectstore (S3-like storage) in livecode

[DDD]

Hi There

I'm new to livecode and developed some simple apps till now with it. Now for my new project, I like to use S3-like objectstore. There are a lot of interesting providers that have S3-like objectstore, so you don't depend on Amazon.

Unfortunately, I can't find any information about it. Not even google or this forum-search gives me usable results. What I found till now
- .....what-does-infinite-livecode-mean-for-users-of-the-platform" (sorry, can't post the link here) ->it claimes to have S3 support. But I can't find anything about it nor how to get ur use it
- Livecode documentation: nothing about S3 or objectstore

So is there an official support for S3-like objectstore? If yes, where can I find the documention for it?

Thank you in advance for your help and I appologize in advance, if I'm maybe just looking for the wrong key words and should be obvious

[bogs]

it claimes to have S3 support. But I can't find anything about it nor how to get ur use it

Actually, it does *not* claim to have support for S3, it is giving you a hypothetical situation where support for S3 *could* happen.

Here is the quote from the page (bold emphasis on keywords are mine) -

What does Infinite LiveCode mean to you, as an end user? Suppose you do not yourself feel you have the time, ability or inclination to write new widgets or extensions for the platform, how does it benefit you?

Well, it’s just this: If wrapping external functionality that already exists in lower level programming languages becomes easier then more widgets and libraries will be developed and your LiveCode toolbox will be unrecognisable to what it is today. New widgets will start appearing in droves, for you to simply pick up and use in your app. Some will be free, some may be paid, but all of them will make LiveCode a bigger, better and more vibrant platform for developing rich full featured apps on all platforms.

Let’s consider some hypothetical examples.

aws

Let us assume that someone has decided to wrap the Amazon Web Services C library using our automated tool. They have then decided to build a LiveCode library that provides easy access to many of the most common AWS services.

Without having to learn any new languages you can easily integrate Amazon AWS Services into your applications and take advantage of Simple Storage Service (S3) SimpleDB DynamoDB Simple Queue Service (SQS) Simple Email Service (SES) Simple Notification Service (SNS) & Security Token Service (STS).

None of the above says or even implies this is currently available or even being worked on, it just says that it is *possible* in the *future* that someone *may* develop a widget to do these things and make the widget public (either for pay or for free).

If that occurs, you can then download the widget without writing the code yourself to do these things, but as it stands now, you'd have to figure out how to write the code yourself.

sorry, can't post the link here

The forum rules will prevent you from posting links until you reach a 10 post count, due to the amount of spam bots around. Post a lot of questions that are on top of your mind, or even in relation to this, and you should be posting links in no time

[FourthWorld]

Amazon-specific APIs are not portable by design. The world has many free and open solutions, and public cloud vendors write their own APIs to address them for vendor lock-in.

So do you want S3, or an object store for Google Cloud, or Microsoft Azure, or do you want something portable?

I think attempting to write one library for all possible proprietary storage APIs will be more work than ROI would support.

[jacque]

There is an AWS library that wraps the S3 API, but I believe it is only available to LC Business licenses. It might be available to other paid licenses but I'm not sure.

[DDD]

Thank you all for your fast replies.

@bogs: Thank you for the clarification. In this case I missunderstood the situation

@FourthWorld

So do you want S3, or an object store for Google Cloud, or Microsoft Azure, or do you want something portable?

S3 is a defacto standard and used by many providers and storage vendors. If you ment with "portable" that you can use it for different providers: it already is. So for your question:

1) Yes, I want to use S3-like storage and I already use it in a lot of my script. A lot of business application uses it too
2) No, I don't want to use a proprietary objectstorage like google Cloud or MS Azure that is not a defacto standard and with this only used by one vendor.

I think attempting to write one library for all possible proprietary storage APIs will be more work than ROI would support

That's why I would go for one of the biggest and that's S3-like object storage (defacto standard). And most important: You don't need all hundrets of API calls. It's fine to start with the 10-20 most common ones.

@jacque

There is an AWS library that wraps the S3 API, but I believe it is only available to LC Business licenses. It might be available to other paid licenses but I'm not sure.

Thank you for this information. Do you have a link to the documentation? In the case that it's only for the LC Business licenses, it's not useable as a teacher or student.

Just as a hint: Objectstore is the way to go in the future for a lot of usecases. I believe that LiveCode is not only missing a big trend but even a big opportunity if S3-like objectstore is not supportet in the future.


Edit on 22. October: I used now the correct quotation to make the text easier to read. Nothing changed from the text

[FourthWorld]

1) Yes, I want to use S3-like storage and I already use it in a lot of my script.

If you already have a solution, what parts are you currently looking for?

2) No, I don't want to use a proprietary objectstorage like google Cloud or MS Azure that is not a defacto standard and with this only used by one vendor.

Thank you for confirming that Amazon's S3 API is not compatible with the other two of the Big Three public cloud vendors.

They all offer object stores, and at least Azure offers import capabilities from Amazon S3, but their APIs are not interoperable. It may be that the Oracle v Google decision is what's keeping replication of Amazon's API limited to smaller vendors at this time.

That said, Amazon's clear marketshare lead does indeed make it a good first choice for API adoption.

And most important: You don't need all hundrets of API calls. It's fine to start with the 10-20 most common ones.

Which of those are not currently supported in your existing scripts? And would the REST API work for your use-case, or does your app specifically require the SDK?

REST should be accomplishable with relative ease in LiveCode Script. In those cases where the SDK is needed we could use LiveCode Builder.

Just as a hint: Objectstore is the way to go in the future for a lot of usecases. I believe that LiveCode is not only missing a big trend but even a big opportunity if S3-like objectstore is not supportet in the future.

Very few languages offer direct support for the vast-and-growing range of storage options as elements of the language itself by the core team. In most cases such things are provided as optional add-ons by the community using the language. This can be the case with LiveCode as well.

Key-value stores with extensible metadata are indeed growing in popularity for their simplicity and scalability. I would be happy to see what we can do to bring together interested members of our community to provide and maintain support for Amazon S3 and other object stores.

We can start with the portions you have working now, and focus efforts on the portions you need to add next. Your suggestion of limiting initial scope to a subset of fewer than two dozen API calls makes the project attractively actionable.

[jacque]

I looked it up and it's available to Indy licenses. You can just search the dictionary for "AWS" and it should come up, I'm not at my Mac right now to get a link.

[FourthWorld]

I looked it up and it's available to Indy licenses. You can just search the dictionary for "AWS" and it should come up, I'm not at my Mac right now to get a link.

I took a moment to try to find the docs on that, and came up empty when searching the Dictionary and the livecode.com site. Maybe they haven't re-indexed since adding that.

I've submitted a request on that to the use-livecode list, and will report back what I find.

Being in Indy still prevents those making GPL-governed apps from using LC with Amazon's S3 service, but at least it covers the most common needs among LC developers.

If a GPL-governed version is needed I'd be happy to see what I can do to help steward such a project.

[DDD]

Thanks again for all your answers and clarifications

If you already have a solution, what parts are you currently looking for?

Which of those are not currently supported in your existing scripts? And would the REST API work for your use-case, or does your app specifically require the SDK?

My post before was a bit misunderstood because how I wrote it (my fault). I meant: I use it in a lof of my bash scripts, tools etc. and would like to use it now in LiveCode too. But: I'm not using it in LiveCode at the moment. I just started with LiveCode to find out if this would help my students for their projects. That's why I'm looking how to use S3-like objectstore for LiveCode. Sorry for this misunderstanding.

It may be that the Oracle v Google decision is what's keeping replication of Amazon's API limited to smaller vendors at this time.

S3-like object store is used by a lot of big vendors and service providers. So it's not limited to smaller vendors only. That's why S3 is a defacto-Standard. Unfortunately it is not a Standard, so the S3-like storage/service don't support all the same API request. Most of them support about 10-30% of it, but this is for the most use cases more then enough.

REST should be accomplishable with relative ease in LiveCode Script. In those cases where the SDK is needed we could use LiveCode Builder.

Yes it could be used over REST. The question is more: How to make it as simple as possible (if this is not the goal for LiveCode). If there is an easy way like S3CMD for bash, why not have something for LiveCode? Enter in your access and private/secrete key, the link to the endpoint from your provider of choice and that's it. From then on, you can just use simple commands like "put" to upload files to a specific bucket. So the students can use it out of the box in their apps.

Very few languages offer direct support for the vast-and-growing range of storage options as elements of the language itself by the core team. In most cases such things are provided as optional add-ons by the community using the language. This can be the case with LiveCode as well.

That may be correct, that only a few languages offers direct support (I don't have the background on this). But why not implement an easy way for S3-like object storage if "S3" is at the moment a defacto-standard. For me it is fine to use an add-on from the community if it's not directly implemented in LiveCode. As I understood there is something to use with the Indy LC, that I cant find at the moment. And the bigger problem for me: if it is only for the Indy LC, I can't use it for my student projects, because they use the opensource version.

[bogs]

I hope you will pardon my gross ignorance on the topic, since I don't do anything with what you are talking about, but Lc can execute stuff from the cli so why not call your already made bash scripts from inside the program to do whatever it is your trying to do?

[DDD]

since I don't do anything with what you are talking about, but Lc can execute stuff from the cli so why not call your already made bash scripts from inside the program to do whatever it is your trying to do?

Thank you for the hint, I didn't know that you can integrate this. Unfortunately for my case it isn't a solution, because I want students to use S3-like ObjectStore directly in their app for different use cases. And for my future apps I'd like to use it directly too for a lot of different things that have nothing to do with my previous bash script(s).

Some background info
If I would just use 1:1 my script in LC, that could be a solution (not the case at the moment). But even then I would prefer to implement it directly in LC instead of adding some external scripts and depend on it. That the script works, I must use a Linux distribution and have a specific tool called "S3CMD". In my opinion this are way to many dependencies. And for your understanding: my script does solve a very specific problem and at some point it writes the generated data to S3. The write to S3 is a simple command that depends on S3CMD, that handles all the specific API calls to S3.

That's why I would really like to use it directly in LC:)