Analysing binary data of an audio file (on mobile)

[dave.kilroy]

Hi all

I know it is possible to rip an image file apart and analyse it bit-by-bit to make pixel-by-pixel changes) - is it possible to do something similar with audio files?

I would like to make a guitar tuning app for my daughter (actually I would like both of us to work on it together) but before I get too far down the road with this idea I just wanted to know if it is possible with LiveCode

I'm imagining that it would be possible to go through the data in an audio recording of my daughter plucking one of the strings on her guitar and in some way figure out what megahertz it is on (as you can see I am ignorant of musical things) - and then compare this with the megahertz (or something else?) of a perfect middle C or other note. I might represent the relationship between the recorded sound and the perfect middle C in various ways on screen and as a result she would tighten or loosen the string and try again...

I hope this makes sense - anyone done something similar? Or got ideas how I might proceed?

Kind regards

Dave

[FourthWorld]

Audio is much more difficult to work with than images, not only because of the size and complexity of the data but also because of patents. Most common audio formats are patented, and LiveCode relies on the host OS to provide decompression and playback. Tapping into the codec output stream would require access to the OS APIs for such routines. This could be done with a LiveCode external if you're familiar with C or other languages that can use those APIs, or perhaps when LiveCode 8 comes out (that'll be a few months yet) you may be able to use the new Open Language extensions to try accessing those OS APIs directly from within LiveCode.

[dave.kilroy]

Thanks Richard

Last night I found some proper time for more research and came across references to the late Mark Smith's audio waveform stack - I found a url where his (related?) 'Audio File Info' library was apparently hosted http://www.webring.org/l/rd?ring=runtim ... evstuff%2F but the link to the file is bad and I couldn't download it.

Mark's audio waveform stack was apparently built for ppc rather than intel macs but Mark presumably figured out a way to read the data sufficiently to be able to produce a waveform. I have no idea if he made use of an external, or a library or api call to the os to make it work - and it may be that the methods he used are no longer available - but if anyone here remembers Mark's stack it would be great to see if you could throw any light on how he got it working.

It also looks like Stephen Barncard was independently working in the same area as Mark (but on an audio file converter), I don't know how far he got but apparently got a lot further on than I am right now!

Yours in a spirit of enquiry

Dave

[SparkOut]

Another approach may be to shell out to a command line processor that can do the job. http://sox.sourceforge.net/ seems to have the option to combine files.

[dave.kilroy]

Thanks sparkout - OSX looks really nice (although I think I only need a tiny bit of it) - however I don't think you can shell out from iOS? Also use of GPL license would not be allowed?

[Thierry]

.... -
anyone done something similar? Or got ideas how I might proceed?

Hi Dave,

Is this what sort of thing you want to achieve:

https://www.youtube.com/watch?v=_KwzY-YMYBk

Regards,

Thierry

[dave.kilroy]

Thierry

Wow that is so impressive! Were you just sending me an example or were you involved in make that?

I actually think I might possibly have dreamt of you and perl last night (but in a good way) pulling up a memory of visiting your sunny-tdz website years ago... will have to pay another visit and check it out again ... do you have any perls of wisdom (ahem) to lay before us at this stage?

Unfortunately I have to get back to work now but will continue my researching tonight...

Kind regards

Dave

[Simon]

Found it
http://forums.livecode.com/viewtopic.ph ... 53&p=36671

Simon

[FourthWorld]

Mark's audio waveform stack was apparently built for ppc rather than intel macs but Mark presumably figured out a way to read the data sufficiently to be able to produce a waveform.

His stack is a good example of the issues I described: it only worked on uncompressed audio in one format, because attempting to write an efficient audio codec in any scripting language would be difficult, and doing so legally with any of the patented formats (e.g. MP3) would be subject to the patent holder's restrictions.

Trevor's external uses the post-decompression stream from the OS APIs provided for that.

[Thierry]

Thierry
Wow that is so impressive! Were you just sending me an example or were you involved in make that?

Dave, unfortunately, at this very moment, I can only share this link..

I actually think I might possibly have dreamt of you and perl last night (but in a good way)

I feel happy for you that it was not nightmares

But, you know, I've more or less stopped talking of sunnYperl here (not much appreciated); I keep using it as my own tool, and luckily it helps me a lot. And as I enjoy connecting tools, Lua was my latest language integration inside Livecode, umm, 5 years already...

Good luck with your project; it's an interesting one.

Kind regards,

Thierry

[dave.kilroy]

Thanks Simon!

Found it
http://forums.livecode.com/viewtopic.ph ... 53&p=36671

That's really helpful - I haven't had time to spend much time rootling through the code (we have new kittens who eat a lot and put out a lot at the other end) but it looks like Mark did it without any external and just pure LiveCode (looks like he got help from Mark Waddingham with some of the functions) - but much of it seems to depend on knowing what is in the binary file and how to pull out relevant bits (it's looking more and more like I will need to spend time with standards documents or similar that describe what is actually in a .wav (for example) file...

Dave

[dave.kilroy]

Thanks Richard

His stack is a good example of the issues I described: it only worked on uncompressed audio in one format, because attempting to write an efficient audio codec in any scripting language would be difficult, and doing so legally with any of the patented formats (e.g. MP3) would be subject to the patent holder's restrictions.

I think I only need to work with uncompressed files. I was thinking that when tuning one of the guitar strings I would only need to record a second or so audio of it twanging to then be able to open the file and find out from it binary data what note it was and compare that against a custom property representation of middle C or some other note.

Mark's stack works with AIF, WAFV and SD2 files - I know audio recordings on iOS are by default saved as m4a files but haven't yet had a chance to see how this compares to what Mark's stack can do and/or is it possible to convert m4a to wav

By the way Mark's stack runs nicely in the IDE (tried it first in LC 5.0.0 and as it was OK in that I tried LC 6.6.4 and it still seems happy, so no powerpc vs intel chip issues yet)

And when I get a chance I'll try building a modified version of it for iOS 8 and running it on my phone (I haven't looked at anything like all the code and it may well call on something which limits to desktop...)

...But interesting - and a testament to his and Mark Waddingham's good work...

Dave

[dave.kilroy]

Hi Thierry

From what I think you've already managed to pull off with perl and LiveCode, it sounds like perl might be better than an external made with objective-C (at least until LC 8 widgets come on stream)?

Can I ask you a couple of questions about sunnYperl? Does it work on iOS or Android? Does it have to have a midi input (as in an electric guitar) or could it work with a recording?

Also - what are you getting Lua to do for you? Sounds exciting

Kind regards

Dave

But, you know, I've more or less stopped talking of sunnYperl here (not much appreciated); I keep using it as my own tool, and luckily it helps me a lot. And as I enjoy connecting tools, Lua was my latest language integration inside Livecode, umm, 5 years already...

Good luck with your project; it's an interesting one.

Kind regards,

Thierry

[Thierry]

Can I ask you a couple of questions about sunnYperl?
Does it work on iOS or Android?
Does it have to have a midi input (as in an electric guitar) or could it work with a recording?

I did use only the Livecode desktop external API, umm, in fact I started with metacard.
And about Perl on iOS, I'm not sure if lawyers will be happy, but I didn't take this road.
About Perl and Midi, I've use it to stress my sunnYmidi code during testing but not more.
You might search in the CPAN for this.

Regards,

Thierry

[Thierry]

Hi Thierry
what are you getting Lua to do for you? Sounds exciting

Sure it is

.....

Regards,

Thierry