Novelty objects limited by fixed algorithms

[AlexHarker]

I'm not sure exactly what the best approaches to this are in each case, but it strikes me that the novelty stuff could be more useful if it was more tweakable.

For algorithm I just get "pitch/mfcc/chroma" etc. which no control over the parameters of those things (even though as individual objects they would have parameters).

One option that might be nice is to calculate the features externally and then feed them to the novelty slice/feature algorithms or clients - this might be the quickest way to getting more functionality without having to add loads of parameters, and then it would also be open to lots of other descriptors/underlying data.

[AlexHarker]

[I would have added an enhancement label, but I don't seem to be able to]

[jamesb93]

This would certainly be a nice-to-have and in line with various separations of objects that have happened recently (NoveltyFeature) for example.

As I am sure you are well aware, being more expert in C++ than myself, the algorithm could be taken pretty neatly into it's own object at the client level which accepts a buffer of 2d features. Maybe you would like to PR it as we are a little more pressed for the current time being? :)

[AlexHarker]

Sure - I might have a go - I was just putting something here to track this - I do not expect it to be high priority - I guess it was just a block where I felt I couldn't really explore as far as I wanted to with the tools as they are. At present I'm at an experimenting phase, so it's always hard to know where to put more time in on something with uncertain results.

I wonder if a separate object is best, or just an "algorithm" that reads the features from a buffer might be better?

[jamesb93]

Sure - I might have a go - I was just putting something here to track this - I do not expect it to be high priority - I guess it was just a block where I felt I couldn't really explore as far as I wanted to with the tools as they are.

It's very valuable so thanks.

I wonder if a separate object is best, or just an "algorithm" that reads the features from a buffer might be better?

For prototyping I would recommend you make it so that the current object accepts a buffer and computes the novelty kernel, self-similarity matrix etc. internally and returns you the slice points. I'm not fully cognisant of the possible roadblocks but it could be easier to duplicate the object and hack on that. Some recent changes @weefuzzy has implemented make it such that you don't need to do as much boilerplate C++ code anymore to get the objects to be picked up in cmake anymore so its a little less annoying to make new clients. Just make sure you peel off a feature branch from dev to get that.

[weefuzzy]

This has been a long-standing thought and not just for novelty. Basically it amounts to having a way to disable windowing and fft (and for novelty all the additional feature stuff), and go straight in with vectors of stuff. I think, in Max, that would probably translate to a distinct object that takes lists / buffer~s.

[AlexHarker]

Thanks for the insights - I guess given the FFT/windowing complexities (I was forgetting that all non-realtime stuff is also realtime), this would likely need to be a specific non-realtime thing, rather than an edit of a pre-existing object. I will see if I feel up to that or not. @weefuzzy - out of interest what are the other things, in case I could make something of use for them too (no promises though).

[weefuzzy]

NMF and HPSS are the other obvious ones for me. NMF is a different sort of beastie though

[AlexHarker]

They both sound quite different in that they must take actual audio, whereas here it is just data, so I think if I try this I'll just write a NRT client for novelty things (feature and then possibly slice). I think I know how to do this, so it's just the typing and the cmake stuff.

[AlexHarker]

I have proof of concept on this - might require some style/naming cleanup to pull into the main thing if that is desirable, but it's fairly straightforward - roughly the size of the bus stats client or similar

[AlexHarker]

And for what it is worth doing novelty with MFCCs with/without the first coefficient is noticeably different.

[AlexHarker]

If @weefuzzy / @tremblap / @jamesb93 have suggestions for the naming of this NRT object I think what I have is ready to make a PR for.

To clarify I currently have made just a feature object (not a slicer, although in theory that's possible, but in my case I'm likely to evaluate the feature externally in order to create slices).

The workflow is thus:

calculate descriptions in multichannel buffer -> my novelty object -> feature out

This allows you to tweak the parameters of the descriptors as you wish, or use arbitrary descriptors.
The results are roughly the same as the bufnoveltyfeature~ object for the same input with the following two caveats:

• I've set the latency as I think is correct (so the same as my other PR) so it doesn't match the public/current dev versions.

• The edges give different results as the all in one object will pad audio vectors with zeros at the edges and function accordingly, whereas this object can only see values for the frames passed in. In practice this is probably not that different but worth noting. I don't see a good workaround to this (although at the end of the file one needs "extra" frames which are currently assumed all zero, but it would be possible to provide a buffer input to set the values to a custom amount - still not quite the same as zero-padding, but potentially useful for descriptors that wouldn't return zero on zero valued input frames - this is noted in the code).

In my current build the object is called fluid.datanoveltyfeature~ to differentiate with the original, but that's not ideal as it should probably be fluid.buf...... However a 4 part name (e.g. fluid.bufcustomnoveltyfeature / fluid.bufdatanoveltyfeature~) is not particularly desirable, but it's hard to see how I can avoid without being inconsistent with the current naming schemes.

[tremblap]

r.e. latency - we spent a long time with test signals to arrive at this, so I'll need to send you the said test signals to see if you get the expected results.

r.e. naming - it is always a long process... we have discussed for years the idea to pass most audio in objects non-fft inputs so I could think of something crazy, like merging it with bufnoveltyfeatyre with @fftsettings -1 to bypass it. What I mean is that this object is a proof of concept and is very likely not to go in v1 so we have the time to get the interface and naming right, so for now you can call it nofftbufnoveltyfeature

i look forward to trying it

p