@seberg, I just don't understand why ASV doesn't provide a direct way to specify custom build options? Am I missing something?

I looked around a bit, and don't see a way other than as part of the config. However, we could use --config to use a custom config file rather than deleting the current config? May want to ping Pauli, though, at least if that doesn't see right.

I think this is a good solution to benchmark flexible universal intrinsic usage. but asv community still needs to considering the necessity of adding {build_args} to asv.config.json. @pv what do you think?

@seberg,

I looked around a bit, and don't see a way other than as part of the config.

I confirm it, thank you!

we could use --config to use a custom config file rather than deleting the current config?

I followed your suggestion, I kept the main config file as-is so it can be used with ASV command.

@Qiyu8,

I think this is a good solution to benchmark flexible universal intrinsic usage.

An effective workaround but we still have other obstacles such as providing auto-generated test data and
provides the ability to check the sanity of it, similar to what OpenCV do, another nightmare I guess.
It would be easier for me to drop ASV and create our own benchmark tool, but I guess the community wouldn't easily
accept such a move like that.

Maybe its right and we should consider upstreaming things to ASV. I am a bit worried that running plain ASV won't have access to this, but there may not be a good way to do that.
Does this give the baseline/optimized/disabled versions different environment names in the ASV run?

@seberg,

Does this give the baseline/optimized/disabled versions different environment names in the ASV run?

No, it generates a config file with the required build options then pass it through
--config, it also clears whole cache the involved commits cache if the user changed the build options.

@seberg,

I am a bit worried that running plain ASV won't have access to this, but there may not be a good way to do that.

I didn't touch the original config file, I created a new asv config instead asv_compare.conf.json and it only works through --bench-compare.

This needs documentation so people will know how to use it. Maybe in the README.rst or runtests.py files?

Thanks Sayed. I put this in to aid the ongoing SIMD work, but more usage documentation would be helpful.