Your Unified Reader

thepetabyteproject, updated 🕥 2023-03-14 12:39:02

your

your stands for Your Unified Reader. This library can read data in Sigproc Filterbank, PSRFITS, and PSRDADA formats in a unified way and can convert from one format to another.

your implements a user-friendly interface to read and write in the data format of choice. It also generates unified metadata corresponding to the input data file for a quick understanding of observation parameters and provides utilities to perform common data analysis operations. your can be used at the data ingestion step of any transient search pipeline and can provide data and observation parameters in a format-independent manner. Generic tools can thus be used to perform the search and further data analysis. It also enables online processing like RFI flagging, decimation, subband search, etc.; functions for some of these are already available in your.

your will not only be useful to experienced researchers but also new undergraduate and graduate students who otherwise have to face a significant bottleneck to understand various data formats and develop custom tools to access the data before any analysis can be done on it.

The inspiration for the name comes from the introduction of every Daily Dose of Internet video.

Installation

You can install your directly using pip

bash pip install your Or if you want to try out the lastest stuff, bash pip install git+https://github.com/thepetabyteproject/your.git

or you can do: bash git clone https://github.com/thepetabyteproject/your.git cd your pip install -r requirements.txt python setup.py install

Note: To use the psrdada format, you would need to install psrdada-python. your_heimdall.py requires Heimdall and psrdada-python. To run the tests you would need to install pytest.

Documentation

Have a look at our docs for the documentation.

Tutorials

Here are some tutorial notebooks to get you started.

Note: To run the tutorial notebooks you would need to install jupyter.

Citation

If you use Your, please cite our JOSS Paper:

bash @article{Aggarwal2020, doi = {10.21105/joss.02750}, url = {https://doi.org/10.21105/joss.02750}, year = {2020}, publisher = {The Open Journal}, volume = {5}, number = {55}, pages = {2750}, author = {Kshitij Aggarwal and Devansh Agarwal and Joseph W. Kania and William Fiore and Reshma Anna Thomas and Scott M. Ransom and Paul B. Demorest and Robert S. Wharton and Sarah Burke-Spolaor and Duncan R. Lorimer and Maura A. Mclaughlin and Nathaniel Garver-Daniels}, title = {Your: Your Unified Reader}, journal = {Journal of Open Source Software} }

Contributions

Code Contributions

We welcome all types of code contribution. Please have a look at our guideline and code of conduct.

Contributors

Issues

your_candmaker GPU underutilization

Describe the bug Every time we run your_candmaker using a GPU we get the following warning:

/numba/cuda/dispatcher.py:488: NumbaPerformanceWarning: Grid size 64 will likely result in GPU under-utilization due to low occupancy.

I understand it's a warning and not an error but it logs a ton of these errors. We've run your_candmaker with several different input files and they all produce this issue

To Reproduce your_candmaker.py -c $FIL.csv -fs 256 -ts 256 -g $SLURM_JOB_GPUS -o ./candidates/

Expected behavior Not sure if the code can be modified to use a better grid size or if the warnings can be suppressed similar to https://stackoverflow.com/questions/29347987/why-cant-i-suppress-numpy-warnings

Versions (please provide the versions of the the following packages): - your: 0.6.7 - matplotlib: 3.5.2 - numpy: 1.22.4 - h5py: 3.7.0 - scikit-image: 0.19.3 - scipy: 1.8.1 - numba: 0.56.0 - astropy: 5.1 - pandas: 1.4.3 - rich: 12.5.1 - python: 3.10.5

Additional context We have your built into it's own Conda environment.
We are always running this code on a compute cluster with a Tesla V100 GPU. We have not tried it on other GPUs

Releases

0.6.7 2022-01-22 20:28:29

What's Changed

• make even to jess_combine, change how the bands are added, don't chec… by @josephwkania in https://github.com/thepetabyteproject/your/pull/91
• fixing data_type in filwriter by @KshitijAggarwal in https://github.com/thepetabyteproject/your/pull/93

Full Changelog: https://github.com/thepetabyteproject/your/compare/0.6.6...0.6.7

0.6.6 2021-11-13 22:10:11

What's Changed

• Gulp size and overflow handling by @KshitijAggarwal in https://github.com/thepetabyteproject/your/pull/75
• Updated your_viewer's timeseries + bandpass autoscaling by @josephwkania in https://github.com/thepetabyteproject/your/pull/76
• updates to your_viewer by @josephwkania in https://github.com/thepetabyteproject/your/pull/77
• removed chisquare test from math.py/bandpass_fitter by @josephwkania in https://github.com/thepetabyteproject/your/pull/80
• Adding candidate h5 test by @devanshkv in https://github.com/thepetabyteproject/your/pull/81
• allow use of sk_sg_filter by passing channel bandwidth and sampling interval by @mef51 in https://github.com/thepetabyteproject/your/pull/83
• Freq parameters in fits files by @KshitijAggarwal in https://github.com/thepetabyteproject/your/pull/82

New Contributors

• @mef51 made their first contribution in https://github.com/thepetabyteproject/your/pull/83

Full Changelog: https://github.com/thepetabyteproject/your/compare/0.6.5...0.6.6

0.6.5 2021-04-25 18:10:04

• Writing 4 pol psrfits 95bd2cdb563d6a423735184169c2206317f10b60
• Fix for 4 bit psrfits data f1406b42ee7d226076478b66d872a32a3d2689bd
• Added bandpass subtraction to your_viewer 90aafcaaf2aac2f810f5a8fa2697437e1ba73e1d
• Added bandpass fitter 90aafcaaf2aac2f810f5a8fa2697437e1ba73e1d
• Fixing nbits for dada 4fdf9eb1d0ba3c8aca3cb567db0d1cd94cca07ba
• Returning 4 pols using get_data https://github.com/thepetabyteproject/your/commit/e111f8db3ec4a24f57a4825a079d262cb49612fe
• Modified selection of individual pol e111f8db3ec4a24f57a4825a079d262cb49612fe
• Added poln_order to pysigproc and header e111f8db3ec4a24f57a4825a079d262cb49612fe
• Handling 4 pol data in Writer a346a3bd0bad65bf52aa9d9e20c40ea7193be1d5
• Changed the default number of spectra per subint in your_writer 57cfaddb0c0c8e7e45ee1444412b6266b936ed07
• Handling AABB pol ec9f39ee5e486d3e8c85962b71f035a75bdfd7a2
• Zero DM subt in Writer e2dcdbad0bc819a6d25ddd59fb4615f1930c98a0
• Fixed telescope mismatch warning 640fb3cc124d732651636a2e64eb6874adcb291c
• Added documentation versioning with d68cca6216870423112ce27a34c41c1c5dee8e14.

0.6.4 2021-03-26 18:25:13

• Fixed filename error that prevented your_viewer from saving display with a757d6b.
• Allowed handling empty psrfits file with 42e10ff and 45d212e.
• your_viewer.py now supports dedispersion with a2c1e53.
• Support for python 3.9 and numpy>=1.20 with 718869c and e805747.

0.6.3 2021-02-09 15:41:05

• Using rich tables in your_header and your_viewer https://github.com/thepetabyteproject/your/commit/505acd8e8cff37f8a28546fef2d032cf4b6610ac
• Can use numpy array and dictionary of header to write a filterbank file https://github.com/thepetabyteproject/your/commit/8e0a9d4d8bd68d6c44d6ef718be269cb018a5175
• Added tutorial to write filterbank from scratch https://github.com/thepetabyteproject/your/commit/82a322b3af964768262791591456cd6dbed0290b
• Gulp size and replacement policy (for RFI) as inputs in your_writer https://github.com/thepetabyteproject/your/commit/0272c22b4a6ab9ba97fa10e5945448a0f8e2ebe9 https://github.com/thepetabyteproject/your/commit/55aacd4f64f7594085541692393cdc3223cf452a
• Can ignore RFI filter if the sigma is set to zero https://github.com/thepetabyteproject/your/commit/a0bcf9024fac8be002c53427f26c5d61602237a7 https://github.com/thepetabyteproject/your/commit/9651fd8a966152202ef2c5053abe0f20175624cb https://github.com/thepetabyteproject/your/commit/311259c158f6ddfa5671d40ad1c957c5ed213a64
• More tests https://github.com/thepetabyteproject/your/commit/c9d943223e0a38cb0525b19cf79d59565d455f81 https://github.com/thepetabyteproject/your/commit/da4a89dc99008ad935bd147b82ba2c1c1a4597d3
• Added RFI mitigation in your_candmaker and Candidate https://github.com/thepetabyteproject/your/commit/515b88065709c27192665b547c01d195c29acd40 https://github.com/thepetabyteproject/your/commit/4f143f10005fa42eb10bd03a64ae2dbdbbf59f58
• Using a minimum Savgol window size https://github.com/thepetabyteproject/your/commit/6e28424369401f74357e3789330c8e301f158bff

0.6.1 2021-01-20 02:50:20

• Black check on push with 8ded440.
• Skip unknown header parameters in filterbank with ccfd38f.
• Update psrdada-python URL with 624f6dc.
• Updated docs with 726c3b9.