Ecosyste.ms: Awesome
An open API service indexing awesome lists of open source software.
https://github.com/miguelcarcamov/csromer
Compressive Sensing and Optimization Framework to reconstruct Faraday Depth signals
https://github.com/miguelcarcamov/csromer
astronomy-astrophysics astrophysics compressed-sensing faraday-depth faraday-rotation faraday-tomography framework linear-polarization magnetic-fields object-oriented object-oriented-programming python signal-reconstruction
Last synced: 3 months ago
JSON representation
Compressive Sensing and Optimization Framework to reconstruct Faraday Depth signals
- Host: GitHub
- URL: https://github.com/miguelcarcamov/csromer
- Owner: miguelcarcamov
- License: gpl-3.0
- Created: 2019-11-08T18:14:06.000Z (about 5 years ago)
- Default Branch: master
- Last Pushed: 2023-07-10T15:55:03.000Z (over 1 year ago)
- Last Synced: 2024-10-07T18:45:45.676Z (4 months ago)
- Topics: astronomy-astrophysics, astrophysics, compressed-sensing, faraday-depth, faraday-rotation, faraday-tomography, framework, linear-polarization, magnetic-fields, object-oriented, object-oriented-programming, python, signal-reconstruction
- Language: Python
- Homepage:
- Size: 149 MB
- Stars: 5
- Watchers: 3
- Forks: 2
- Open Issues: 1
-
Metadata Files:
- Readme: README.md
- License: LICENSE.txt
Awesome Lists containing this project
README
# CS-ROMER
*Compressed Sensing ROtation MEasure Reconstruction*
Compressed sensing reconstruction framework for Faraday depth spectra.
Please feel free to open an issue if you spot a bug. This is an open source project, and therefore you can fork, make changes and submit a [pull request](https://github.com/miguelcarcamov/csromer/pulls) of any of your additions and modifications.- This paper explains what is [Faraday rotation measure synthesis](https://www.aanda.org/articles/aa/abs/2005/39/aa2990-05/aa2990-05.html)
- Wikipedia information about [Faraday effect](https://en.wikipedia.org/wiki/Faraday_effect)## Features
- Simulation of Faraday depth sources
- Subtraction of Galactic RM
- Reconstruction of Faraday depth sources from linearly polarized data
- Reconstruction of Faraday depth sources using Compressed Sensing
- More than 100 wavelet filters provided by `Pywavelets`This code will run in a Python >= 3.9.7 environment with all the packages installed (see `requirements.txt` file).
## Examples
Examples and use of cases can be found [here](https://github.com/miguelcarcamov/cs-romer-notebooks)
## Citing
The paper of this software is under submission but if you use it you can cite it as:
```tex
@article{10.1093/mnras/stac3031,
author = {Cárcamo, Miguel and Scaife, Anna M M and Alexander, Emma L and Leahy, J Patrick},
title = "{CS-ROMER: A novel compressed sensing framework for Faraday depth reconstruction}",
journal = {Monthly Notices of the Royal Astronomical Society},
year = {2022},
month = {10},
abstract = "{The reconstruction of Faraday depth structure from incomplete spectral polarization radio measurements using the RM Synthesis technique is an under-constrained problem requiring additional regularisation. In this paper we present cs-romer: a novel object-oriented compressed sensing framework to reconstruct Faraday depth signals from spectro-polarization radio data. Unlike previous compressed sensing applications, this framework is designed to work directly with data that are irregularly sampled in wavelength-squared space and to incorporate multiple forms of compressed sensing regularisation. We demonstrate the framework using simulated data for the VLA telescope under a variety of observing conditions, and we introduce a methodology for identifying the optimal basis function for reconstruction of these data, using an approach that can also be applied to datasets from other telescopes and over different frequency ranges. In this work we show that the delta basis function provides optimal reconstruction for VLA L-band data and we use this basis with observations of the low-mass galaxy cluster Abell 1314 in order to reconstruct the Faraday depth of its constituent cluster galaxies. We use the cs-romer framework to de-rotate the Galactic Faraday depth contribution directly from the wavelength-squared data and to handle the spectral behaviour of different radio sources in a direction-dependent manner. The results of this analysis show that individual galaxies within Abell 1314 deviate from the behaviour expected for a Faraday-thin screen such as the intra-cluster medium and instead suggest that the Faraday rotation exhibited by these galaxies is dominated by their local environments.}",
issn = {0035-8711},
doi = {10.1093/mnras/stac3031},
url = {https://doi.org/10.1093/mnras/stac3031},
note = {stac3031},
eprint = {https://academic.oup.com/mnras/advance-article-pdf/doi/10.1093/mnras/stac3031/46643343/stac3031.pdf},
}
```## Installation
The software can be installed as a python package locally or using Pypi
### Locally after cloning the project
```shell
git clone https://github.com/miguelcarcamov/csromer.git
cd csromer
pip install .
```### Locally as developer
```shell
git clone [email protected]:miguelcarcamov/csromer.git
cd csromer
pip install -e .
```We highly recommend installing [pre-commit](https://pre-commit.com) to develop over this code.
This will allow you to run hooks that reformat the project files according to our style.### From PyPI
`pip install csromer`
### From Github
`pip install -U git+https://github.com/miguelcarcamov/csromer.git`
### From latest docker container
`docker pull ghcr.io/miguelcarcamov/csromer:latest`
## Simulate Faraday sources directly in frequency space
CS-ROMER is able to simulate Faraday depth spectra directly in wavelength-squared space. The classes `FaradayThinSource` and `FaradayThickSource` inherit directly from `Dataset`, and therefore you can directly use them as an input to your reconstruction.
### Thin sources
```python
import numpy as np
from csromer.simulation import FaradayThinSource
# Let's create an evenly spaced frequency vector from 1.008 to 2.031 GHz (JVLA setup)
nu = np.linspace(start=1.008e9, stop=2.031e9, num=1000)
# Let's say that the peak polarized intensity will be 0.0035 mJy/beam with a spectral index = 1.0
peak_thinsource = 0.0035
# The Faraday source will be positioned at phi_0 = -200 rad/m^2
thinsource = FaradayThinSource(nu=nu, s_nu=peak_thinsource, phi_gal=-200, spectral_idx=1.0)
```### Thick sources
```python
import numpy as np
from csromer.simulation import FaradayThickSource
# Let's create an evenly spaced frequency vector from 1.008 to 2.031 GHz (JVLA setup)
nu = np.linspace(start=1.008e9, stop=2.031e9, num=1000)
# Let's say that the peak polarized intensity will be 0.0035 mJy/beam with a spectral index = 1.0
peak_thicksource = 0.0035
# The Faraday source will be positioned at phi_0 = 200 rad/m^2 and will have a width of 140 rad/m^2
thicksource = FaradayThickSource(nu=nu, s_nu=peak_thicksource, phi_fg=140, phi_center=200, spectral_idx=1.0)
```### Simulate
Once you have set your source parameters, you can call the `simulate()` function as
```python
thinsource.simulate()
thicksource.simulate()
```This call will simulate the linealy polarized emission and it will assign the data to the `data` attribute.
### Mixed sources
A thin+thick or mixed source is simply a superposition/sum of a thin source and thick source. Therefore we have overriden the `+` operator in order to sum these two objects.
```python
mixedsource = thinsource + thicksource
```The result will be a `FaradaySource` object.
### Remove frequency channels randomly as if you were doing RFI flagging
The framework also allows you to randomly remove data with the function `remove_channels` to simulate RFI flagging
```python
# Let's say that we want to randomly remove 20% of the data
mixedsource.remove_channels(0.2)
```### Adding noise to your simulations
If we want to add random Gaussian noise to our simulation we can simply call the function `apply_noise`
```python
# Let's add Gaussian random noise with mean 0 and standard deviation equal
# to 20% the peak of the signal.
sigma = 0.2*mixedsource.s_nu
mixedsource.apply_noise(sigma)
```## Reconstruct 1D Faraday sources
To illustrate how to reconstruct Faraday depth signals with CS-ROMER first we will reconstruct the mixed source that we have just constructed
### Dirty Faraday depth spectra
```python
from csromer.reconstruction import Parameter
from csromer.transformers import DFT1D
# We first need to initialize the parameter object that will contain our Faraday depth
# data either in Faraday-depth space or in wavelet space
parameter = Parameter()
# We calculate the cellsize in Faraday depth space using an oversampling factor of 8
# Here parameter.data is set as a complex array of zeros
parameter.calculate_cellsize(dataset=mixedsource, oversampling=8)
# We instantiate our discrete Fourier transform
dft = DFT1D(dataset=mixedsource, parameter=parameter)
# We calculate the dirty Faraday depth spectra
F_dirty = dft.backward(mixedsource.data)
```### Reconstruct simulated data
```python
from csromer.transformers import NUFFT1D
# We instantiate our non-uniform FFT
nufft = NUFFT1D(dataset=mixedsource, parameter=parameter, solve=True)
# At this point we can use either the parameter data set with zeros or we can
# use the dirty Faraday depth spectra
parameter.data = F_dirty
parameter.complex_data_to_real() # We convert the complex data to real
# You can set the L1 lambda regularization manually or estimate it as
lambda_l1 = np.sqrt(mixedsource.m + 2*np.sqrt(mixedsource.m)) * np.sqrt(2) * np.mean(mixedsource.sigma)
```### Objective function
```python
from csromer.objectivefunction import L1, Chi2
from csromer.objectivefunction import OFunction
# We instantiate each part of our objective function
chi2 = Chi2(dft_obj=nufft, wavelet=None) # chi-squared
l1 = L1(reg=lambda_l1) # L1-norm regularizationF_obj = OFunction([chi2, l1]) # Whole objective function
f_obj = OFunction([chi2]) # Only chi-squared
g_obj = OFunction([l1]) # Just regularizations
```### Optimization algorithm
One of the ways to optimize the objective function is to use the FISTA algorithm.
```python
from csromer.optimization import FISTA
# We instantiate our FISTA object as
opt = FISTA(guess_param=parameter, F_obj=F_obj, fx=chi2, gx=g_obj, noise=mixedsource.theo_noise, verbose=False)
# We run the optimization algorithm
obj, X = opt.run()
X.real_data_to_complex() # We convert the data back to complex when the optimization finishes
```This returns the objective function value `obj` and `X`a `Parameter` instance object. Therefore in this case `X.data` will hold the reconstructed Faraday depth spectra.
At this point you can also access to the model and residual data in wavelength-squared as `mixedsource.model_data` and `mixedsource.residual`, respectively. You can calculate the residuals in Faraday depth space by using the DFT object as```python
F_residual = dft.backward(mixedsource.residual)
```### Using discrete or undecimated wavelets
CS-ROMER has about 100 filters to user with discrete wavelet transforms or undecimated wavelet transforms. We use the `Pywavelets` package, for more information please refer to [PyWavelets](https://pywavelets.readthedocs.io/en/latest/index.html). To use the wavelets in cs-romer you can do:
```python
from csromer.dictionaries import DiscreteWavelet, UndecimatedWavelet
# This line instantiates a discrete wavelet
wav = DiscreteWavelet(wavelet_name="coif3", mode="periodization", append_signal=False)
# This line instantiates an undecimated wavelet
wav = UndecimatedWavelet(wavelet_name="sym2", mode="periodization", append_signal=True)
```The `append_signal` parameter plugs the Faraday depth spectrum to your coefficients resulting in redundancy in your coefficients. If you just want the wavelet coefficients then set `append_signal=False`.
At this point our parameter object data needs to be our coefficients and not our Faraday depth spectra, therefore, we do```python
parameter.data = F_dirty # Suppose that you set your parameter data with your dirty Faraday depth spectrum
parameter.complex_data_to_real() # We convert the data to real
# Here we do a wavelet decomposition of our Faraday depth space
# We set the coefficients of the decomposition as our parameter data
parameter.data = wav.decompose(parameter.data)
# Don't forget to change your chi-squared
chi2 = Chi2(dft_obj=nufft, wavelet=wav)
```You might have noticed that at the end of the optimization we will end up with fitted coefficients instead of a Faraday depth spectrum.
Therefore, we need to reconstruct the Faraday depth spectrum from our coefficients doing```python
X.data = wav.reconstruct(X.data) # We reconstruct the Faraday depth spectrum from coefficients
X.real_data_to_complex() # We convert the real Faraday depth spectrum into complex
```### Reconstruct a real line of sight data
To reconstruct real data your main script should follow the same workflow. The only difference is that you need to instantiate a `Dataset` object.
```python
from csromer.base import Dataset
# nu is the irregular spaced frequency
# data is the polarized emission
# sigma is the error per channel (this can be an array of ones or rms calculation per image channel)
# alpha is the spectral index at this line of sight
dataset = Dataset(nu=nu, data=data, sigma=sigma, spectral_idx=alpha)
```### Subtracting the Milky Way RM contribution
We use [S. Hutschenreuter et al.](https://www.aanda.org/articles/aa/full_html/2022/01/aa40486-21/aa40486-21.html) Faraday sky HealPIX image to subtract the galactic RM contribution at a certain position of the sky using the object `FaradaySky`.
Note that you can omit this step, and subtract any RM value that you might find appropiate.```python
from csromer.faraday_sky import FaradaySky
from astropy.coordinates import SkyCoord
import astropy.units as unf_sky = FaradaySky()
coord = SkyCoord(ra=173.694*un.deg, dec=48.957*un.deg, frame="fk5")
gal_mean, gal_std = f_sky.galactic_rm(coord.ra, coord.dec, frame="fk5")
dataset.subtract_galacticrm(gal_mean.value)
```## Reconstruct a cube
We warn the users that not all framework functions are yet implemented to work with data cubes. Therefore, we need to use `numpy` broadcasting and the package `joblib`. Let's say that you have read your polarized cube and frequency array using `np.load`. For this example we will assume that you will reconstruct with uniform weights.
```python
import numpy as np
from csromer.reconstruction import Parameter
from csromer.base import Dataset
from joblib import Parallel, delayedQU_cubes = np.load('qu_cubes.npy') # Shape (freqs, m, n)
nu = np.load('nu.npy') # Shape (freqs,)
m = QU_cubes.shape[1]
n = QU_cubes.shape[2]Q = QU_cubes[0]
U = QU_cubes[1]
data = Q + 1j * U
sigma = np.ones_like(nu) # Uniform weights
# We will construct a dataset only to obtain Faraday-space array shapes
foo_dataset = Dataset(nu=nu, sigma=sigma, spectral_idx=0.0)
foo_parameter = Parameter()
parameter.calculate_cellsize(dataset=foo_dataset, oversampling=8)
# Faraday dispersion function cube
# Note that ee add another dimension to store dirty, model, residual and restored signals
F = np.zeros(4, foo_parameter.n, m, n, dtype=np.complex64)# Parallelize your for loop using joblib
total_pixels = m*n
nthreads = 8
workers_1d_idxs = np.arange(total_pixels)
workers_idxs = np.unravel_index(workers_1d_idxs, (M,N))
Parallel(n_jobs=nthreads, backend="multiprocessing", verbose=10)(delayed(reconstruct_cube)(
F, data, sigma, nu, 0.0, workers_idxs, i, eta, False) for i in range(0, total_pixels))
``````python
def reconstruct_cube(F=None, data=None, sigma=None, nu=None, spectral_idx=None, noise=None,
workers_idxs=None, idx=None, eta=1.0, use_wavelet=True):
i = workers_idxs[0][idx]
j = workers_idxs[1][idx]if spectral_idx is None:
spectral_idx = 0.0dataset = Dataset(nu=nu, sigma=sigma, data=data[:, i, j], spectral_idx=spectral_idx)
parameter = Parameter()
parameter.calculate_cellsize(dataset=dataset, oversampling=8, verbose=False)dft = DFT1D(dataset=dataset, parameter=parameter)
nufft = NUFFT1D(dataset=dataset, parameter=parameter, solve=True)F_dirty = dft.backward(dataset.data)
# We can estimate the noise from the edges of the FDF
edges_idx = np.where(np.abs(parameter.phi) > parameter.max_faraday_depth / 1.5)
noise = eta * 0.5 * (np.std(F_dirty[edges_idx].real) + np.std(F_dirty[edges_idx].imag))# We store the FDF
F[0, :, i, j] = F_dirty# Let's say that if use_wavelet is True then we use the coif2 wavelet
if use_wavelet:
wav = UndecimatedWavelet(wavelet_name="coif2")
else:
wav = None# We estimate lambda for L1 norm
lambda_l1 = np.sqrt(2 * len(dataset.data) + np.sqrt(4 * len(dataset.data))) * noise
chi2 = Chi2(dft_obj=nufft, wavelet=wav)
l1 = L1(reg=lambda_l1)
F_func = [chi2, l1]
f_func = [chi2]
g_func = [l1]F_obj = OFunction(F_func)
g_obj = OFunction(g_func)parameter.data = F_dirty
parameter.complex_data_to_real()if use_wavelet:
parameter.data = wav.decompose(parameter.data)opt = FISTA(guess_param=parameter, F_obj=F_obj, fx=chi2, gx=g_obj, noise=noise, verbose=False)
obj, X = opt.run()if use_wavelet:
X.data = wav.reconstruct(X.data)X.real_data_to_complex()
F_residual = dft.backward(dataset.residual)
F[1, :, i, j] = X.data
F[2, :, i, j] = X.convolve(normalized=True) + F_residual
F[3, :, i, j] = F_residual
```Note that if your Faraday depth cube is large, then probably it won't fit in your memory. Therefore, we can use `memory map`. In that case you would need to define your Faraday depth cube as:
```python
output_file_mmap = os.path.join(folder, 'output_mmap')
F = np.memmap(output_file_mmap, dtype=np.complex64, shape=(4, foo_parameter.n, M, N), mode='w+')
```## Contact
Please if you have any problem, issue or you catch a bug using this software please use the [issues tab](https://github.com/miguelcarcamov/csromer/issues) if you have a common question or you look for any help please use the [discussions tab](https://github.com/miguelcarcamov/csromer/discussions).